Internet of vehicles millimeter wave wireless communication beam alignment method based on vision assistance

Abstract

The invention provides an Internet of Vehicles millimeter wave wireless communication beam alignment method based on visual assistance, which comprises the following steps of: 1, configuring a visual sensor at a base station, and constructing a visual information and channel information coexistence knowledge graph; 2, in the beam alignment process, collecting a real-time image in a base station service area, and extracting visual features of a communication target; 3, performing coarse positioning on the visual features of the communication target in the visual information and channel information coexistence knowledge graph by using a visual feature domain retrieval method; step 4, predicting wireless channel path information between the communication target and the base station by using a threshold weighted K nearest neighbor algorithm to obtain optimal wireless channel path information; and step 5, determining an optimal beam pair according to the predicted optimal wireless channel path information, performing beam forming parameter adjustment, and performing beam alignment control on the array antenna. According to the invention, while pilot frequency consumption is effectively reduced, millimeter wave beam alignment can be accurately realized, and the data transmission quality between the base station and the vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the field of beam alignment technology, and in particular to a vision-assisted vehicle networking millimeter wave wireless communication beam alignment method. Background Art

[0002] Millimeter wave communication has significant advantages in the Internet of Vehicles; first, it can support extremely high data transmission rates, and can achieve Gbps-level data exchange using wide bandwidth, meeting the needs of autonomous driving and high-definition video transmission. At the same time, the low latency characteristics of millimeter wave communication are essential for applications in the Internet of Vehicles that require real-time response, such as collision warning and cooperative driving. In addition, the short wavelength of millimeter waves enables high-precision positioning capabilities, which helps improve safety in vehicle-to-vehicle (V2V) communications. Through massive MIMO technology, millimeter waves can also support the connection of massive devices, ensuring smooth communication between a large number of vehicles, infrastructure and pedestrian devices in the Internet of Vehicles.

[0003] However, due to the propagation characteristics of millimeter waves, beamforming technology must be used in Internet of Vehicles applications to overcome challenges such as high path loss and susceptibility to obstacles. Beamforming can concentrate signal energy, compensate for high losses in propagation, and extend communication distance. In addition, it can dynamically adjust the beam direction to ensure that the vehicle can maintain a stable connection when moving at high speeds, while effectively avoiding obstacles in the environment. By reducing interference between signals and improving spectrum utilization, beamforming technology can also support high-density device connections in the Internet of Vehicles, ensure network capacity and coverage, and thus ensure that the millimeter wave communication system in the Internet of Vehicles operates efficiently and reliably.

[0004] However, in today's complex urban environments, the communication transmission between highly mobile vehicles and fixed base stations consumes a lot of pilot signals and cannot accurately achieve millimeter wave beam alignment, which affects the data transmission quality between base stations and vehicles and is not conducive to vehicles obtaining high-quality communication services while driving in urban environments. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention aims to provide a vision-assisted vehicle-to-vehicle millimeter-wave wireless communication beam alignment method, which can solve the existing problems.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] The present invention is implemented by the following technical solution: a vision-assisted vehicle networking millimeter wave wireless communication beam alignment method, comprising the following steps:

[0008] Step 1: Configure visual sensors at the base station to build a knowledge graph of coexistence of visual information and channel information;

[0009] Step 2: During the beam alignment process, real-time images within the base station service area are collected and visual features of the communication target are extracted;

[0010] Step 3: Use the visual feature domain retrieval method to roughly locate the visual features of the communication target in the visual information and channel information coexistence knowledge graph;

[0011] Step 4: Use the threshold weighted K nearest neighbor algorithm to predict the wireless channel path information between the communication target and the base station to obtain the optimal wireless channel path information;

[0012] Step 5: Determine the best beam pair based on the predicted best wireless channel path information, adjust the beamforming parameters, and perform beam alignment control on the array antenna.

[0013] Furthermore, configuring a visual sensor at the base station to construct a knowledge graph of coexistence of visual information and channel information includes:

[0014] 1.2. Configure visual sensors for each base station to simultaneously collect visual information and wireless radio frequency information within the base station service area;

[0015] 1.2. Extracting wireless channel features and visual features from the visual information and wireless radio frequency information;

[0016] 1.3. Establish a feature fingerprint of wireless communication and visual coexistence for each potential communication site in the base station server;

[0017] 1.4. Construct a knowledge graph of coexistence of visual information and channel information based on the feature fingerprints of each communication site.

[0018] Furthermore, the synchronous collection of visual information and wireless radio frequency information within the base station service area includes:

[0019] The images collected by the base station visual sensor in the connected vehicle environment are segmented, and each segmented grid becomes a data collection site; the visual information and wireless radio frequency information are synchronously collected at each collection site by simulation or actual measurement;

[0020] Any image I(t) taken by a visual sensor in the base station service area has a size of W I ×H I , after downsampling, a two-dimensional feature map vector is generated H o and W o The mathematical relationship with the input image is expressed as:

[0021]

[0022] where stride Hand stride W is the downsampling step size, H o and W o is the length and width of a grid, and each grid becomes a data collection site; collecting visual information and wireless radio frequency information.

[0023] Furthermore, the visual feature extraction includes extracting 5 indicators (L IOU ,L x ,L y ,W,H), where L IOU represents the confidence of the bounding box, (L x ,L y ) represents the offset of the center point of the bounding box compared to the upper left corner of the grid, and (W,H) represents the width and height of the bounding box.

[0024] Furthermore, the wireless channel feature extraction includes extracting the path gain, phase, departure angle and arrival angle of the wireless channel path.

[0025] Furthermore, the method of using the visual feature domain retrieval method to roughly locate the visual features of the communication target in the visual information and channel information coexistence knowledge graph includes:

[0026] The visual channel path graph is segmented at the pixel level to obtain multiple first-level sub-knowledge graphs, and then the first-level sub-knowledge graph is segmented to obtain the second-level sub-knowledge graph of each first-level sub-knowledge graph, and so on to obtain n-level sub-knowledge graphs; each retrieval first determines that the communication target belongs to a certain level of sub-knowledge graph, and then continues to detect in the second-level sub-knowledge graph, and so on, until the final n-level sub-knowledge graph to which the communication target belongs is found.

[0027] Furthermore, the core idea of ​​the domain retrieval method using visual features is to recursively divide the data set into subsets until a certain stopping condition is met; wherein:

[0028] Build phase:

[0029] Select the visual feature center point of the sub-knowledge graph. For a node point set X, select the centroid C of the point set as the sub-domain center:

[0030]

[0031] Where n is the number of points contained in the node, x i is the point in the node;

[0032] Calculate the radius r of the visual feature subdomain of the sub-knowledge graph so that the subdomain can contain all the points in the node:

[0033]

[0034] Where ||xC|| is the distance from point x to the center of the sphere C;

[0035] Recursively partition the point set until each node contains fewer points than a preset threshold; the partitioning method is to find the maximum distance between two points in the data set, then divide the point set into two parts, and recursively create child nodes for each part;

[0036] Search phase:

[0037] Given a query point q, the nearest neighbors are found by traversing the visual channel knowledge graph; the steps of the search process are as follows:

[0038] For each node, calculate the distance between the query point q and the center C of the sphere:

[0039] d(q,c)=||qc|| (4)

[0040] Use the triangle inequality to speed up the search:

[0041] d(q,x i )≥d(q,c)-r (5)

[0042] Among them, d(q,x i ) is the query point and a point x in the sphere i , d(q,c) is the distance between the query point and the center of the sphere, and r is the radius of the sphere.

[0043] Furthermore, the optimal wireless channel path prediction process uses the Euclidean distance as a metric for predicting the visual features of the communication target to calculate the distance of the center coordinates; specifically:

[0044]

[0045] Among them, L x,y is the image feature descriptor of the query image, is the jth image feature descriptor in the image corresponding to the i-th camera in the visual channel knowledge graph;

[0046] For the candidate objects, the candidate visual feature vectors with the K closest distances are calculated. The total number of candidate visual feature vectors is K. K is a preset value. A threshold screening strategy is added to remove abnormal values. Therefore, the set of candidate visual feature vectors is less than or equal to K. The distance between candidate visual feature vectors is expressed as (q=[1,2,3,...,N],N≤K); the weighted average method is used to predict the final channel information; the distance between each candidate visual feature vector and the query point is different, and the closer the distance, the greater the impact; therefore, a weight is assigned to each candidate visual feature vector, and this weight is inversely proportional to the distance from the candidate visual feature vector to the query point. The smaller the distance, the greater the weight; the weight is normalized to obtain the weight of each candidate visual feature vector, that is,

[0047]

[0048] The predicted channel information is the weighted sum of the channel information of the candidate visual feature vectors:

[0049]

[0050] P ij is the channel information of the i-th candidate visual feature vector; finally, the wireless channel path information of the input image is obtained.

[0051] A vision-assisted millimeter-wave wireless communication beam alignment device for an Internet of Vehicles, comprising a processor and a memory; the memory is used to store a program; the processor executes the program to implement any of the methods described above.

[0052] A computer-readable storage medium, characterized in that: the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0053] Compared with the prior art, the beneficial effects of the present invention include:

[0054] In the process of beam alignment, the base station first collects images of the service area in real time and extracts the visual features of the communication target; then, the visual feature domain retrieval method is used to preliminarily locate the target in the visual channel path map; next, the channel path information between the communication target and the base station is predicted by the threshold weighted K nearest neighbor algorithm. Finally, the best beam pair is determined, and the beam alignment parameters are adjusted to control the antenna array to achieve beamforming control and ensure the accuracy and efficiency of communication.

[0055] The present invention is used for communication transmission between highly mobile vehicles and fixed base stations in complex urban environments. While effectively reducing pilot consumption, the method can accurately achieve millimeter wave beam alignment and improve the data transmission quality between the base station and the vehicle. It is beneficial for vehicles in urban environments to obtain high-quality communication services while driving, and is also beneficial to the safe and stable operation of urban transportation communication networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The disclosure of the present invention is described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components. Among them:

[0057] Figure 1 It is a schematic diagram of the flow of the vision-assisted vehicle networking millimeter wave wireless communication beam alignment method of the present invention;

[0058] Figure 2 : is a schematic diagram of a scenario of a vision-assisted millimeter-wave wireless communication beam alignment method for Internet of Vehicles according to the present invention;

[0059] Figure 3 The middle is a schematic diagram of the visual feature domain retrieval of the present invention;

[0060] Figure 4 A flowchart of a threshold weighted K nearest neighbor of the present invention;

[0061] Figure 5 It is a flowchart after overall refinement of the present invention. DETAILED DESCRIPTION

[0062] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or restriction to the technical solution of the present invention.

[0063] The present invention is illustrated by millimeter wave alignment, but the beam alignment method of the present invention is not limited to the alignment of millimeter wave beams; the present invention provides a vehicle network millimeter wave wireless communication beam alignment method based on vision assistance, and the process is as follows Figure 1 As shown, the following steps are included:

[0064] Step 1: Configure visual sensors at the base station to build a knowledge graph of coexistence of visual information and channel information;

[0065] Specifically include: Figure 5 As shown:

[0066] 1.1. Configure visual sensors for each base station to simultaneously collect visual information and wireless radio frequency information within the base station service area;

[0067] Due to a large-scale MIMO (multiple input multiple output) wireless communication system, the images collected by the base station in the connected vehicle environment are segmented, and each segmented grid becomes a data collection site; the visual information and wireless radio frequency information are synchronously collected at each collection site by simulation or actual measurement;

[0068] Specifically, any image I(t) taken by a visual sensor in the base station service area has a size of W I ×H I , after downsampling, a two-dimensional feature map vector is generated H o and W o The mathematical relationship with the input image is expressed as:

[0069]

[0070] where stride H and stride W is the downsampling step size, H o and W o It can be imagined as the length and width of a grid, with each grid becoming a data collection site; collecting visual information as well as wireless radio frequency information.

[0071] 1.2. Extracting wireless channel features and visual features from the visual information and wireless radio frequency information;

[0072] Extract features from the image. The features can represent the position, direction, speed, attributes, etc. of the communication target in the image. Specifically, the extraction of visual features mainly includes five indicators (L IOU ,L x ,L y ,W,H), where L IOU represents the confidence of the bounding box, (L x ,L y ) represents the offset of the center point of the bounding box compared to the upper left corner of the grid, and (W, H) represents the width and height of the bounding box. The camera is used as a visual sensor for illustration. During the visual feature extraction process, the image sequence taken by each camera is I x = {I x,1 ,I x,2 ,I x,3 ,...,I x,n}, requiring the camera group’s field of view to cover the entire street. In the visual feature extraction phase, the prior box mechanism is used to predict the visual features of the target vehicle. The deep neural network predicts five indicators (L IOU ,L x ,L y ,W,H), where L IOU represents the confidence of the bounding box, (L x ,L y ) represents the offset of the center point of the bounding box compared to the upper left corner of the grid, and (W,H) represents the width and height of the bounding box.

[0073] The wireless channel feature extraction process mainly includes extracting the path gain, phase, departure angle and arrival angle of the wireless channel path; the extracted wireless channel features can represent the channel information between the base station and the user, and each collection site contains the corresponding wireless channel information.

[0074] 1.3. Establish a feature fingerprint of wireless communication and visual coexistence for each potential communication site in the base station server; that is, the wireless channel characteristics and visual characteristics form a feature fingerprint;

[0075] 1.4. Construct a knowledge graph of coexistence of visual information and channel information based on the characteristic fingerprints of each communication site;

[0076] The visual and wireless information are collected synchronously at each collection site by simulation or actual measurement, and the knowledge graph elements of the coexistence of visual and channel paths are constructed. The knowledge graph elements of the coexistence of visual and channel paths are stored in the form of a list, and each row of information includes visual features and corresponding wireless channel features.

[0077] Step 2: During the beam alignment process, real-time images within the base station service area are collected and visual features of the communication target are extracted;

[0078] Step 3: Use the visual feature domain retrieval method to roughly locate the visual features of the communication target in the visual information and channel information coexistence knowledge graph;

[0079] Specifically, the knowledge graph of coexistence of visual information and channel information is first segmented, specifically at the pixel level. In view of the characteristics of V2X wireless communication, the x-axis is selected as the segmentation axis, that is, the axis of parallel roads in the data set. The midpoint pixel of the X-axis is selected as the segmentation value to divide the visual channel path map into two subsets of visual feature information. In this way, the left child node contains all pixels on the segmentation axis that are less than or equal to the median. The right child node contains all pixels greater than the median. This operation is repeated recursively for the corresponding subsets until the leaf node is reached; the coarse positioning of the visual features is achieved.

[0080] The visual channel path graph is segmented at the pixel level to obtain multiple first-level sub-knowledge graphs, and then the first-level sub-knowledge graph is segmented to obtain the second-level sub-knowledge graph of each first-level sub-knowledge graph, and so on to obtain n-level sub-knowledge graphs; each retrieval first determines that the communication target belongs to a certain level of sub-knowledge graph, and then continues to detect in the second-level sub-knowledge graph, and so on, until the final n-level sub-knowledge graph to which the communication target belongs is found.

[0081] Specifically, the core idea of ​​the domain-specific retrieval of visual features of communication targets is to recursively divide the data set into subsets until a certain stopping condition is met (such as the subset size is less than a given threshold).

[0082] Build phase:

[0083] Select the visual feature center point (geometric center) of the sub-knowledge graph. For a node point set X, select the centroid C of the point set as the sub-domain center:

[0084]

[0085] Where n is the number of points contained in the node, x i is the point in the node.

[0086] Calculate the radius r of the visual feature subdomain of the sub-knowledge graph so that the subdomain can contain all the points in the node:

[0087]

[0088] Where ||xC|| is the distance from point x to the center of the sphere C.

[0089] Recursively partition the point set until each node contains fewer points than a preset threshold. A common partitioning method is to find the maximum distance between two points in the data set, then divide the point set into two parts and recursively create child nodes for each part.

[0090] Search phase:

[0091] Given a query point q, we search for the nearest neighbors by traversing the visual channel knowledge graph. The key steps of the search process are as follows:

[0092] For each node, calculate the distance between the query point q and the center C of the sphere:

[0093] d(q,c)=||qc|| (4)

[0094] Use the triangle inequality to speed up the search:

[0095] d(q,x i )≥d(q,c)-r (5)

[0096] Among them, d(q,x i ) is the query point and a point x in the sphere i , d(q,c) is the distance between the query point and the center of the sphere, and r is the radius of the sphere. According to this inequality, spheres far from the query point can be excluded to reduce the amount of calculation.

[0097] Step 4: Use the threshold weighted K nearest neighbor algorithm to predict the wireless channel path information between the communication target and the base station to obtain the optimal wireless channel path information;

[0098] The optimal wireless channel path prediction process performs distance measurement calculation on the real-time collected visual features and the visual features in the knowledge graph to obtain the nearest K candidate distances, where K is a preset value; the obtained K candidate distances are compared with the preset threshold, and the distance points that do not meet the threshold requirements are eliminated, and finally n candidate visual feature points are obtained, 0<n≤K; finally, the weighted average method is used to predict the final channel information.

[0099] Specifically, the optimal wireless channel path prediction process uses the Euclidean distance as a metric for predicting the visual features of the communication target and calculates the distance of the center coordinates.

[0100]

[0101] Among them, L x,y is the image feature descriptor of the query image, is the jth image feature descriptor in the image corresponding to the i-th camera in the visual channel knowledge graph.

[0102] Further, for the candidate objects, some candidate visual feature vectors with K closest distances are calculated, and the total number of candidate visual feature vectors is K. K is a preset value. Further, a threshold screening strategy is added to remove abnormal values; specifically, specifically:

[0103] Calculate the distance: Calculate the distance between the query point q and each sample point x in the data set i The distance d(q,x i ):

[0104]

[0105] Apply distance threshold: Define a distance threshold τ. Remove points whose distance is greater than τ. This step can be achieved using the following formula:

[0106] If d(q,x i )≤τ, then keep the point x i (10)

[0107] Otherwise, remove the point.

[0108] Therefore, the candidate visual feature vector set is less than or equal to K. The candidate visual feature vector distance is expressed as (q=[1,2,3,...,N],N≤K). The weighted average method is used to predict the final wireless channel path information. The distance between each candidate visual feature vector and the query point is different. The closer the distance, the greater the impact. To reflect this, a weight is assigned to each candidate visual feature vector. This weight is usually inversely proportional to the distance from the candidate visual feature vector to the query point. The smaller the distance, the greater the weight. The weight is normalized to obtain the weight of each candidate visual feature vector, that is,

[0109]

[0110] The predicted wireless channel path information is the weighted sum of the candidate visual feature vector channel information:

[0111]

[0112] P ij is the channel information of the ith candidate visual feature vector. Finally, the predicted channel knowledge of the input image is obtained, thereby obtaining the optimal wireless channel path information.

[0113] Step 5: Determine the best beam pair based on the predicted best wireless channel path information, adjust the beamforming parameters, and perform beam alignment control on the array antenna.

[0114] Implementation Cases

[0115] Step 1. Use simulation to establish a knowledge graph of the coexistence of visual information and channel information, use the autonomous driving simulation platform CARLA to simulate the environment and vehicles, use the traffic simulation software Simulation of Urban MObility, control the speed and movement trajectory of all vehicles through the joint simulation interface of CARLA, and establish a 100-meter-long and 40-meter-wide vehicle network wireless communication scenario with fixed objects such as tall buildings and trees on the roadside. Use the ray tracing software Wireless Insite to simulate the communication channel. The channel generated by the ray tracing technology has consistent spatial correlation with the communication environment established in the early stage.

[0116] Step 2: Any image I(t) taken by the base station has a size of 1280×720, and is downsampled to generate a two-dimensional feature map vector H o and W o The mathematical relationship with the input image is expressed as:

[0117]

[0118] where stride H and stride W is the downsampling step size, which is set to 640 and 360 respectively for example, H o and W o It can be imagined as the length and width of a grid, and each grid becomes a data collection site. In this example, H o =2,W o =2.

[0119] Based on steps 1 and 2, visual and wireless information are collected synchronously at each collection site to build a knowledge graph of coexistence of visual information and channel information.

[0120] Step 3: During the visual feature extraction process, the image sequence captured by each camera is

[0121] I x = {I x,1 ,I x,2 ,I x,3 ,...,I x,n}, requiring the camera group’s field of view to cover the entire street. In the visual feature extraction phase, we use the YOLO algorithm to predict the visual features of the target vehicle. The deep neural network predicts five indicators (L IOU ,L x ,L y ,W,H), where L IOU represents the confidence of the bounding box, (L x ,L y ) represents the offset of the center point of the bounding box compared to the upper left corner of the grid, (W,H) represents the width and height of the bounding box. In this example, the above five indicators are used as visual features.

[0122] Step 4: During the wireless feature extraction process, each collection site contains the corresponding wireless channel information. The visual channel path map includes the path gain, phase, departure angle and arrival angle of the path. In this example, the above four indicators are used as wireless features.

[0123] Step 5: Store the visual information and channel information coexistence knowledge graph in the form of a list, where each row of information includes visual features and corresponding wireless channel path features.

[0124] Step 6: Segment the knowledge graph where visual information and channel information coexist at the pixel level. Figure 3 As shown in the figure, in view of the characteristics of V2X wireless communication, we select the x-axis as the segmentation axis, which is the axis of parallel roads in the data set. The midpoint pixel of the x-axis is selected as the segmentation value to divide the visual channel path map into two subsets of visual feature information. In this way, the left child node contains all pixels less than or equal to the median on the segmentation axis. The right child node contains all pixels greater than the median. This operation is repeated recursively for the corresponding subsets until the leaf node is reached; the coarse positioning of the visual fingerprint is achieved.

[0125] Step 7: The best wireless channel prediction process, such as Figure 4 As shown, it mainly includes initializing K value and distance threshold X; real-time acquisition of visual feature vector L x,y ; Traverse the visual feature vector of the knowledge graph where visual information and channel information coexist For the prediction of the correspondence target visual features, Euclidean distance is used as the metric to calculate the distance of the visual features.

[0126]

[0127] Among them, L x,y is the image feature visual feature vector descriptor of the query image, is the jth visual feature vector descriptor in the image corresponding to the i-th camera in the visual channel knowledge map.

[0128] Furthermore, for the candidate objects, some candidate visual feature vectors with K closest distances are calculated, and the total number of candidate visual feature vectors is K. In the example, K is set to 3, and further, a threshold screening strategy is added to remove outliers. Therefore, the set of candidate visual feature vectors is less than or equal to 3. The candidate visual feature vector distance is expressed as (q=[1,...,k],k≤3). The weighted average method is used to predict the final channel information. The distance between each candidate visual feature and the query point is different. The closer the distance, the greater the impact. To reflect this, a weight is assigned to each candidate visual feature vector. This weight is usually inversely proportional to the distance from the candidate visual feature vector to the query point. The smaller the distance, the greater the weight. The weight is normalized to obtain the weight of each candidate visual feature vector, that is,

[0129]

[0130] The predicted channel information is the weighted sum of the channel information of the candidate visual feature vectors:

[0131]

[0132] P ij is the channel information of the i-th candidate visual feature vector; finally, the predicted channel knowledge of the input image is obtained. The scene graph of the beam alignment is as follows Figure 2 shown.

[0133] Step 8: Determine the best beam pair based on the predicted best wireless channel path information, adjust the beamforming parameters, and perform beam alignment control on the array antenna.

[0134] The present invention also provides a vision-assisted vehicle network millimeter wave wireless communication beam alignment method, including a processor and a memory; the memory is used to store programs; the processor executes the program to implement any of the above methods.

[0135] This paper proposes a vision-assisted millimeter-wave wireless communication beam alignment method for Internet of Vehicles. The main idea is to first synchronously collect visual and radio frequency information of the base station service area in the offline stage when deploying visual sensors in the Internet of Vehicles wireless communication system. Then, channel features and visual features are extracted through deep neural networks, and a fingerprint library combining communication and visual features is established for each potential communication point in the base station coverage area, thereby generating a visual channel path knowledge graph.

[0136] In the actual operation of beam alignment, the base station first collects images of the service area in real time and extracts the visual features of the communication target. Then, the visual feature domain retrieval method is used to preliminarily locate the target in the visual channel path map. Next, the channel path information between the communication target and the base station is predicted by the threshold weighted K nearest neighbor algorithm. Finally, the best beam pair is determined, and the beam alignment parameters are adjusted to control the antenna array to achieve beamforming control and ensure the accuracy and efficiency of communication.

[0137] The vision-assisted beam alignment method for the Internet of Vehicles proposed in the present invention can be applied to wireless communications in high-frequency bands such as millimeter waves and terahertz, providing a novel and efficient solution for beam alignment of ultra-large-scale array antenna wireless communications, and providing an advanced technical method for strengthening the data transmission guarantee of the Internet of Vehicles in urban environments.

[0138] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0142] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0144] Specific embodiments are used in this article to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for general technicians in this field, according to the ideas of this article, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this article.

Claims

1. A vision-assisted vehicle networking millimeter wave wireless communication beam alignment method, characterized in that: The following steps are involved: Step 1: Configure visual sensors at the base station to build a knowledge graph of coexistence of visual information and channel information; Step 2: During the beam alignment process, real-time images within the base station service area are collected and visual features of the communication target are extracted; Step 3: Use the visual feature domain retrieval method to roughly locate the visual features of the communication target in the visual information and channel information coexistence knowledge graph; Step 4: Use the threshold weighted K nearest neighbor algorithm to predict the wireless channel path information between the communication target and the base station to obtain the optimal wireless channel path information; Step 5: Determine the best beam pair based on the predicted best wireless channel path information, adjust the beamforming parameters, and perform beam alignment control on the array antenna.

2. The vision-assisted vehicle networking millimeter wave wireless communication beam alignment method according to claim 1 is characterized in that: The configuring of a visual sensor at the base station to construct a knowledge graph of coexistence of visual information and channel information includes: 1.

1. Configure visual sensors for each base station to simultaneously collect visual information and wireless radio frequency information within the base station service area; 1.

2. Extracting wireless channel features and visual features from the visual information and wireless radio frequency information; 1.

3. Establish a feature fingerprint of wireless communication and visual coexistence for each potential communication site in the base station server; 1.

4. Construct a knowledge graph of coexistence of visual information and channel information based on the feature fingerprints of each communication site.

3. The vision-assisted vehicle networking millimeter wave wireless communication beam alignment method according to claim 2 is characterized in that: The synchronous collection of visual information and wireless radio frequency information within the base station service area includes: The images collected by the base station visual sensor in the connected vehicle environment are segmented, and each segmented grid becomes a data collection site; the visual information and wireless radio frequency information are synchronously collected at each collection site by simulation or actual measurement; Any image I(t) taken by a visual sensor in the base station service area has a size of W I ×H I , after downsampling, a two-dimensional feature map vector is generated H o and W o The mathematical relationship with the input image is expressed as: where stride H and stride W is the downsampling step size, H o and W o is the length and width of a grid, and each grid becomes a data collection site; collecting visual information and wireless radio frequency information.

4. The vision-assisted vehicle networking millimeter wave wireless communication beam alignment method according to claim 2, characterized in that: The visual feature extraction includes extracting 5 indicators (L IOU ,L x ,L y ,W,H), where L IOU represents the confidence of the bounding box, (L x ,L y ) represents the offset of the center point of the bounding box compared to the upper left corner of the grid, and (W,H) represents the width and height of the bounding box.

5. The vision-assisted vehicle networking millimeter wave wireless communication beam alignment method according to claim 2, characterized in that: The wireless channel feature extraction includes extracting the path gain, phase, departure angle and arrival angle of the wireless channel path.

6. The vision-assisted vehicle networking millimeter wave wireless communication beam alignment method according to claim 1, characterized in that: The method of using visual feature domain retrieval to roughly locate the visual features of the communication target in the visual information and channel information coexistence knowledge graph includes: The visual channel path graph is segmented at the pixel level to obtain multiple first-level sub-knowledge graphs, and then the first-level sub-knowledge graph is segmented to obtain the second-level sub-knowledge graph of each first-level sub-knowledge graph, and so on to obtain n-level sub-knowledge graphs; each retrieval first determines that the communication target belongs to a certain level of sub-knowledge graph, and then continues to detect in the second-level sub-knowledge graph, and so on, until the final n-level sub-knowledge graph to which the communication target belongs is found.

7. The vision-assisted vehicle networking millimeter wave wireless communication beam alignment method according to claim 6 is characterized in that: The core idea of ​​the domain retrieval method using visual features is to recursively divide the data set into subsets until a certain stopping condition is met; wherein: Build phase: Select the visual feature center point of the sub-knowledge graph. For a node point set X, select the centroid C of the point set as the sub-domain center: Where n is the number of points contained in the node, x i is the point in the node; Calculate the radius r of the visual feature subdomain of the sub-knowledge graph so that the subdomain can contain all the points in the node: Where ||xC|| is the distance from point x to the center of the sphere C; Recursively partition the point set until each node contains fewer points than a preset threshold; the partitioning method is to find the maximum distance between two points in the data set, then divide the point set into two parts, and recursively create child nodes for each part; Search phase: Given a query point q, the nearest neighbors are found by traversing the visual channel knowledge graph; the steps of the search process are as follows: For each node, calculate the distance between the query point q and the center C of the sphere: d(q,c)=||qc|| (4) Use the triangle inequality to speed up the search: d(q,x i )≥d(q,c)-r(5) Among them, d(q,x i ) is the query point and a point x in the sphere i , d(q,c) is the distance between the query point and the center of the sphere, and r is the radius of the sphere.

8. The vision-assisted vehicle networking millimeter wave wireless communication beam alignment method according to claim 1, characterized in that: The optimal wireless channel path prediction process uses the Euclidean distance as a metric for predicting the visual features of the communication target and calculates the distance of the center coordinates; specifically: Among them, L x,y is the image feature descriptor of the query image, is the jth image feature descriptor in the image corresponding to the i-th camera in the visual channel knowledge graph; For the candidate objects, the candidate visual feature vectors with the K closest distances are calculated. The total number of candidate visual feature vectors is K. K is a preset value. A threshold screening strategy is added to remove abnormal values. Therefore, the set of candidate visual feature vectors is less than or equal to K. The distance between candidate visual feature vectors is expressed as The weighted average method is used to predict the final channel information; the distance between each candidate visual feature vector and the query point is different, and the closer the distance, the greater the impact; therefore, a weight is assigned to each candidate visual feature vector, and this weight is inversely proportional to the distance from the candidate visual feature vector to the query point. The smaller the distance, the greater the weight; the weight is normalized to obtain the weight of each candidate visual feature vector, that is, The predicted channel information is the weighted sum of the channel information of the candidate visual feature vectors: P ij is the channel information of the i-th candidate visual feature vector; finally, the wireless channel path information of the input image is obtained.

9. A vision-assisted millimeter-wave wireless communication beam alignment device for Internet of Vehicles, characterized in that: It comprises a processor and a memory; the memory is used to store a program; the processor executes the program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images