A millimeter wave vehicle networking beam prediction method and device, and equipment

Abstract

The embodiment of the application provides a millimeter wave vehicle networking beam prediction method and device, and equipment, and belongs to the technical field of vehicle networking communication. The method comprises the following steps: obtaining a target vehicle networking beam prediction model, the target vehicle networking beam prediction model comprising a target view feature extraction network, a target text encoding network and a target beam prediction network; based on the target view feature extraction network, performing visual patch direct injection on a roadside base station visual image sequence to obtain view token features; based on the target text encoding network, performing text encoding on a millimeter wave beam prediction task constraint to obtain text token features; based on the target beam prediction network, performing multi-modal feature fusion on the view token features and the text token features to obtain double-modal fusion features; and based on the target beam prediction network and the double-modal fusion features, performing beam index prediction to obtain a target predicted beam. The embodiment of the application can improve the accuracy of millimeter wave vehicle networking beam prediction.

Description

Technical Field

[0001] This application relates to the field of vehicle-to-everything (V2X) communication technology, and in particular to a millimeter-wave V2X beam prediction method, apparatus, and device. Background Technology

[0002] Millimeter-wave vehicle-to-everything (V2X) beam prediction methods are used to analyze the roadside communication environment and requirements of millimeter-wave V2X networks in order to predict suitable millimeter-wave communication beams. For example, in V2X communication fields such as highways and urban arterial roads, millimeter-wave V2X beam prediction can reduce the transmission latency of V2X communication links or improve the transmission efficiency of millimeter-wave communication data.

[0003] Currently, common millimeter-wave vehicle-to-everything (V2X) beam prediction methods involve first having a target detector frame the vehicle from the image captured by a roadside base station to obtain the vehicle's coordinates. These coordinates are then used as prompts to input into a large language model, which infers the optimal beam. However, this method compresses the complete image into coordinates, losing crucial information such as building reflectors and obstacle locations, resulting in significant information loss. Consequently, the large language model's beam index prediction is not accurate enough. Therefore, improving the accuracy of millimeter-wave V2X beam prediction has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to propose a millimeter-wave vehicle-to-everything (V2X) beam prediction method, apparatus, and device, aiming to improve the accuracy of millimeter-wave V2X beam prediction.

[0005] To achieve the above objectives, a first aspect of this application proposes a millimeter-wave vehicle-to-everything (V2X) beam prediction method, the method comprising: Obtain visual image sequences from roadside base stations and millimeter-wave beam prediction task constraints; Obtain a target vehicle network beam prediction model, wherein the target vehicle network beam prediction model includes a target view feature extraction network, a target text encoding network, and a target beam prediction network; Based on the target view feature extraction network, visual patching is directly projected onto the visual image sequence of the roadside base station to obtain view token features; Based on the target text encoding network, the constraints of the millimeter-wave beam prediction task are text encoded to obtain text token features; Based on the target beam prediction network, multimodal feature fusion is performed on the view token feature and the text token feature to obtain dual-modal fused features; Based on the target beam prediction network and the dual-modal fusion features, beam index prediction is performed to obtain the target predicted beam.

[0006] In some embodiments, the target view feature extraction network includes a visual patch segmentation unit, a visual encoding unit, and a projection unit; The method of visually patching the visual image sequence of the roadside base station based on the target view feature extraction network to obtain view token features includes: Based on the visual patch segmentation unit, the visual image sequence of the roadside base station is segmented to obtain a visual image patch set of the roadside base station. Based on the visual coding unit, visual features are extracted from the visual image patch set of the roadside base station to obtain the patch view feature set; Based on the projection unit, feature projection is performed on the patch view feature set to obtain the view token feature.

[0007] In some embodiments, the step of performing view segmentation on the roadside base station visual image sequence based on the visual patch segmentation unit to obtain a roadside base station visual image patch set includes: Based on the visual patch segmentation unit and the preset patch segmentation strategy, mesh division is performed to obtain visual patch parameters; Based on the visual patch segmentation unit and the visual patch parameters, the roadside base station visual image sequence is segmented to obtain multiple independent roadside base station visual image patches. Based on the visual patch segmentation unit, the multiple independent roadside base station visual image patches are integrated in an orderly manner to obtain the roadside base station visual image patch set.

[0008] In some embodiments, the step of projecting features onto the patch view feature set based on the projection unit to obtain the view token features includes: Based on the projection unit, the patch view feature set is projected into the linguistic domain to obtain the view semantic projection feature set; Based on the projection unit, the view semantic projection feature set is sorted to obtain the view token feature.

[0009] In some embodiments, the target beam prediction network includes a multi-head attention unit and a feature fusion unit; The target beam prediction network is used to perform multimodal feature fusion on the view token features and the text token features to obtain dual-modal fused features, including: Based on the multi-head attention unit, semantic association mining is performed on the view token features and the text token features to obtain semantic association features; Based on the feature fusion unit, feature interaction is performed on the semantic association features to obtain the dual-modal fusion features.

[0010] In some embodiments, the step of performing feature interaction on the semantic association features based on the feature fusion unit to obtain the dual-modal fusion features includes: Based on the feature fusion unit, the semantic association features are semantically coupled and enhanced to obtain deep fusion features; Based on the feature fusion unit, the deep fusion features are normalized to obtain the dual-modal fusion features.

[0011] In some embodiments, after performing beam index prediction based on the target beam prediction network and the dual-modal fusion features to obtain the target predicted beam, the method further includes: Based on the target vehicle-to-everything (V2X) beam prediction model, the target predicted beam is vectorized to obtain the beam index feature vector. The beam index feature vector and the dual-mode fusion feature are fused to obtain the temporal enhancement fusion feature; Based on the aforementioned temporal enhancement fusion features, beam index prediction is performed to obtain subsequent predicted beams.

[0012] To achieve the above objectives, a second aspect of this application provides a millimeter-wave vehicle-to-everything (V2X) beam prediction device, the device comprising: The basic information acquisition module is used to acquire visual image sequences of roadside base stations and millimeter-wave beam prediction task constraints; The target model acquisition module is used to acquire the target vehicle network beam prediction model, wherein the target vehicle network beam prediction model includes a target view feature extraction network, a target text encoding network, and a target beam prediction network; The visual image parsing module is used to perform visual patching on the visual image sequence of the roadside base station based on the target view feature extraction network to obtain view token features; The constraint text encoding module is used to perform text encoding on the constraints of the millimeter-wave beam prediction task based on the target text encoding network to obtain text token features; The dual-modal feature fusion module is used to perform multimodal feature fusion on the view token feature and the text token feature based on the target beam prediction network to obtain dual-modal fused features; The beam index prediction module is used to perform beam index prediction based on the target beam prediction network and the dual-modal fusion features to obtain the target predicted beam.

[0013] To achieve the above objectives, a third aspect of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method of the first aspect described above.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect described above.

[0015] The millimeter-wave vehicle-to-everything (V2X) beam prediction method, apparatus, electronic device, and storage medium proposed in this application provide comprehensive and realistic data for beam prediction in millimeter-wave V2X by acquiring visual image sequences from roadside base stations and millimeter-wave beam prediction task constraints. Then, based on the target view feature extraction network in the target V2X beam prediction model, visual patches are directly projected onto the roadside base station visual image sequences, ensuring the complete preservation of environmental context information such as building geometric features and road layout, thereby improving the beam prediction accuracy of the target V2X beam prediction model. Finally, based on the target text encoding network in the target V2X beam prediction model, text encoding is applied to the millimeter-wave beam prediction task constraints. The text token feature is encoded to achieve standardized encoding of beam prediction task constraints. This allows task constraint information to be adapted to the view token feature in a unified format, ensuring that the target vehicular network beam prediction model can understand the information mentioned in the millimeter-wave beam prediction task constraints. As a result, the text token feature can effectively guide beam index prediction, reducing the beam prediction bias of the target vehicular network beam prediction model. Finally, beam index prediction is performed through multimodal fusion of view token feature and text token feature, breaking the limitations of single-modal data. This allows the obtained target predicted beam to simultaneously combine roadside visual scene information and beam prediction task constraints, improving the accuracy of millimeter-wave vehicular network beam prediction. Attached Figure Description

[0016] Figure 1 This is a flowchart of the millimeter-wave vehicle-to-everything (V2X) beam prediction method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S103 in the process; Figure 3 yes Figure 2 The flowchart of step S201 in the text; Figure 4 yes Figure 2 The flowchart of step S203 in the process; Figure 5 yes Figure 1 The flowchart of step S105 in the process; Figure 6 yes Figure 5 The flowchart of step S502 in the document; Figure 7 This is a flowchart of a millimeter-wave vehicle-to-everything (V2X) beam prediction method provided in another embodiment of this application; Figure 8 This is a schematic diagram of the structure of the millimeter-wave vehicle-to-everything (V2X) beam prediction device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] First, let's analyze some of the terms used in this application: Visual patches: In millimeter-wave vehicle-to-everything (V2X) beam prediction technology, visual patches are independent image regions with fixed dimensions that carry local visual information of the roadside environment. They are obtained by dividing each frame of the visual image sequence of the roadside base station into a grid according to a preset strategy. Visual patches are the basic processing unit for fine-grained visual feature extraction of the original visual image. They are used in visual image processing of various roadside communication scenarios such as highways and urban main road intersections. Each visual patch corresponds independently to a specific local area in the original image and can independently carry the local semantic and spatial information of the roadside environment such as vehicles, roads, obstacles, and green belts in that area. Moreover, each patch can completely cover a single frame of the original image after being segmented according to rules, without overlap or omission. This provides a standardized processing object for the subsequent visual coding unit to perform feature extraction on local areas, making the extraction of roadside visual features more targeted and better preserving the detailed information of the original visual image.

[0021] Low-rank adaptation matrix: The low-rank adaptation matrix is ​​a low-rank structure matrix in the LoRA (Low-Rank Adaptation) fine-tuning strategy, specifically designed to adapt to millimeter-wave vehicle-to-everything (V2X) beam prediction scenarios and embedded in the attention layer of the visual language model. The rank of the low-rank adaptation matrix is ​​much lower than the rank of the original weight matrix of the visual language model's attention layer. It is a core component in building the LoRA fine-tuning module. This matrix is ​​used when conducting lightweight micro-surveys on millimeter-wave V2X scenarios using a pre-trained visual language model. The low-rank adaptation matrix does not need to change the original basic parameters of the model. It only uses a small number of its own trainable parameters to complete the adaptation update of the model in a specific scenario. This can significantly reduce the amount of computation and the scale of parameter training during the fine-tuning process, reduce the hardware resource consumption of model fine-tuning, and effectively avoid the overfitting problem when the model is trained on the millimeter-wave V2X test dataset. It allows the pre-trained visual language model to quickly adapt to the specific scenario requirements of millimeter-wave V2X beam prediction, improving the accuracy and adaptability of beam prediction in this scenario.

[0022] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0023] Millimeter-wave communication, with its advantages of high bandwidth, high transmission rate, and low transmission latency, has become a core supporting technology for achieving high-speed, low-latency wireless communication in vehicle-to-everything (V2X) communication. It is widely used in intelligent connected vehicle communication scenarios such as highways and urban arterial roads. Millimeter-wave signals have strong directional propagation characteristics, and the establishment and stability of communication links highly depend on accurate beam matching. However, the high-speed movement of vehicles and the dynamic changes in the roadside environment in V2X scenarios easily lead to beam mismatch and communication link interruptions. Therefore, millimeter-wave V2X beam prediction technology has become a key link in ensuring the stability of communication links and improving the quality of V2X communication. Its prediction accuracy and scenario adaptability directly determine the communication performance of millimeter-wave V2X.

[0024] Current millimeter-wave vehicle-to-everything (V2X) beam prediction methods still suffer from several technical shortcomings, failing to meet the high reliability and adaptability requirements of V2X communication. Firstly, most methods rely solely on single-modal information for beam prediction, or only utilize environmental information reflected in roadside visual images, or only refer to task constraints such as beam index range and prediction time step. This lack of effective multimodal information fusion results in biased predictions and low matching accuracy with actual communication scenarios. Secondly, even methods attempting to fuse visual and textual dual-modal information lack standardized feature processing procedures. Roadside base station visual image sequences lack professional patching, feature projection, and tokenization processing, and textual beam prediction task constraints are not properly encoded for multimodal fusion. The two types of features cannot achieve deep interaction due to differences in format and dimension. Thirdly, existing multimodal feature fusion processes lack effective dimensional calibration mechanisms and fail to explore the intrinsic relationships between visual and textual features specific to millimeter-wave V2X scenarios. In terms of semantic association, simple feature concatenation is often used, and the fused features cannot fully carry the core information of the dual-modality, resulting in insufficient feature representation capabilities and directly affecting the accuracy of beam index prediction. Fourth, existing methods are mostly limited to single-step beam prediction and do not utilize the temporal information of the preceding predicted beams for continuous beam prediction. This makes them unsuitable for dynamic scenarios involving vehicle movement and environmental changes, resulting in poor continuity of beam switching and easy communication link fluctuations. Fifth, some beam prediction methods based on visual language models directly use general pre-trained models for deployment without lightweight fine-tuning for the characteristics of millimeter-wave vehicle networking scenarios. General models have a large number of parameters, poor adaptability to roadside base stations with limited hardware resources, and high deployment and computation costs. If the model is trained with all parameters, it is very easy to overfit on the actual test dataset, reducing the actual prediction effect of the model.

[0025] The aforementioned problems result in significant deficiencies in the prediction accuracy, dynamic scene adaptability, and continuous prediction capabilities of existing millimeter-wave vehicle-to-everything (V2X) beam prediction methods. These shortcomings make it difficult to meet the high demands of V2X for millimeter-wave communication and hinder the large-scale deployment and application of millimeter-wave communication technology in the V2X field. Therefore, developing a millimeter-wave V2X beam prediction method capable of standardized processing and deep semantic fusion of multimodal information, supporting temporally continuous beam prediction, and adapting to roadside base station hardware deployment conditions has become an urgent technical challenge in this field.

[0026] Based on this, embodiments of this application provide a millimeter-wave vehicle-to-everything (V2X) beam prediction method and apparatus, electronic device and storage medium, aiming to improve the accuracy of millimeter-wave V2X beam prediction.

[0027] The millimeter-wave vehicle-to-everything (V2X) beam prediction method, device, electronic equipment, and storage medium provided in this application are specifically described through the following embodiments. First, the millimeter-wave V2X beam prediction method in this application embodiment is described.

[0028] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0029] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0030] The millimeter-wave vehicle-to-everything (V2X) beam prediction method provided in this application relates to the field of V2X communication technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the millimeter-wave V2X beam prediction method, but is not limited to the above forms.

[0031] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0032] Figure 1 This is an optional flowchart of the millimeter-wave vehicle-to-everything (V2X) beam prediction method provided in this application embodiment. The millimeter-wave V2X beam prediction method can be applied to millimeter-wave V2X roadside base stations. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0033] Step S101: Obtain the visual image sequence of the roadside base station and the millimeter-wave beam prediction task constraints; Step S102: Obtain the target vehicle network beam prediction model, wherein the target vehicle network beam prediction model includes a target view feature extraction network, a target text encoding network, and a target beam prediction network; Step S103: Based on the target view feature extraction network, visual patching is directly projected onto the visual image sequence of the roadside base station to obtain view token features; Step S104: Based on the target text encoding network, text encoding is performed on the millimeter-wave beam prediction task constraints to obtain text token features; Step S105: Based on the target beam prediction network, multimodal feature fusion is performed on the view token features and text token features to obtain dual-modal fused features; Step S106: Based on the target beam prediction network and dual-mode fusion features, perform beam index prediction to obtain the target predicted beam.

[0034] Steps S101 to S106 of this application embodiment, by acquiring the visual image sequence of the roadside base station and the millimeter-wave beam prediction task constraints, provide comprehensive and realistic data for beam prediction of millimeter-wave vehicle-to-everything (V2X) networks. Then, based on the target view feature extraction network in the target V2X beam prediction model, visual patching is directly projected onto the visual image sequence of the roadside base station, so that the environmental context information such as building geometric features and road layout in the visual information is completely preserved, thereby improving the beam prediction accuracy of the target V2X beam prediction model. Next, based on the target text encoding network in the target V2X beam prediction model, the millimeter-wave beam prediction task constraints are text encoded to obtain... By using text token features, standardized encoding of beam prediction task constraints is achieved, enabling task constraint information to be adapted to view token features in a unified format. This also ensures that the target vehicular network beam prediction model can understand the information mentioned in the millimeter-wave beam prediction task constraints, thus allowing text token features to effectively guide beam index prediction and reducing beam prediction bias in the target vehicular network beam prediction model. Finally, beam index prediction is performed through multimodal fusion of view token features and text token features, breaking the limitations of single-modal data. This allows the obtained target predicted beam to simultaneously combine roadside visual scene information and beam prediction task constraints, improving the accuracy of millimeter-wave vehicular network beam prediction.

[0035] In some application scenarios, taking the millimeter-wave vehicle-to-everything (V2X) communication scenario at the intersection of urban main roads as an example, millimeter-wave V2X roadside base stations are deployed at the intersection of core urban main roads. These roadside base stations can provide low-latency, high-stability millimeter-wave communication services to intelligent connected vehicles traveling at high speeds and changing lanes in the surrounding area using the methods described above. Specifically, the roadside base station can continuously collect scene information such as vehicle position, driving direction, road conditions, and surrounding obstacles within the area of ​​the urban main road intersection using its onboard high-definition image acquisition equipment, generating a visual image sequence of 30 frames per second. Simultaneously, combined with the urban main road intersection... For millimeter-wave communication requirements at arterial road intersections, pre-defined task constraints for beam prediction are established. These constraints include predicting the communication beam for one future millimeter-wave communication time slot, a legal beam index range of 1-32, and an adjacent beam index jump threshold not exceeding 4. These constraints then constitute the millimeter-wave beam prediction task constraints. Next, using the pre-built target vehicle-to-everything (V2X) beam prediction model from the millimeter-wave V2X roadside base station, visual patching is applied to the visual image sequences acquired by the aforementioned image acquisition equipment. This process transforms visual information such as vehicle positions and road layouts from the roadside base station visual image sequences into standardized view token features. Simultaneously, this... The target vehicle-to-everything (V2X) beam prediction model can also perform text encoding on the millimeter-wave beam prediction task constraints, such as predicting the communication beam for the next millimeter-wave communication time slot, the legal beam index range of 1-32, and the adjacent beam index jump threshold not exceeding 4. This results in text token features with the same dimension and format as the view token features. Furthermore, the target V2X beam prediction model merges the generated view token features and text token features to uncover the intrinsic relationship between vehicle movement trends in the view token features and beam prediction constraints in the text token features. This generates a model that simultaneously carries visual information about the roadside environment at the intersection and beam prediction task constraint information. The dual-modal fusion characteristics are then used as the core basis for the target vehicle-to-everything (V2X) beam prediction model. Combined with the beam codebook information of millimeter-wave communication, feature calculations are performed to obtain beam indices that meet the constraints of millimeter-wave beam prediction task, such as predicting a communication beam for one future millimeter-wave communication time slot, a legal beam index range of 1-32, and an adjacent beam index jump threshold of no more than 4. The beam index is then mapped to the actual millimeter-wave communication beam parameters to form the target prediction beam. Finally, the millimeter-wave V2X roadside base station can then use the target prediction beam to establish communication links with intelligent connected vehicles around the intersection of urban main roads.

[0036] In step S101 of some embodiments, the roadside base station visual image sequence refers to a sequence of continuous visual images collected by the roadside base station that can reflect the roadside environmental conditions. In the highway roadside communication scenario, the roadside base station visual image sequence can be a sequence of visual images captured by the roadside base station that includes the highway roadside environmental conditions.

[0037] Millimeter-wave beam prediction task constraints refer to the relevant conditions that limit the beam prediction results in millimeter-wave vehicle-to-everything (V2X) networks, such as beam prediction time steps, legal beam index range, and beam index jump threshold.

[0038] In this embodiment, an image acquisition device can be pre-built on the roadside base station of the millimeter-wave vehicle-to-everything (V2X) network. Then, when a vehicle passes by on the road, the image acquisition device is used to continuously acquire visual images of the traffic environment, surrounding facilities, and other scene information on the roadside. The acquired continuous images are then integrated in chronological order to obtain the visual image sequence of the roadside base station. It should be noted that before this, it is also necessary to pre-set constraints including beam prediction time steps, legal beam index range, and beam index jump restrictions, in combination with the beam communication requirements of the millimeter-wave V2X network. Furthermore, these constraints are integrated to obtain the millimeter-wave beam prediction task constraints.

[0039] In step S102 of some embodiments, the target vehicle-to-everything (V2X) beam prediction model refers to a dedicated model adapted to millimeter-wave V2X beam prediction scenarios, which can achieve multimodal feature processing and beam index prediction after training and optimization. In the roadside communication scenario at the intersection of urban main roads, the target V2X beam prediction model can be a dedicated beam prediction model trained with urban road measured datasets and integrating a multi-feature processing network.

[0040] The target view feature extraction network refers to the functional network in the target vehicle network beam prediction model that is specifically used to process the visual image sequence of the roadside base station. It should be noted that the target view feature extraction network usually includes a visual patch segmentation unit, a visual encoding unit, and a projection unit, which are used to perform visual patch segmentation, patch feature extraction, and feature projection on the visual image, respectively.

[0041] The target text encoding network refers to the functional network in the target vehicle-to-everything (V2X) beam prediction model that is specifically designed to perform text encoding of the constraints of the millimeter-wave beam prediction task.

[0042] The target beam prediction network refers to the functional network in the target vehicle network beam prediction model, which is used to realize multimodal feature fusion and beam index prediction based on the fused features. It should be noted that the target beam prediction network typically includes a dimension calibration unit for multimodal feature dimension alignment, a multi-head attention unit for multimodal feature semantic association mining, a feature fusion unit for multimodal feature interaction, and a beam index prediction unit for predicting the beam index of millimeter-wave vehicle networks.

[0043] In this embodiment, a basic model adapted to multimodal feature processing can be selected first. Then, combined with the actual scenario requirements of millimeter-wave vehicle-to-everything (V2X) beam prediction, the basic model is trained and optimized to obtain a target V2X beam prediction model specifically for millimeter-wave V2X scenarios. It should be noted that the target V2X beam prediction model integrates multiple functional networks such as a target view feature extraction network, a target text encoding network, and a target beam prediction network. Among them, the target view feature extraction network is responsible for visual feature processing, the target text encoding network is responsible for text feature processing, and the target beam prediction network is responsible for multimodal feature fusion and beam index prediction.

[0044] In this embodiment, a target vehicle-to-everything (V2X) beam prediction model can be obtained by training a pre-selected base model adapted for multimodal feature processing using millimeter-wave V2X roadside visual image sample data, millimeter-wave beam prediction task constraint samples, and actual millimeter-wave V2X beam index data. Specifically, the millimeter-wave V2X roadside visual image sample data and millimeter-wave beam prediction task constraint samples can be input into the base model. The base model can perform visual patching on the millimeter-wave V2X roadside visual image sample data to obtain view sample token features. At the same time, it can perform text encoding on the millimeter-wave beam prediction task constraint samples to obtain text sample token features. Furthermore, the base model can perform dual-modal sample fusion based on the fusion of view sample token features and text sample token features. The model first predicts the beam index of the next moment in the scenario, i.e., predicts the millimeter-wave vehicle-to-everything (V2X) beam index data. Then, the base model substitutes the predicted millimeter-wave V2X beam index data and the input actual millimeter-wave V2X beam index data into the cross-entropy loss function to calculate the beam index loss value, which represents the difference between the predicted and actual millimeter-wave V2X beam index data. Next, the beam index loss value is judged to meet the standard. If the beam index loss value does not meet the standard, the parameters of the base model are adjusted according to the beam index loss value, and a new round of millimeter-wave V2X beam prediction is performed using the base model with adjusted parameters until the beam index loss value meets the standard. If the beam index loss value meets the standard, the base model at this time can be used as the target V2X beam prediction model.

[0045] It is also important to know that the aforementioned basic model includes a pre-trained view feature extraction network, a text encoding network, and a beam prediction network. The pre-trained view feature extraction network has the ability to convert millimeter-wave vehicle-to-everything (V2X) roadside visual image sample data into view sample token features. The pre-trained view feature extraction network has the ability to convert millimeter-wave beam prediction task constraint samples into text sample token features. The pre-trained view feature extraction network has the ability to predict beam indices based on view sample token features and text sample token features.

[0046] In step S103 of some embodiments, the view token feature refers to a standardized visual feature that can be multimodally fused with other modal features (such as text token features).

[0047] In this embodiment of the application, the obtained roadside base station visual image sequence is input into the target vehicle network beam prediction model. The target view feature extraction network in the target vehicle network beam prediction model can perform a series of processes such as view segmentation, feature extraction and feature projection on the roadside base station visual image sequence of image form, until the text information in the millimeter wave beam prediction task constraint is transformed into a standardized feature form that can interact with the text token feature, namely the view token feature.

[0048] For details, please refer to Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S203: Step S201: Based on the visual patch segmentation unit, the visual image sequence of the roadside base station is segmented to obtain a visual image patch set of the roadside base station. Step S202: Based on the visual coding unit, visual features are extracted from the visual image patch set of the roadside base station to obtain the patch view feature set; Step S203: Based on the projection unit, perform feature projection on the patch view feature set to obtain the view token feature.

[0049] In step S201 of some embodiments, the roadside base station visual image patch set refers to a set formed by integrating several independent visual patches in a specific order. In the highway roadside communication scenario, when processing the continuous visual image sequence of the highway, the roadside base station visual image patch set can be a set of independent pixel patches formed by grid segmentation of the highway image and integrated in an orderly manner.

[0050] In this embodiment of the application, when the target view feature extraction network receives the visual image sequence of the roadside base station, it can divide each frame of the visual image sequence of the roadside base station into grids according to the patch segmentation strategy built into its internal visual patch segmentation unit. Then, according to the temporal order of each frame of the visual image sequence of the roadside base station and the spatial position order of the patches in a single frame, the independent visual patches after grid segmentation are orderly integrated to obtain the visual image patch set of the roadside base station.

[0051] For details, please refer to Figure 3 In some embodiments, step S201 may include, but is not limited to, steps S301 to S303: Step S301: Based on the visual patch segmentation unit and the preset patch segmentation strategy, perform mesh division to obtain visual patch parameters; Step S302: Based on the visual patch segmentation unit and visual patch parameters, the visual image sequence of the roadside base station is segmented to obtain multiple independent visual image patches of the roadside base station. Step S303: Based on the visual patch segmentation unit, multiple independent roadside base station visual image patches are systematically integrated to obtain a roadside base station visual image patch set.

[0052] In step S301 of some embodiments, the patch segmentation strategy refers to the unified segmentation criteria when segmenting view data. The patch segmentation strategy usually includes criteria such as visual patch size, grid division rules, and image segmentation method.

[0053] The visual patch parameters include specific segmentation parameters such as the spatial pixel coordinates, size, and division boundaries of each individual visual patch.

[0054] In this embodiment, before segmenting the roadside base station visual image sequence, the visual patch segmentation unit needs to load a patch segmentation strategy pre-configured for millimeter-wave vehicle-to-everything (V2X) roadside visual image processing. This patch segmentation strategy defines core rules such as the pixel size, whether they overlap, and the number of grid rows and columns for independent visual patches. Subsequently, based on the actual resolution of each frame in the roadside base station visual image sequence, global grid planning is performed on each frame according to the patch segmentation strategy. The spatial pixel coordinates, division boundaries, and size of each independent visual patch in each frame are calculated one by one. Finally, the aforementioned parameter information is integrated into a standardized parameter set to obtain visual patch parameters that can guide image segmentation.

[0055] In step S302 of some embodiments, an independent roadside base station visual image patch refers to a visual image fragment with independent local image region information. In the roadside communication scenario of suburban roads, when processing continuous visual images of suburban roads, the independent roadside base station visual image patch can be an independent image fragment containing roads, vehicles or green belts obtained by segmenting according to fixed parameters.

[0056] In this embodiment, the visual patch segmentation unit can use the generated visual patch parameters as the sole basis to perform a region-by-region image segmentation operation on each frame of the roadside base station visual image sequence. Specifically, the image segmentation operation includes accurately extracting the corresponding local image region from a single frame image according to the spatial pixel coordinates and size of each patch in the visual patch parameters, forming a single independent roadside base station visual image patch, i.e., an independent roadside base station visual image patch. Finally, after completing the segmentation of all grid regions of each frame image in sequence, multiple independent roadside base station visual image patches with a number matching the number of grids can be obtained.

[0057] In step S303 of some embodiments, after obtaining multiple independent roadside base station visual image patches, all the segmented independent roadside base station visual image patches can be arranged and integrated in an orderly manner according to the temporal order of each frame image in the roadside base station visual image sequence and the spatial position order of the independent roadside base station visual image patches in a single frame image, to obtain a roadside base station visual image patch set. It should be noted that after obtaining the roadside base station visual image patch set, it is necessary to ensure that the independent roadside base station visual image patches in the roadside base station visual image patch set completely retain the temporal information of the roadside base station visual image sequence and the spatial structure information of the single frame image.

[0058] Steps S301 to S303 as shown in the embodiments of this application first determine the visual patch parameters based on the preset patch segmentation strategy in the visual patch segmentation unit, so that the image segmentation process has a unified and clear execution basis, completely avoiding the arbitrariness of the segmentation process and reducing the risk of image information loss caused by arbitrary segmentation during the image segmentation process. Then, the roadside base station visual image sequence is segmented according to the visual patch parameters, which can ensure that the obtained multiple independent visual image patches are of uniform specifications, without overlap or omission, and can completely cover all image areas of the roadside base station visual image sequence. Finally, the multiple independent roadside base station visual image patches are orderly integrated according to frame time sequence and spatial position, so that the obtained roadside base station visual image patch set can completely retain the time sequence information of the roadside base station visual image sequence and the spatial structure information of the single frame image.

[0059] In step S202 of some embodiments, the patch view feature set refers to the set formed by integrating the visual features corresponding to each independent visual patch in the roadside base station visual image patch set in the corresponding order, wherein the visual features include the semantic features and spatial features of the independent visual patches.

[0060] In this embodiment, after obtaining the set of roadside base station visual image patches, the visual encoding unit in the target view feature extraction network performs convolution, pooling, and feature mapping on each independent roadside base station visual image patch in the set to obtain a patch view feature set. Specifically, the visual encoding unit includes structures such as convolutional layers, pooling layers, and feature mapping layers. Therefore, the visual encoding unit can first extract the low-level spatial features and high-level semantic features of each independent roadside base station visual image patch through convolutional and pooling layers, and then convert the extracted low-level spatial features and high-level semantic features into fixed-dimensional visual feature vectors through the feature mapping layer. Finally, according to the original order of each independent roadside base station visual image patch in the set, the visual feature vectors corresponding to all independent roadside base station visual image patches are orderly integrated to obtain the patch view feature set.

[0061] In step S203 of some embodiments, the projection unit in the target view feature extraction network sequentially maps each patch view feature in the patch view feature set from the visual feature space to the language feature space to obtain multiple projection features. Then, according to the order of each patch view feature in the patch view feature set, the multiple projection features are sorted and integrated to obtain the view token feature.

[0062] For details, please refer to Figure 4 In some embodiments, step S203 may include, but is not limited to, steps S401 to S402: Step S401: Based on the projection unit, perform linguistic domain projection on the patch view feature set to obtain the view semantic projection feature set; Step S402: Based on the projection unit, sort the view semantic projection feature set to obtain the view token feature.

[0063] In steps S401 and S402 of some embodiments, the view semantic projection feature set refers to the feature set formed by integrating the language domain semantic projection features corresponding to each patch view feature in the patch view feature set in the original order.

[0064] In this embodiment, after receiving the patch view feature set, the projection unit can first perform dimension calibration on each patch view feature in the patch view feature set sequentially to make the dimension of the patch view feature consistent with the dimension of the language domain feature, so as to facilitate the projection of the patch view feature onto the language domain. Then, each patch view feature after dimension calibration is multiplied with a pre-set linear transformation matrix to complete the cross-modal mapping of each patch view feature from the visual feature space to the language feature space, and obtain the view semantic projection feature set. Finally, the view semantic projection features in the view semantic projection feature set are ordered and integrated according to the original arrangement order of the features in the patch view feature set to obtain the view token feature.

[0065] Steps S401 to S402, as illustrated in this embodiment, involve projecting the patch view feature set into the linguistic domain using a projection unit. This achieves cross-spatial mapping of each patch view feature from the visual feature space to the linguistic feature space, while simultaneously completing the unified calibration of feature dimensions. This provides the projected visual features with the basic conditions for multimodal fusion with text features. Furthermore, the linguistic domain projection process fully preserves the visual semantic information of the roadside environment, ensuring the integrity of visual semantic information during multimodal fusion. Then, the projection unit sorts the view semantic projection feature set according to the spatiotemporal characteristics of each patch view feature in the patch view feature set. This ensures that the feature arrangement order is consistent with the spatiotemporal characteristics of the original roadside visual information in the roadside base station visual image sequence, eliminating numerical distribution deviations between features and making the feature form more standardized, thereby improving the beam prediction accuracy of millimeter-wave vehicle-to-everything (V2X) communication.

[0066] Steps S201 to S203 as illustrated in this embodiment first transform the roadside base station visual image sequence into a standardized roadside base station visual image patch set through a visual patch segmentation unit. This allows visual feature extraction to be performed on local image regions, improving the completeness of feature extraction. Then, a visual encoding unit extracts core visual features from the roadside base station visual image patch set and forms a patch view feature set, fully preserving the visual semantics and spatial information of the roadside environment. Finally, a projection unit projects each patch view feature in the patch view feature set onto the language domain, forming view token features. This realizes the transformation of patch view features into a standardized feature form adapted for multimodal fusion, ensuring the complete preservation of roadside visual information during processing and reducing the difficulty of multimodal feature fusion.

[0067] In step S104 of some embodiments, the text token feature refers to a standardized visual feature that can be multimodally fused with other modal features (such as view token features).

[0068] In this embodiment of the application, the millimeter-wave beam prediction task constraints obtained above are input into the target vehicle network beam prediction model. The target text encoding network in the target vehicle network beam prediction model can perform a series of encoding processes such as word segmentation, semantic feature extraction and feature standardization on the text form of the millimeter-wave beam prediction task constraints, until the text information in the millimeter-wave beam prediction task constraints is transformed into a standardized feature form that can interact with the view token feature, namely the text token feature.

[0069] In step S105 of some embodiments, the dual-modal fusion feature refers to the combined feature of visual semantic information in the fusion view token feature and textual semantic information in the text token feature.

[0070] In this embodiment of the application, after the target vehicle network beam prediction model obtains both the view token feature and the text token feature, it can use the target beam prediction network, which includes a multi-head attention unit and a feature fusion unit, to perform feature fusion processing on the view token feature and the text token feature, thereby obtaining a bimodal fused feature that integrates visual and textual bimodal semantic information.

[0071] For details, please refer to Figure 5 In some embodiments, step S105 may include, but is not limited to, steps S501 to S502: Step S501: Based on the multi-head attention unit, perform semantic association mining on the view token features and text token features to obtain semantic association features; Step S502: Based on the feature fusion unit, perform feature interaction on the semantic association features to obtain dual-modal fusion features.

[0072] In step S501 of some embodiments, semantic association features refer to features formed by splicing together two types of information that have semantic relationships: visual information in view token features and text information in text token features. In the roadside communication scenario at an intersection of urban main roads, when processing the view token features and text token features of the intersection, semantic association features can be features formed by splicing or fusing the visual environment information of the intersection in the view token features and the beam prediction constraint information in the text token features.

[0073] The target beam prediction network can utilize multiple independent attention heads in a multi-head attention unit to perform attention calculations on view token features and text token features from different semantic dimensions. This allows it to uncover the inherent semantic relationships between the roadside environment information contained in each visual feature of the view token features and the beam prediction constraint information contained in each text feature of the text token features. For example, there is an inherent semantic relationship between the vehicle movement trend information in the view token features and the beam prediction time step constraint in the text token features, and between the road scene information in the view token features and the legal beam index range constraint in the text token features. By concatenating and fusing the calculation results of each attention head, a feature vector that integrates multi-dimensional semantic relationship information can be obtained. Finally, these feature vectors are integrated together to form semantic relationship features.

[0074] In step S502 of some embodiments, after receiving the semantic association features output by the multi-head attention unit, the feature fusion unit in the target beam prediction network can perform dimensional feature fusion and information complementation on the visual and text association features in the semantic association features to strengthen the related semantic information in the two types of features and weaken irrelevant redundant information, thereby obtaining deeply fused visual and text association features. Then, by performing feature normalization on the deeply fused visual and text association features, the visual and text association features that have completed the interaction processing can be integrated into a feature vector with complete semantic information in both modalities, namely, a dual-modal fused feature.

[0075] For details, please refer to Figure 6 In some embodiments, step S502 may include, but is not limited to, steps S601 to S602: Step S601: Based on the feature fusion unit, semantic coupling enhancement is performed on the semantic association features to obtain deep fusion features; Step S602: Based on the feature fusion unit, the deep fusion features are normalized to obtain dual-modal fusion features.

[0076] In step S601 of some embodiments, deep fusion features refer to fusion features that achieve deep binding and highly complementary information through bimodal semantics of view token features and text token features.

[0077] It is important to know that the feature fusion unit incorporates semantic coupling rules, which match the association between view token features and text token features in millimeter-wave vehicle networking scenarios. Therefore, in this embodiment, the feature fusion unit can utilize the aforementioned semantic coupling rules to perform in-depth mining and binding of the semantic association between visual and text modal features in a dimension-by-dimensional manner. This can strengthen the effective semantic information of highly correlated features between the two types of modal features in the semantic association features, while weakening the redundant semantic information of unrelated features between the two types of modal features in the semantic association features. This achieves semantic coupling enhancement of the semantic association features, and then integrates the semantic association features that have been coupled and enhanced into deep fusion features.

[0078] In step S602 of some embodiments, the feature fusion unit can select a normalization method adapted to millimeter-wave vehicle networking feature processing, calculate and process the values ​​of each dimension in the deep fusion feature to eliminate the numerical distribution deviation between different dimensions, and then uniformly map the calculated feature values ​​of each dimension to a preset reasonable range to complete the normalization processing of the deep fusion feature, thereby obtaining a dual-modal fusion feature that can be directly used for beam index prediction.

[0079] Steps S601 to S602 as shown in the embodiments of this application enhance the semantic coupling of semantically related features through the feature fusion unit, enabling high complementarity of dual-modal information. This makes the obtained deep fusion features more aligned with the actual needs of millimeter-wave vehicle-to-everything (V2X) beam prediction, improving the semantic representation capability of the deep fusion features. Furthermore, the feature fusion unit normalizes the deep fusion features, eliminating distribution biases and scale differences in feature values, unifying feature data standards, suppressing the influence of outliers on features, and improving the stability of the dual-modal fusion features, thereby enhancing the accuracy of millimeter-wave V2X beam prediction.

[0080] Steps S501 to S502, as illustrated in this embodiment, utilize a multi-head attention unit to mine the inherent semantic association between view token features and text token features from multiple dimensions. This allows for a deep connection between the visual roadside environment information in the view token features and the text beam prediction constraint information in the text token features, avoiding simple splicing of bimodal features. Furthermore, by performing deep interactive processing on the semantically associated features through a feature fusion unit, information complementarity and enhancement of visual and text features in the semantically associated features can be achieved, while fully preserving the core semantic information of the bimodal view token features and text token features. This effectively improves the representational capability of the bimodal fusion features, providing a high-quality feature foundation for beam index prediction, thereby significantly improving the accuracy of beam prediction.

[0081] In step S106 of some embodiments, the target predicted beam refers to the final predicted beam that meets the requirements of millimeter-wave vehicle network communication, predicted by the target vehicle network beam prediction model. In the highway roadside communication scenario, the target predicted beam can be a millimeter-wave communication beam adapted to highway mobile vehicle communication, obtained after beam indexing calculation and verification.

[0082] In this embodiment of the application, after obtaining the dual-mode fusion features, the target beam prediction network can perform feature operations (such as linear transformation or nonlinear transformation) and feature analysis on the dual-mode fusion features based on the beam codebook information built into the millimeter-wave vehicle network roadside base station. The corresponding beam index can be obtained, and then the beam index is mapped to the beam parameters that can actually be used in millimeter-wave vehicle network communication. After the legality of the beam parameters is verified in combination with the above-mentioned millimeter-wave beam prediction task constraints, the target predicted beam that meets the millimeter-wave beam prediction task constraints is finally obtained.

[0083] In this embodiment of the application, after the millimeter-wave vehicle-to-everything (V2X) roadside base station predicts the target prediction beam, it can use the target prediction beam to establish a communication link with the intelligent connected vehicles in the surrounding area, thereby realizing the rapid transmission of messages between the roadside base station and the vehicle.

[0084] It should be noted that, in one embodiment of this application, if the millimeter-wave beam prediction task constraint includes instruction information for predicting the beam index of the two subsequent time nodes after the current time node, then the target prediction beam of the first time node after the current time node, which has already been predicted, can be used as the beam index prediction input for the second subsequent time node after the current time node. Then, the beam index prediction for the second subsequent time node after the current time node is performed based on the beam index prediction input to obtain the subsequent prediction beam. In addition, it is also necessary to know that if the millimeter-wave beam prediction task constraint includes instruction information to predict the beam index of two or more subsequent time nodes after the current time node, the output of the previous time node corresponding to the beam prediction can be used as the input of the time node corresponding to the beam prediction. For example, when t represents the current time node, if the millimeter-wave beam prediction task constraint includes predicting the beam index of three time nodes t+1, t+2 and t+3 at once, the beam index of time node t+1 can be predicted according to the above scheme. When predicting the beam index of time node t+2, the beam index of time node t+1 is used as the input, and when predicting the beam index of time node t+3, the beam index of time node t+2 is used as the input.

[0085] For details, please refer to Figure 7 In other embodiments, the millimeter-wave vehicle-to-everything (V2X) beam prediction method may also include, but is not limited to, steps S701 to S703: Step S701: Based on the target vehicle network beam prediction model, the target predicted beam is vectorized to obtain the beam index feature vector. Step S702: Perform feature fusion on the beam index feature vector and the dual-mode fusion feature to obtain the temporal enhancement fusion feature; Step S703: Based on the temporal enhancement fusion features, perform beam index prediction to obtain the subsequent predicted beam.

[0086] In step S701 of some embodiments, the beam index feature vector refers to a standardized feature vector that can characterize the beam index information contained in the target predicted beam.

[0087] In this embodiment, the target vehicle network beam prediction model can first extract the beam index information corresponding to the target predicted beam, and then, based on the beam-to-vector conversion rules learned during the model training phase, map the beam index information corresponding to the target predicted beam into a continuous numerical feature vector. Subsequently, the continuous numerical feature vector is subjected to dimension calibration and standardization processing to ensure that its dimension and data distribution are consistent with the dual-modal fusion feature, thereby obtaining a beam index feature vector that can interact with the dual-modal fusion feature.

[0088] In step S702 of some embodiments, the temporal enhanced fusion feature refers to an enhanced fusion feature that fuses roadside vision, beam prediction constraints, and beam index information contained in the target prediction beam.

[0089] In this embodiment, the feature fusion unit of the target vehicle-to-everything (V2X) beam prediction model can be used to perform dimension-wise feature concatenation of the beam index feature vector and the aforementioned dual-modal fusion feature, forming a concatenated feature containing the dual-modal fusion feature's basic information and the preceding beam temporal information of the beam index feature vector. Then, semantic association enhancement is performed on this concatenated feature to uncover the intrinsic connection between the beam index information represented by the beam index feature vector and the roadside vision and beam prediction constraint information represented by the dual-modal fusion feature, and to enhance the semantic information of this intrinsic connection. Finally, feature compression and optimization are performed on the semantically enhanced concatenated feature to obtain the temporally enhanced fusion feature.

[0090] In step S703 of some embodiments, the subsequent predicted beam refers to the subsequent predicted beam after the target predicted beam. For example, in the roadside communication scenario of suburban roads, after obtaining the target predicted beam of the suburban road at time t, the subsequent predicted beam can be the millimeter-wave communication predicted beam at time t+1 in this scenario.

[0091] In this embodiment of the application, after obtaining the temporal enhancement fusion features, the target beam prediction network can perform feature operations (such as linear transformation or nonlinear transformation) and feature analysis on the temporal enhancement fusion features based on the beam codebook information built into the millimeter-wave vehicle network roadside base station, to obtain the subsequent beam index of the corresponding temporal enhancement fusion features, and then map the subsequent beam index to the subsequent beam parameters that can actually be used in millimeter-wave vehicle network communication. After verifying the legality of the subsequent beam parameters in combination with the above-mentioned millimeter-wave beam prediction task constraints, the subsequent predicted beam that meets the millimeter-wave beam prediction task constraints is finally obtained.

[0092] Steps S701 to S703 of this embodiment first convert the target predicted beam into a beam index feature vector, realizing the transformation of discrete beam index information into feature form. This allows the temporal information of the preceding beam represented by the target predicted beam to participate in subsequent feature interactions. Then, the obtained beam index feature vector is deeply fused with the original dual-modal fusion feature to obtain a temporal enhanced fusion feature containing temporal beam information, roadside visual information, and beam prediction constraint information. This compensates for the lack of temporal information in a single dual-modal fusion feature, making the basis for beam prediction more comprehensive. Finally, subsequent beam index prediction is performed based on the temporal enhanced fusion feature, which can make full use of the temporal correlation information of the preceding beam, so that the subsequent predicted beam can fit the actual scenario of vehicle movement and dynamic environmental changes in millimeter-wave vehicle-to-everything (V2X) networks. This achieves continuous temporal prediction of the beam, improves the continuity and accuracy of beam prediction, and enables roadside base stations to provide continuous and adapted millimeter-wave communication beams for moving vehicles, thereby improving the stability and continuity of the millimeter-wave V2X communication link.

[0093] It should also be noted that, in another embodiment of this application, if the target vehicle-to-everything (V2X) beam prediction model changes its application scenario, for example, from a highway scenario to an urban road scenario, the LoRA fine-tuning strategy can be used to update the parameters of the target V2X beam prediction model. Specifically, a low-rank adaptation matrix is ​​first inserted into the multi-head attention unit of the target V2X beam prediction model, and then a LoRA fine-tuning module specifically for millimeter-wave V2X scenarios is constructed based on the low-rank adaptation matrix. Then, the parameters of the LoRA fine-tuning module are specifically trained and updated using the collected visual image sequence of the roadside base station of the changed application scenario and the measured beam data, so that the target V2X beam prediction model can be adapted to the changed application scenario. This achieves lightweight fine-tuning of model parameters when changing application scenarios and also reduces the training cost of the model.

[0094] This application provides comprehensive and realistic data for beam prediction in millimeter-wave vehicular networks by acquiring visual image sequences from roadside base stations and millimeter-wave beam prediction task constraints. Then, based on the target view feature extraction network in the target vehicular network beam prediction model, visual patches are directly projected onto the roadside base station visual image sequences, ensuring the complete preservation of environmental context information such as building geometric features and road layout, thereby improving the beam prediction accuracy of the target vehicular network beam prediction model. Next, based on the target text encoding network in the target vehicular network beam prediction model, text encoding is performed on the millimeter-wave beam prediction task constraints to obtain text token features, thus achieving... The standardized encoding of beam prediction task constraints enables the task constraint information to be adapted to view token features in a unified format. This also ensures that the target vehicular network beam prediction model can understand the information mentioned in the millimeter-wave beam prediction task constraints, thereby enabling text token features to effectively guide beam index prediction and reducing the beam prediction bias of the target vehicular network beam prediction model. Finally, beam index prediction is performed through multimodal fusion of view token features and text token features, breaking the limitations of single-modal data. This allows the obtained target predicted beam to simultaneously combine roadside visual scene information and beam prediction task constraints, improving the accuracy of millimeter-wave vehicular network beam prediction.

[0095] Please see Figure 8 This application also provides a millimeter-wave vehicle-to-everything (V2X) beam prediction device, which can implement the above-described millimeter-wave V2X beam prediction method. The device includes: The basic information acquisition module 801 is used to acquire visual image sequences of roadside base stations and millimeter-wave beam prediction task constraints. The target model acquisition module 802 is used to acquire the target vehicle network beam prediction model, wherein the target vehicle network beam prediction model includes a target view feature extraction network, a target text encoding network, and a target beam prediction network; The visual image parsing module 803 is used to perform visual patching on the visual image sequence of the roadside base station based on the target view feature extraction network to obtain view token features. The constraint text encoding module 804 is used to encode the text of the millimeter-wave beam prediction task constraints based on the target text encoding network to obtain text token features. The dual-modal feature fusion module 805 is used to perform multimodal feature fusion on view token features and text token features based on the target beam prediction network to obtain dual-modal fused features; The beam index prediction module 806 is used to perform beam index prediction based on the target beam prediction network and dual-mode fusion features to obtain the target predicted beam.

[0096] The specific implementation of this millimeter-wave vehicle-to-everything (V2X) beam prediction device is basically the same as the specific implementation of the millimeter-wave V2X beam prediction method described above, and will not be repeated here.

[0097] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described millimeter-wave vehicle-to-everything (V2X) beam prediction method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0098] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 901 using the millimeter-wave vehicle networking beam prediction method of the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0099] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described millimeter-wave vehicle networking beam prediction method.

[0100] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0101] The millimeter-wave vehicle-to-everything (V2X) beam prediction method, device, electronic equipment, and storage medium provided in this application obtain a target V2X beam prediction model by acquiring a visual image sequence from a roadside base station and millimeter-wave beam prediction task constraints. The target V2X beam prediction model includes a target view feature extraction network, a target text encoding network, and a target beam prediction network. Based on the target view feature extraction network, visual patching is directly projected onto the roadside base station visual image sequence to obtain view token features. Based on the target text encoding network, text encoding is performed on the millimeter-wave beam prediction task constraints to obtain text token features. Based on the target beam prediction network, multimodal feature fusion is performed on the view token features and text token features to obtain dual-modal fusion features. Based on the target beam prediction network and the dual-modal fusion features, beam index prediction is performed to obtain the target predicted beam.

[0102] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0103] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0106] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0107] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0109] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A millimeter-wave vehicle-to-everything (V2X) beam prediction method, characterized in that, The method includes: Obtain visual image sequences from roadside base stations and millimeter-wave beam prediction task constraints; Obtain a target vehicle network beam prediction model, wherein the target vehicle network beam prediction model includes a target view feature extraction network, a target text encoding network, and a target beam prediction network; Based on the target view feature extraction network, visual patching is directly projected onto the visual image sequence of the roadside base station to obtain view token features; Based on the target text encoding network, the constraints of the millimeter-wave beam prediction task are text encoded to obtain text token features; Based on the target beam prediction network, multimodal feature fusion is performed on the view token feature and the text token feature to obtain dual-modal fused features; Based on the target beam prediction network and the dual-modal fusion features, beam index prediction is performed to obtain the target predicted beam.

2. The method according to claim 1, characterized in that, The target view feature extraction network includes a visual patch segmentation unit, a visual encoding unit, and a projection unit; The method of visually patching the visual image sequence of the roadside base station based on the target view feature extraction network to obtain view token features includes: Based on the visual patch segmentation unit, the visual image sequence of the roadside base station is segmented to obtain a visual image patch set of the roadside base station. Based on the visual coding unit, visual features are extracted from the visual image patch set of the roadside base station to obtain the patch view feature set; Based on the projection unit, feature projection is performed on the patch view feature set to obtain the view token feature.

3. The method according to claim 2, characterized in that, The step of performing view segmentation on the roadside base station visual image sequence based on the visual patch segmentation unit to obtain a roadside base station visual image patch set includes: Based on the visual patch segmentation unit and the preset patch segmentation strategy, mesh division is performed to obtain visual patch parameters; Based on the visual patch segmentation unit and the visual patch parameters, the roadside base station visual image sequence is segmented to obtain multiple independent roadside base station visual image patches. Based on the visual patch segmentation unit, the multiple independent roadside base station visual image patches are integrated in an orderly manner to obtain the roadside base station visual image patch set.

4. The method according to claim 2, characterized in that, The step of projecting features onto the patch view feature set based on the projection unit to obtain the view token features includes: Based on the projection unit, the patch view feature set is projected into the linguistic domain to obtain the view semantic projection feature set; Based on the projection unit, the view semantic projection feature set is sorted to obtain the view token feature.

5. The method according to claim 1, characterized in that, The target beam prediction network includes a multi-head attention unit and a feature fusion unit; The target beam prediction network is used to perform multimodal feature fusion on the view token features and the text token features to obtain dual-modal fused features, including: Based on the multi-head attention unit, semantic association mining is performed on the view token features and the text token features to obtain semantic association features; Based on the feature fusion unit, feature interaction is performed on the semantic association features to obtain the dual-modal fusion features.

6. The method according to claim 5, characterized in that, The step of performing feature interaction on the semantic association features based on the feature fusion unit to obtain the dual-modal fusion features includes: Based on the feature fusion unit, the semantic association features are semantically coupled and enhanced to obtain deep fusion features; Based on the feature fusion unit, the deep fusion features are normalized to obtain the dual-modal fusion features.

7. The method according to any one of claims 1-6, characterized in that, After obtaining the target predicted beam by performing beam index prediction based on the target beam prediction network and the dual-modal fusion features, the process further includes: Based on the target vehicle-to-everything (V2X) beam prediction model, the target predicted beam is vectorized to obtain the beam index feature vector. The beam index feature vector and the dual-mode fusion feature are fused to obtain the temporal enhancement fusion feature; Based on the aforementioned temporal enhancement fusion features, beam index prediction is performed to obtain subsequent predicted beams.

8. A millimeter-wave vehicle-to-everything (V2X) beam prediction device, characterized in that, The device includes: The basic information acquisition module is used to acquire visual image sequences of roadside base stations and millimeter-wave beam prediction task constraints; The target model acquisition module is used to acquire the target vehicle network beam prediction model, wherein the target vehicle network beam prediction model includes a target view feature extraction network, a target text encoding network, and a target beam prediction network; The visual image parsing module is used to perform visual patching on the visual image sequence of the roadside base station based on the target view feature extraction network to obtain view token features; The constraint text encoding module is used to perform text encoding on the constraints of the millimeter-wave beam prediction task based on the target text encoding network to obtain text token features; The dual-modal feature fusion module is used to perform multimodal feature fusion on the view token feature and the text token feature based on the target beam prediction network to obtain dual-modal fused features; The beam index prediction module is used to perform beam index prediction based on the target beam prediction network and the dual-modal fusion features to obtain the target predicted beam.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the millimeter-wave vehicle networking beam prediction method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the millimeter-wave vehicle networking beam prediction method according to any one of claims 1 to 7.

Images