Multi-level environment perception method based on visual language large model and millimeter wave / terahertz channel observation

Abstract

The application provides a multi-level environment perception method based on a visual language large model and millimeter wave / terahertz channel observation, and belongs to the technical field of communication. The method comprises the following steps: acquiring channel gain observation data, combining real environment semantic information of a training scene to construct text domain and radio frequency domain prompt information and environment semantic description, and forming a training sample set; constructing a radio frequency-based environment perception framework comprising a routing module and a plurality of semantic level adaptation experts based on a pre-trained visual language large model, and establishing a corresponding parameter adaptation relationship; training the radio frequency-based environment perception framework by using the training sample set to obtain a trained routing module and a plurality of adaptation experts; acquiring channel gain observation data and environment perception requirements of a scene to be perceived, inputting the data into the radio frequency-based environment perception framework, and reasoning to obtain multi-level environment perception results of the scene. The application can realize unified processing from sparse radio frequency observation to multi-level environment semantic generation.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, and in particular relates to a multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation. Background Technology

[0002] Integrated communication and sensing technology in the millimeter-wave / terahertz band is an important development direction for future sixth-generation mobile communication systems. In integrated communication and sensing systems, radio frequency (RF) sensing aims to utilize RF observation information generated during wireless signal propagation to obtain environmental information related to the propagation environment. Because millimeter-wave / terahertz signals are affected by environmental structures, obstacle distribution, and changes in the propagation path during propagation, RF observation data often implicitly contains information related to the actual communication environment. How to extract effective environmental information from RF observation data is a crucial issue in integrated millimeter-wave / terahertz communication and sensing systems.

[0003] Existing radio frequency (RF) sensing methods are capable of utilizing channel state information, channel gain, or other RF observation data for environmental perception to a certain extent. For example, some methods focus on estimating obstacle properties, some on reconstructing the physical environment, and others process predefined sensing tasks through generative base models. These methods are effective for specific tasks, but they typically depend on task-specific model structures, output formats, or training methods, and the environmental information obtained often only covers a portion of environmental attributes. When sensing requirements change, or when new sensing tasks not explicitly defined during the training phase need to be handled, existing methods often require redesigning the task format, adjusting the model structure, or retraining the model, making it difficult to form a unified RF environmental sensing paradigm.

[0004] As future communication applications demand increasingly higher levels of environmental understanding, radio frequency (RF) sensing systems need to acquire not only local or singular environmental information but also a more comprehensive description of environmental semantics. Different sensing tasks may correspond to different levels of environmental semantics, or may require combinations of multiple semantic levels. For example, some tasks focus on environmental information at the overall layout level, some on environmental information at the obstacle level, and some on communication-related information caused by the environment. Therefore, RF environmental sensing methods need to be able to handle different sensing requirements within a unified model framework, rather than simply outputting fixed-form results for a single task.

[0005] In recent years, visual language models have demonstrated strong capabilities in multimodal representation learning and text generation. Visual language models can simultaneously process visual input and text cues, generating corresponding semantic descriptions based on the text cues. Inspired by this, the radio frequency (RF) sensing problem can be transformed into an environmental semantic description generation problem. This involves specifying sensing requirements using text cues and generating corresponding environmental semantic text from RF observation data such as millimeter-wave / terahertz channel gain. In this way, different RF sensing tasks can be specified by different text cues, thus enabling unified processing of multiple sensing tasks.

[0006] However, directly applying pre-trained visual language models to radio frequency (RF) environmental perception remains challenging. Pre-trained visual language models are typically learned from natural image and natural language data, with their visual and textual priors primarily derived from natural visual scenes. In contrast, millimeter-wave / terahertz channel gains reflect RF characteristics during wireless propagation. Significant differences exist between RF observations and natural images, making it difficult for pre-trained visual language models to accurately understand the environmental semantic information contained within the channel gains.

[0007] On February 2, 2024, an invention patent with publication number CN117498953A and titled "A Method for Constructing Terahertz Spectrum Environment Maps Based on Hybrid Active and Passive Sensing" was published. This patent utilizes terahertz communication node information to obtain coarse-grained and fine-grained obstacle information, mainly targeting obstacle environment map construction and obstacle attribute perception tasks. However, it does not establish a unified task description for different environmental perception needs, and its output content mainly focuses on obstacle-related information, making it difficult to simultaneously cover different levels of environmental semantics such as spatial layout, target attribute, and communication-related layers. It also cannot generate environmental perception results for different perception tasks based on text prompts.

[0008] On June 28, 2024, an invention patent with publication number CN118264976A, entitled "A Federated Obstacle Perception Method and Device Assisted by Directional Terahertz Spectrum Environment Map," was published. This patent constructs a signal power estimation network through federated learning to generate spectrum environment maps with different beam directions. It then uses image segmentation and intersection of multi-beam uncovered areas to obtain obstacle positions and shapes, primarily targeting obstacle detection and attribute recognition tasks. However, this method has relatively fixed task objectives and output formats, making it difficult to simultaneously handle multiple semantic levels such as spatial layout, target attributes, and communication-related aspects within a unified model framework. In particular, it cannot further complete communication-related perception tasks such as feasible beam directions and communication scenario categories. Furthermore, its perception process relies on pre-set spectrum maps, image segmentation, and intersection operations, lacking the ability to flexibly specify environmental perception requirements based on text prompts and generate corresponding environmental semantic text descriptions.

[0009] To address the aforementioned issues, a novel radio frequency (RF)-based environment sensing method is needed. This method should fully utilize channel gain observation data from millimeter-wave / terahertz communication processes to extract multi-layered environmental semantic information of real-world communication scenarios from RF domain data without relying on additional complex sensing equipment. Furthermore, this method should be able to uniformly specify different environment sensing tasks through text prompts and reduce the discrepancy between RF channel data and prior knowledge of natural image language through an adaptation mechanism. This would enable unified generation of environmental semantics at the spatial layout, target attribute, and communication-related layers, providing more comprehensive environmental information support for subsequent channel modeling, beam management, network optimization, and wireless digital twin construction. Summary of the Invention

[0010] The technical problem this application aims to solve is that existing radio frequency (RF) environment perception methods for millimeter-wave / terahertz communication scenarios are typically task-specific designs, making it difficult to handle environment perception tasks at different levels and with different combinations within a unified model framework. Since channel gain observations are usually sparsity and finite, and environmental attributes such as the number, location, shape, size, material, and degree of occlusion of obstacles have high dimensionality, relying solely on limited RF observation information may correspond to multiple candidate environment interpretations, resulting in a large feasible solution space for environmental semantic generation. This increases the ambiguity of environmental semantic inference and reduces the certainty of perception results.

[0011] Meanwhile, pre-trained visual-language large models are mainly trained based on natural image and natural language data, lacking prior representations of radio frequency (RF) propagation knowledge such as wireless propagation, occlusion, reflection, scattering, and beam reachability. Simply combining different adaptation parameters directly during the inference stage cannot guarantee that the routing module and experts at different semantic levels can learn stable and mutually distinguishable RF environment perception capabilities. Therefore, it is necessary to construct an RF-based environment perception method that can utilize multi-level semantic training samples and jointly train the routing module and multiple semantic-level low-rank adaptation experts.

[0012] To this end, this application provides a radio frequency-based multi-level environment perception method that covers both the training and inference phases. By constructing multi-level semantic training samples and jointly training the routing module and multiple low-rank adaptation experts, the model can dynamically call radio frequency perception knowledge at different semantic levels based on text prompts during the inference phase.

[0013] The technical means adopted by this invention to solve the above-mentioned technical problems is a multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation, comprising the following steps:

[0014] Sample construction steps: Obtain millimeter-wave / terahertz channel gain observation data for different beam directions and positions in multiple training scenarios, and combine the real environmental semantic information of the training scenarios to construct text domain and radio frequency domain prompt information, as well as environmental semantic descriptions, to form a training sample set;

[0015] Network architecture construction steps: Based on the pre-trained visual language large model, construct a radio frequency-based environment perception framework including a routing module and multiple semantic-level low-rank adaptation experts, and establish the parameter adaptation relationship between the routing module, multiple low-rank adaptation experts and the pre-trained visual language large model.

[0016] Joint training steps: Use the training sample set to train the radio frequency-based environment awareness framework to obtain a trained routing module and multiple low-rank adaptation experts;

[0017] Environmental perception generation steps: Obtain channel gain observation data and environmental perception requirements of the scene to be perceived, input them into the radio frequency-based environmental perception framework, and infer the multi-level environmental perception results of the scene.

[0018] Furthermore, the sample construction step includes the following steps:

[0019] Step S111: Select multiple training scenarios with different spatial structures, number of obstacles, obstacle attributes and communication propagation characteristics, and obtain millimeter wave / terahertz channel gain observation data for different positions with different beam directions in each training scenario;

[0020] Step S112: Arrange the channel gain observation data of each training scenario according to the spatial location of the communication node and the direction of the candidate transmission beam to obtain a sparse channel gain tensor. Input the sparse channel gain tensor into the channel gain reconstruction module to obtain a channel gain tensor. Visualize the channel gain tensor to obtain a training channel map.

[0021] Step S113: Based on the real environment information of each training scenario, obtain the spatial layout layer environment semantics, the target attribute layer environment semantics, and the communication-related layer environment semantics;

[0022] Step S114: Construct text prompts and corresponding real-world semantic text descriptions based on different environmental perception requirements;

[0023] Step S115: Combine the training channel map, text prompts, and real-world semantic text descriptions to form a training sample set for model training.

[0024] Furthermore, the specific steps for constructing the radio frequency-based environment awareness framework in the network architecture construction step are as follows:

[0025] A pre-trained visual language large model, including a visual encoding module, a text encoding module, and a text generation module, is selected as the base model. The visual encoding module is used to extract spatial distribution features in the channel map, the text encoding module is used to extract task requirement features in the text prompt information, and the text generation module is used to generate environmental semantic text descriptions.

[0026] For different semantic levels, low-rank adaptation experts for spatial layout, target attribute, and communication-related semantics are constructed respectively, which are responsible for parameter adaptation functions for overall spatial structure, obstacle attributes, and communication-related semantics.

[0027] A low-rank matrix group is set up for each low-rank adaptation expert, where the low-rank matrix group corresponding to the low-rank adaptation expert in the spatial layout layer is as follows. , This is called the low-rank input mapping matrix of the spatial layout layer. This is called the low-rank output mapping matrix of the spatial layout layer, and the parameter update matrix corresponding to the spatial layout layer. The low-rank matrix group corresponding to the low-rank adaptation expert in the target attribute layer is represented as follows: , This is called the low-rank input mapping matrix of the target attribute layer. This is called the low-rank input mapping matrix of the target attribute layer, and the parameter update matrix corresponding to the spatial layout layer. The low-rank matrix group corresponding to the low-rank adaptation expert in the communication-related layer is represented as follows: , This is called the low-rank input mapping matrix of the communication-related layer. This is called the low-rank output mapping matrix of the communication-related layer, and the parameter update matrix corresponding to the spatial layout layer. ;

[0028] A routing module consisting of a multi-layered perceptron is configured. The weight vectors generated by the routing module will be used for dynamic combination of multiple low-rank adaptation experts during the training and inference phases.

[0029] Furthermore, in the network architecture construction step, the parameter adaptation relationship between the routing module, multiple low-rank adaptation experts, and the pre-trained visual-language large model includes:

[0030] Text prompt message Input the pre-trained word embedding module or the text encoding module to obtain a cue embedding vector containing task requirements, semantic hierarchy, and generation target. ;

[0031] Embed the prompt vector Input routing module The routing module generates a training route weight vector that includes multiple low-rank adaptation expert weights. ,in These are parameters for the routing module;

[0032] Based on the training route weight vector Update the set of expert parameters By performing a weighted combination, we obtain the overall low-rank parameter update: ,in, Indicates transpose;

[0033] Update the comprehensive low-rank parameter The original parameters superimposed on the pre-trained visual language model This forms the parameters of the large visual language model that corresponds to the current training samples. ,in To adapt the scaling factor to the low rank, The rank of the low-rank matrix group used by each low-rank adaptation expert.

[0034] Furthermore, the specific steps for jointly training the routing module and multiple low-rank adaptation experts in the joint training step are as follows:

[0035] Training channel map in the training sample set Text prompts and real-world semantic text description The radio frequency-based environment perception framework is input, and the adapted visual language large model generates a prediction lexical probability matrix in an autoregressive manner. The predicted word probability matrix consists of the predicted probability distribution corresponding to each word position, i.e. ,in, , Indicates the first The predicted probability distribution of each word position. Indicates the total number of lexical units. Indicates the first The sequence of lexical units already generated before each lexical unit, Represents a probability generating function;

[0036] Using the cross-entropy loss between the predicted lexical probability matrix and the true lexical target matrix as the training objective, the routing module parameters and the low-rank matrix groups of multiple low-rank adaptation experts are jointly optimized. The training objective is:

[0037]

[0038] in, The number of training samples. To predict the cross-entropy loss between the lexical probability matrix and the true lexical target matrix, Indicates the first The predicted word probability matrix of each training sample. This is the set of parameters for the low-rank output mapping matrix. The set of parameters for the low-rank input mapping matrix;

[0039] During joint training, the original parameters of the pre-trained visual language large model are maintained. Freeze, only update routing module parameters and the set of parameters for the low-rank output mapping matrix and the set of parameters of the low-rank input mapping matrix The process continues until the preset training termination conditions are met, resulting in a trained routing module and multiple low-rank adaptation experts.

[0040] Furthermore, the specific steps of the environmental perception generation step are as follows:

[0041] Step S141: In the real-world communication scenario to be perceived, the millimeter-wave / terahertz base station communicates with several communication nodes using a set of candidate transmit beam directions to obtain channel gain observation data for different positions of different beam directions; the channel gain observation data is arranged according to spatial position and beam direction to obtain the sparse channel gain tensor to be perceived.

[0042] Step S142: Reconstruct the sparse channel gain tensor to be perceived using the channel gain reconstruction module to obtain the channel gain tensor. Then, perform beam direction slicing, stitching, normalization, and image processing on the channel gain tensor to obtain the channel map to be perceived. ;

[0043] Step S143: Obtain the current environmental perception requirement, determine the semantic level corresponding to the current environmental perception requirement, and construct inference text prompt information. ;

[0044] Step S144: Provide reasoning text prompts Convert to inference cue embedding vector The trained routing module is then input to obtain the inference routing weight vector. ;

[0045] Step S145: Based on the inference route weight vector The parameter update set for multiple low-rank fitting experts that have completed training. By performing a weighted combination, the overall low-rank parameter update is obtained. = ;

[0046] Step S146: Update the comprehensive low-rank parameters This is applied to a pre-trained visual language model to form an adapted visual language model. The parameters of the adapted visual language model are: and the channel map to be perceived and inference text prompts Input the adapted visual language large model to obtain the predicted word probability matrix. ; Generate environment perception results based on the predicted word probability matrix ;

[0047] Step S147: Perform structured analysis on the environmental perception results to obtain multi-level environmental perception results;

[0048] Step S148: Output multi-level environmental perception results and use the multi-level environmental perception results for subsequent communication decisions or environmental modeling.

[0049] Furthermore, the channel gain tensor obtained through channel gain reconstruction is: Then the average channel strength at each grid position can be calculated as follows:

[0050]

[0051] in, Indicates grid position The average channel strength at that location, Indicates the number of candidate beam directions. Indicates the first Okay, number Ranked in Channel gain in each transmit beam direction.

[0052] Furthermore, when the visual language large model outputs the first... When a beam direction is a candidate low-blocking beam direction, the average channel gain of all effective communication nodes under that beam direction can be calculated:

[0053]

[0054] in, Indicates the first The average channel gain of each effective communication node at the grid location under each candidate transmit beam direction. This represents the set of grid positions participating in the consistency check. This indicates the number of grid positions in the set.

[0055] Compared with the prior art, the present invention has the following beneficial technical effects:

[0056] 1. This invention integrates sparse radio frequency (RF) observations, environmental perception requirements, and semantic-level information into a unified visual language large-scale model processing flow through the collaborative cooperation of channel gain reconstruction, text prompt routing, and multiple semantic-level low-rank adaptation experts. Specifically, channel gain reconstruction improves the information integrity of RF visual input, text prompts and routing modules enable dynamic adaptation between different perception tasks, and multiple low-rank adaptation experts learn RF environmental perception knowledge corresponding to different semantic levels, thereby achieving multi-level environmental semantic generation from channel gain observation within a unified model framework.

[0057] 2. This invention combines the same training channel map with multiple text prompts and corresponding real-world semantic text descriptions, enabling the model to learn the environmental perception requirements and semantic levels indicated by different text prompts. This allows for the unified processing of one or more radio frequency environmental perception tasks in the spatial layout layer, target attribute layer, and communication-related layer within the same large visual language model framework.

[0058] 3. This invention enables the routing module to learn the correspondence between different text prompts and semantic level adaptation experts by jointly optimizing the routing module and multiple low-rank adaptation experts, and enables each low-rank adaptation expert to learn the radio frequency environment perception knowledge of the corresponding semantic level, thereby realizing dynamic adaptation and expert parameter combination between different environment perception tasks.

[0059] 4. During the inference phase, this invention can dynamically combine multiple trained low-rank adaptation experts according to the current environmental perception requirements to generate one or more environmental semantics in the spatial layout layer, target attribute layer, and communication-related layer. The resulting multi-level environmental perception results can serve as environmental information support for subsequent tasks such as channel modeling, beam management, link optimization, network deployment, and wireless digital twin construction. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of Example 1.

[0062] Figure 2 This is a schematic diagram of obstacles in a real-world communication scenario provided in Example 1.

[0063] Figure 3 This is a schematic diagram of multi-layered environmental semantics in a real-world communication scenario provided in Example 1.

[0064] Figure 4 This is a schematic diagram of the radio frequency-based environment sensing framework provided in Example 1.

[0065] Figure 5 This is a structural block diagram of the electronic device provided in Example 3. Detailed Implementation

[0066] To enable those skilled in the art to better understand the present application, the methods in the embodiments of the present application will be clearly and completely described below in conjunction with the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. Other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort should all fall within the scope of protection of the present application.

[0067] It should be noted that the millimeter-wave / terahertz communication scenario described in this application can be either a two-dimensional planar communication scenario or a three-dimensional stereoscopic communication scenario. For ease of explanation, the following embodiments are mainly described using a two-dimensional field communication scenario as an example, but this does not constitute a limitation on the scope of protection of this application. In practical applications, the field communication scenario can be a city block, indoor office area, road traffic area, park communication scenario, island communication scenario, jungle communication scenario, or other wireless communication scenario with complex obstacle distribution. The millimeter-wave / terahertz base station can be a fixed base station, mobile base station, vehicle-mounted base station, UAV base station, or other communication node with millimeter-wave / terahertz communication capabilities. The communication node can be a user terminal, vehicle terminal, sensor node, UAV terminal, edge device, or other device capable of receiving millimeter-wave / terahertz signals and acquiring channel observation information.

[0068] This application proposes a multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation, comprising a training phase and an inference phase. The training phase includes sample construction, network architecture construction, and joint training steps, used to enable the routing module to learn the dependencies between text prompts and different low-rank adaptation experts, and to enable multiple low-rank adaptation experts to learn radio frequency environment perception knowledge at different semantic levels. The inference phase includes an environment perception generation step, used to dynamically combine multiple trained low-rank adaptation experts according to the scene to be perceived and the current environment perception requirements, and output multi-level environment perception results. The following three embodiments provide a detailed description of this application.

[0069] Example 1

[0070] Please refer to Figure 1 , Figure 1This is a flowchart illustrating a multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation, provided in Embodiment 1 of this application. The method may include the following steps:

[0071] Step S110: Construct a multi-level radio frequency-based environment awareness training sample set to obtain training data for jointly training the routing module and multiple low-rank adaptation experts.

[0072] In an exemplary embodiment, each training sample includes a training channel map. Text prompts and real-world semantic text description The training channel map represents the radio frequency propagation characteristics of the training scenario at different spatial locations and with different beam directions; textual prompts specify the environmental awareness requirements and semantic levels corresponding to the current training sample; and real-world semantic text descriptions provide supervised generation targets corresponding to the textual prompts. Thus, the training samples contain not only radio frequency domain prompts but also textual domain prompts and real-world semantic text descriptions, enabling joint training of the subsequent routing module and low-rank adaptation experts. Radio frequency domain prompts refer to the training channel map, and textual domain prompts refer to the textual prompts themselves.

[0073] This application provides an example of a possible real-world communication scenario, such as... Figure 2 As shown. Figure 2 This diagram illustrates the distribution of obstacles in a real-world communication scenario. The scenario includes several obstacles of varying sizes, shapes, and locations. The millimeter-wave / terahertz base station is situated at a predetermined location within the scenario, while several communication nodes are distributed in spaces not occupied by obstacles. Because millimeter-wave / terahertz signals possess strong directionality and weak diffraction capability, different obstacles will have varying impacts on the channel gain distribution under different beam directions.

[0074] In an exemplary embodiment, step S110 includes sub-steps S111 to S115, each of which sequentially completes the selection of training scenarios, construction of channel maps, annotation of real semantics, construction of prompt information, and combination of training samples.

[0075] Sub-step S111: Select multiple training scenarios and obtain millimeter-wave / terahertz channel gain observation data for different positions in different beam directions in each training scenario.

[0076] Specifically, multiple training scenarios with different spatial structures, number of obstacles, obstacle attributes, and communication propagation characteristics are selected, and each training scenario is divided into... Each of the 10 grid cells has its center coordinates labeled as _ ... ,in, Indicates the center of the grid at Raster index along the axis, , Indicates the center of the grid at Raster index along the axis, The training scenarios can be obtained through field measurements, ray tracing simulation, or a combination of both, to ensure that the training sample set covers different environmental layouts, obstacle distributions, and channel propagation states. Please refer to [reference needed]. Figure 2 , Figure 2 This is a schematic diagram of obstacles in a real-world communication scenario provided in Example 1.

[0077] Millimeter-wave / terahertz base stations are set up in various training scenarios and The communication node, the first The grid location of each communication node is marked as Setting up millimeter-wave / terahertz base stations with candidate transmit beam direction sets ,in For beam direction index, Indicates the first One candidate transmit beam direction, This represents the total number of candidate transmit beam directions, and .

[0078] Millimeter-wave / terahertz base stations communicate with communication nodes under each candidate transmit beam direction, enabling each communication node to obtain the channel response, received signal power, or channel power gain under the corresponding spatial location and beam direction, and to obtain channel gain observation data for different locations under different beam directions in multiple training scenarios.

[0079] Sub-step S112: Based on the channel gain observation data of each training scenario, perform tensor quantization, channel gain reconstruction, and visualization processing to construct a training channel map.

[0080] Specifically, channel gain observation data from the same training scenario are arranged according to the spatial location of the communication node and the direction of the candidate transmit beam to obtain a sparse channel gain tensor. ; Element index of the sparse channel gain tensor Corresponding grid position and the One candidate transmit beam direction.

[0081] The sparse channel gain tensor Input channel gain reconstruction module The channel gain tensor is obtained as follows:

[0082]

[0083] in, Represents the channel gain tensor. This indicates the parameters of the channel gain reconstruction module.

[0084] The channel gain tensor is visualized to obtain a training channel map; the visualization process includes: visualizing the channel gain tensor... Slicing the beam along the beam direction yields the two-dimensional channel gain matrix corresponding to the nth candidate transmit beam direction, denoted as: , Indicates the first The training channel map is obtained by stitching together, normalizing, and visualizing the two-dimensional channel gain matrices corresponding to the candidate transmit beam directions, and then performing other processing on the two-dimensional channel gain matrices for different beam directions. This refers to the cue information in the radio frequency domain. In this way, the trained channel map retains both the channel gain variations in the spatial dimension and the propagation differences between different beam directions.

[0085] Sub-step S113: Obtain the spatial layout layer environmental semantics based on the real environment information of each training scenario. Target attribute layer environment semantics and communication-related layer environment semantics .

[0086] Specifically, real-world environmental information for each training scenario is acquired and categorized into three layers based on common environmental perception requirements: spatial layout layer environmental semantics, target attribute layer environmental semantics, and communication-related layer environmental semantics. The spatial layout layer environmental semantics describes the overall spatial structure of the training scenario, including the total area occupied by obstacles, the spatial distribution of obstacle centroids, and the average position of multiple obstacle centroids. The target attribute layer environmental semantics describes the attributes of one or more obstacles, including the number of obstacles, the position, centroid, shape, area, size, and orientation of a single obstacle. The communication-related layer environmental semantics describes the correlation between the environmental structure and wireless propagation, including the communication scenario category, candidate low-blocking beam directions under different base station locations or configurations, link blocking risk, and channel propagation status. Please refer to [reference needed]. Figure 3 , Figure 3 This is a schematic diagram of multi-level environmental semantics in a real-world communication scenario provided in Example 1.

[0087] Sub-step S114: Based on different environmental perception requirements, construct text prompt information and corresponding real-world semantic text descriptions.

[0088] Specifically, environmental awareness requirements are set for single semantic levels and combinations of multiple semantic levels in the spatial layout layer, target attribute layer, and communication-related layer; the environmental awareness requirements, corresponding semantic levels, and expected generated content are then organized into text prompts. The text prompts can take the form of natural language questions, task descriptions, or preset template statements, and are used to clarify the environmental semantic content that the visual language big data model needs to extract from the training channel map.

[0089] Same training channel map Able to respond to multiple different text prompts This allows the same training scenario to generate multiple training samples, each targeting a different environment perception task, enabling the routing module to learn the semantic hierarchy information contained in the text prompts and the dependencies between multiple low-rank adaptation experts.

[0090] Based on the semantic level corresponding to the environmental perception requirements, from the spatial layout layer environmental semantics Target attribute layer environment semantics and communication-related layer environment semantics Select one or more corresponding real-world semantics and construct a real-world semantic text description according to natural language form, semi-structured text form, or a preset text template. This enables the semantic text description of the real environment. Includes the text prompt information The required environmental semantics.

[0091] For example, when text prompt information When corresponding to the target attribute layer, from The text prompt selects real-world semantic information such as the number, location, shape, or area of ​​obstacles; when the text prompt information... When corresponding to the communication-related layer, from The selection process considers real-world semantics such as candidate low-blocking beam directions, link blocking risks, or communication scenario categories.

[0092] Sub-step S115: Combine the training channel map, text prompt information, and real-world semantic text description to form a training sample set for model training.

[0093] Specifically, this refers to training channel maps that correspond to the same training scenario and the same environmental perception requirements. Text prompts and real-world semantic text description A training sample is formed. Since the same training channel map can match multiple different text prompts, the same training scenario can be expanded to form multiple training samples targeting different semantic levels or semantic combinations. The same training channel map can be combined with multiple different text prompts and corresponding real-world semantic text descriptions to form multiple training samples corresponding to different environment perception tasks, enabling the subsequent routing module to learn the correspondence between text prompts and low-rank adaptation experts at different semantic levels.

[0094] Training samples are repeatedly constructed for multiple training scenarios and different semantic levels or combinations thereof, so that the training sample set covers different spatial structures, obstacle attributes, communication propagation features and environmental perception requirements, thereby improving the generalization ability of the trained model in different scenarios to be perceived.

[0095] In one alternative implementation, the training sample set can be divided into a training subset and a validation subset; the training subset is used to update the parameters of the routing module and the low-rank adaptation expert, and the validation subset is used to determine the training termination condition, select the model parameters after training is completed, or evaluate the generation effect of different semantic level tasks.

[0096] Step S120: Construct a radio frequency (RF)-based environment perception framework based on a pre-trained visual language large model. The RF-based environment perception framework includes the pre-trained visual language large model, a routing module, and multiple semantic-level low-rank adaptation experts. Establish parameter adaptation relationships between the routing module, the multiple low-rank adaptation experts, and the pre-trained visual language large model. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of the radio frequency-based environment sensing framework provided in Example 1.

[0097] In an exemplary embodiment, step S120 includes sub-steps S121 to S123, each sub-step sequentially completing the selection of the basic model, the setting of the routing module and the low-rank adaptation expert, and the construction of parameter adaptation relationships.

[0098] Sub-step S121: Select a pre-trained visual language large model and set up the routing module.

[0099] Specifically, a pre-trained visual language large-scale model, including a visual encoding module, a text encoding module, and a text generation module, is selected as the base model. The visual encoding module extracts spatial distribution features from the channel map, the text encoding module extracts task requirement features from text prompts, and the text generation module generates environmental semantic text descriptions. These visual encoding, text encoding, and text generation modules are existing functional modules of the pre-trained visual language large-scale model, and this application does not modify their structure or original parameters. A routing module composed of multilayer perceptrons is set in the pre-trained visual language large-scale model. The weight vector generated by this routing module will be used for the dynamic combination of multiple low-rank adaptation experts during the training and inference phases.

[0100] Sub-step S122: Set up multiple semantic level low-rank adaptation experts.

[0101] Specifically, multiple low-rank adaptation experts are set at one or more target weight matrices in the pre-trained visual language large model. Taking any target weight matrix as an example, let its dimension be... And let the rank of the low-rank fitness matrix be . ,in Less than and For different semantic levels, low-rank adaptation experts are constructed for spatial layout, target attribute, and communication-related semantics, respectively, to undertake parameter adaptation functions for overall spatial structure, obstacle attributes, and communication-related semantics.

[0102] Low-rank adaptation experts at the spatial layout layer are used to learn the mapping relationship between the channel map and the semantics of the overall spatial structure; low-rank adaptation experts at the target attribute layer are used to learn the mapping relationship between the channel map and the semantics of obstacle attributes; and low-rank adaptation experts at the communication-related layer are used to learn the mapping relationship between the channel map and communication-related tasks.

[0103] For the target weight matrix in the pre-trained visual language large model, let the intermediate hidden state feature vector received by its network layer be... A separate low-rank matrix group is set up for each low-rank adaptation expert.

[0104] Among them, the low-rank matrix group corresponding to the low-rank adaptation expert in the spatial layout layer is represented as follows: , This is called the low-rank input mapping matrix of the spatial layout layer, used to map the received hidden state feature vectors. Mapping to the low-rank adaptation space yields low-dimensional adapted feature vectors. This is called the low-rank output mapping matrix of the spatial layout layer, used to map the low-dimensional fitting feature vectors back to the output feature space corresponding to the target weight matrix, thereby obtaining the parameter update matrix corresponding to the spatial layout layer. The low-rank matrix group This is used to enhance the model's ability to model spatial layout semantics such as overall spatial structure, obstacle occupancy, and obstacle spatial distribution.

[0105] The low-rank matrix group corresponding to the low-rank adaptation expert in the target attribute layer is represented as follows: , This is called the low-rank input mapping matrix of the target attribute layer, used to map the received hidden state feature vectors. Mapping to the low-rank adaptation space yields low-dimensional adapted feature vectors. This is called the low-rank input mapping matrix of the target attribute layer, used to map the low-dimensional adapted feature vectors back to the output feature space corresponding to the target weight matrix, thereby obtaining the parameter update matrix corresponding to the spatial layout layer. The low-rank matrix group It is used to enhance the model's ability to model the semantics of target attributes such as the number, location, shape, area, and size of obstacles.

[0106] The low-rank matrix group corresponding to the low-rank adaptation expert in the communication-related layer is represented as follows: , This is called the low-rank input mapping matrix of the communication correlation layer, used to map the received hidden state feature vectors. Mapping to the low-rank adaptation space yields low-dimensional adapted feature vectors. This is called the low-rank output mapping matrix of the communication-related layer, which represents the output feature space used to map the low-dimensional adaptation feature vector back to the target weight matrix, thereby obtaining the parameter update matrix corresponding to the spatial layout layer. The low-rank matrix group This is used to enhance the model's ability to model communication-related semantics such as communication scenario categories and candidate low-blocking beam directions.

[0107] The rank of the low-rank matrix group is less than the dimension of the low-rank matrix group corresponding to the pre-trained visual language large model. Thus, while keeping the original parameters of the pre-trained visual language large model stable, the adaptation knowledge required for the radio frequency environment perception task can be learned with fewer trainable parameters.

[0108] With the above settings, different low-rank adaptation experts can learn radio frequency environment perception knowledge at different semantic levels without significantly changing the original parameters of the pre-trained visual language large model.

[0109] Sub-step S123: Construct the parameter adaptation relationship between the routing module, the low-rank adaptation expert, and the pre-trained visual language large model.

[0110] Text prompts from training samples Input the pre-trained word embedding module or the text encoding module to obtain a cue embedding vector that contains environmental awareness requirements and semantic hierarchical information. Specifically, the text prompts in the training samples The input word segmenter performs word segmentation to obtain a word sequence composed of multiple word units and their corresponding attention masks. Using the input word embedding layer of a pre-trained visual language model, each word unit is mapped to its corresponding word embedding vector. Valid word units are selected based on the attention masks, and the valid word embedding vectors are then subjected to average pooling to obtain a fixed-dimensional cue embedding vector. Hint: Embedded vector It includes environmental perception requirements and semantic-level information.

[0111] Embed the prompt vector Input routing module The routing module generates a training route weight vector that includes multiple low-rank adaptation expert weights. ,in For routing module parameters, This represents the embedding vector of the prompt. The training route weight vector is used to represent the degree of dependence of the current text prompt information's environment awareness task on low-rank adaptation experts at different semantic levels.

[0112] Based on the training route weight vector Update the set of expert parameters By performing a weighted combination, we obtain the overall low-rank parameter update: ,in, Indicates transpose

[0113] Update the comprehensive low-rank parameter The original parameters superimposed on the pre-trained visual language model This forms the parameters of the large visual language model that corresponds to the current training samples. ,in To adapt the scaling factor to the low rank, The rank of the low-rank matrix group used by each low-rank adaptation expert.

[0114] Will have the aforementioned adaptation parameters The large-scale visual language model is called the adapted visual language model. The adapted visual language model adopts the same model structure as the pre-trained visual language model. The difference is that the adapted visual language model adds parameter updates obtained by combining multiple low-rank adaptation experts to the original parameters of the pre-trained visual language model.

[0115] When the text prompt corresponds to a single semantic level, the routing module increases the weight of the low-rank adaptation expert corresponding to that semantic level, so that the model prioritizes calling the radio frequency environment perception knowledge most relevant to the current perception task.

[0116] When the text prompt information corresponds to multiple semantic levels, the routing module performs joint weighting on the low-rank adaptation experts corresponding to the multiple semantic levels, so that the same visual language large model can complete the environmental perception task of multiple semantic level combinations.

[0117] Through the parameter adaptation relationship, different low-rank adaptation experts learn radio frequency environment perception knowledge corresponding to their semantic level during the training process, and are dynamically invoked according to text prompts during the inference process.

[0118] Step S130: Jointly train the radio frequency-based environment awareness framework constructed in step S120 using the training sample set. Based on the training channel map, text prompt information and real environment semantic text description in the training samples, optimize the routing module parameters and the low-rank matrix group of multiple low-rank adaptation experts to obtain the trained routing module and multiple low-rank adaptation experts.

[0119] In an exemplary embodiment, step S130 includes sub-steps S131 to S135, each of which sequentially completes the generation of training route weight vector, calculation of comprehensive low-rank parameter update amount, generation of predicted environment semantic text, calculation of training loss, and parameter optimization.

[0120] Sub-step S131: Generate training route weight vectors.

[0121] Specifically, the text prompts in the training samples Convert into a cue embedding vector that includes environmental awareness requirements and semantic hierarchy information. The prompt is embedded into the vector input routing module. The training route weight vector is obtained. ,in These are parameters for the routing module.

[0122] Sub-step S132: Form a large-scale visual language model that is adapted to the current training samples.

[0123] Based on the parameter adaptation relationship established in step S123, the trained routing weight vector is used. The parameter updates from multiple low-rank fitting experts are weighted and combined to obtain the comprehensive low-rank parameter update for the current training sample. .

[0124] The updated low-rank parameters are applied to the pre-trained visual language large model to form the adapted visual language large model corresponding to the current training sample, thus obtaining the parameters of the adapted visual language large model for the current training sample. .

[0125] Sub-step S133: Generate the prediction lexical probability matrix.

[0126] Specifically, the training channel map in the training sample set Text prompts and real-world semantic text description The radio frequency-based environment perception framework is input, and the adapted visual language large model generates a prediction lexical probability matrix in an autoregressive manner. The predicted word probability matrix consists of the predicted probability distribution corresponding to each word position, i.e. ,in, , Indicates the first The predicted probability distribution of each word position. Indicates the total number of lexical units. Indicates the first The sequence of lexical units already generated before each lexical unit, The parameter is The adapted visual language large model outputs a probability generation function of the probability distribution of the next word based on the input training channel map, text prompt information and the generated word sequence.

[0127] Sub-step S134: Construct the training objective and calculate the training loss.

[0128] Convert real-world semantic text descriptions into real-world lexical target matrices Using the cross-entropy loss between the predicted lexical probability matrix and the true lexical target matrix as the training objective, the routing module parameters and the low-rank matrix groups of multiple low-rank adaptation experts are jointly optimized. The training objective is:

[0129]

[0130] in, The number of training samples. To predict the cross-entropy loss between the lexical probability matrix and the true lexical target matrix, Indicates the first The predicted word probability matrix of each training sample. The set of parameters for the low-rank output mapping matrix consists of the low-rank output mapping matrix of the low-rank adaptation experts from the spatial layout layer, target attribute layer, and communication-related layer. composition, The set of parameters for the low-rank input mapping matrix is ​​given by the corresponding low-rank input mapping matrix. composition.

[0131] Sub-step S135: Jointly optimize the routing module and multiple low-rank adaptation experts.

[0132] During the joint training process, the original parameters of the pre-trained visual language large model are maintained. Freeze, only update routing module parameters and the set of parameters for the low-rank output mapping matrix and the set of parameters of the low-rank input mapping matrix The process continues until the preset training termination conditions are met, resulting in a trained routing module and multiple low-rank adaptation experts.

[0133] Step S140: Obtain channel gain observation data and inference text prompts for the scene to be perceived, and generate a multi-level environmental description of the scene to be perceived through the trained radio frequency-based environmental perception framework.

[0134] In an exemplary embodiment, the inference input includes a channel map to be perceived. and inference text prompts The channel map to be perceived is obtained by reconstructing and visualizing channel gain observation data of the actual communication scenario to be perceived. The inference text prompts are constructed based on the current environmental perception requirements and their corresponding semantic levels. By simultaneously constructing radio frequency domain prompts and text domain prompts, the model can invoke the radio frequency environmental perception knowledge learned during the training phase according to the current perception task during the inference phase.

[0135] In an exemplary embodiment, step S140 includes sub-steps S141 to S148, each of which sequentially completes the construction of the channel map to be perceived, the generation of inference text prompts, the generation of routing weights, the generation of environmental semantic text, and the parsing of results.

[0136] Sub-step S141: In the actual communication scenario to be perceived, the millimeter-wave / terahertz base station communicates with several communication nodes using a set of candidate transmit beam directions to obtain channel gain observation data for different locations with different beam directions; the channel gain observation data is arranged according to spatial location and beam direction to obtain the sparse channel gain tensor to be perceived; the sparse channel gain tensor to be perceived is used to describe the beam-related channel gain distribution at the actual observable location of the communication node.

[0137] Sub-step S142: Reconstruct the sparse channel gain tensor to be perceived using the channel gain reconstruction module to obtain the channel gain tensor, and perform beam direction slicing, stitching, normalization, and image processing on the channel gain tensor to obtain the channel map to be perceived. .

[0138] Sub-step S143: Obtain the current environmental perception requirement, determine the semantic level corresponding to the current environmental perception requirement, and construct inference text prompt information. .

[0139] Specifically, the current environmental perception requirements are determined based on the environmental information needed for actual communication functions, and these requirements are then categorized into one or more semantic levels, including the spatial layout layer, target attribute layer, and communication-related layer. The current environmental perception requirements, corresponding semantic levels, and expected generated content are then organized into inference text prompts. and the channel map to be perceived and inference text prompts Together they serve as the reasoning input for the environmental perception generation step.

[0140] Sub-step S144: Provide inference text prompts Convert to inference cue embedding vector The trained routing module is then input to obtain the inference routing weight vector. .

[0141] Sub-step S145: Based on inference route weight vector The parameter update set for multiple low-rank fitting experts that have completed training. By performing a weighted combination, the overall low-rank parameter update is obtained. = .

[0142] Specifically, when the inference text prompt corresponds to a single semantic level, the routing module increases the weight of the low-rank adaptation expert at the corresponding semantic level; when the inference text prompt corresponds to multiple semantic levels, the routing module jointly weights the low-rank adaptation experts corresponding to multiple semantic levels.

[0143] Sub-step S146: Update the comprehensive low-rank parameters This is applied to a pre-trained visual language model to form an adapted visual language model. The parameters of the adapted visual language model are: and the channel map to be perceived and inference text prompts Input the adapted visual language large model to obtain the predicted word probability matrix. .

[0144] Specifically, the predicted lexical probability matrix corresponding to the environmental semantic text description output by the adapted visual language large model is... The environment perception result is obtained from the predicted word probability matrix using maximum probability decoding, beam search decoding, or other text decoding methods. When the inference text prompts information When instructing environmental perception tasks at the spatial layout layer, target attribute layer, or communication-related layer, the environmental perception results include environmental perception information at the corresponding layer.

[0145] Sub-step S147: Perform structured analysis on the environmental perception results to obtain multi-level structured environmental information.

[0146] Specifically, regarding the environmental perception results Semantic parsing is performed to extract the environmental attribute names, values, and semantic levels they belong to, and then the results are parsed into a spatial layout layer structure. Target attribute layer structured results Structured results related to communication layers One or more of them.

[0147] Sub-step S148: Output multi-level environmental perception results and use the multi-level environmental perception results for subsequent communication decisions or environmental modeling.

[0148] Specifically, the structuring result of the spatial layout layer Target attribute layer structured results Structured results related to communication layers The splicing and combination results in a multi-level environmental perception. The multi-level environment perception results can be used to assist in completing one or more communication functions, such as channel modeling, beam selection, beam tracking, network deployment optimization, communication scenario identification, low-blocking link selection, or wireless digital twin environment construction. When the current environment perception requirement only corresponds to a part of the semantic level, the structured results corresponding to the semantic level can also be output only.

[0149] The method described in this embodiment enables the joint training of a routing module and multiple low-rank adaptation experts based on multi-level radio frequency environment semantic training samples during the training phase. During the inference phase, the trained low-rank adaptation experts are dynamically combined according to current environmental perception requirements, thereby achieving automatic conversion from millimeter-wave / terahertz channel gain observation data to multi-level and structured environmental perception results. Compared to schemes that only describe the inference process, this embodiment further clarifies the training sources of the routing module and multiple low-rank adaptation experts, forming a complete closed loop between the training and inference phases.

[0150] Example 2

[0151] This embodiment provides a multi-layered environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation. Building upon the training and inference phases described in Embodiment 1, Embodiment 2 provides a semantic consistency verification and enhanced prompt generation method performed in the environment perception generation step.

[0152] In the environment perception generation step of Example 1, the trained routing module and multiple low-rank adaptation experts generate corresponding environment perception results based on the channel map to be perceived and the inference text prompts. These results are then further analyzed to obtain the environment perception results for the spatial layout layer, target attribute layer, and communication-related layer. Because millimeter-wave / terahertz channel gain images differ from natural images, and because the environment perception results are generative outputs, in some complex scenarios, the model may generate semantic results that are not entirely consistent with the physical features of the channel map.

[0153] Therefore, to further improve the stability and reliability of the generated environmental semantics, Example 2 adds a semantic consistency verification process based on Example 1. This process mainly includes four parts: channel feature extraction, semantic result parsing, consistency scoring, and cue enhancement, focusing on spatial coverage features, obstacle blocking features, and candidate beam reachability features in the channel map.

[0154] Specifically, in Example 2, several simple channel statistical features are first extracted from the channel map obtained in Example 1. Let the channel gain tensor obtained through channel gain reconstruction in Example 1 be... Then the average channel strength at each grid position can be calculated as follows:

[0155]

[0156] in, Indicates grid position The average channel strength at that location, Indicates the number of candidate beam directions. Indicates the first Okay, number Ranked in Channel gain in each transmit beam direction.

[0157] The average channel strength described above can provide a preliminary indication of signal coverage at different spatial locations. If the average channel strength for a particular area remains consistently low, it suggests that the area may be experiencing obstruction, severe fading, or signal unavailability.

[0158] Furthermore, the environmental perception results generated by the large visual language model are subjected to structured analysis to obtain candidate multi-level environmental perception results. Let the structured result of the spatial layout layer obtained by analysis be... The target attribute layer structured result is The structured result of the communication-related layer is The multi-level environmental perception results output by the model can be expressed as:

[0159]

[0160] in, It represents the results of multi-level environmental perception, is used to represent the structured results of the spatial layout layer, and is used to represent the semantics of the overall spatial layout. This represents the structured result of the target attribute layer, used to represent the semantics of the target attribute layer, such as the number, location, shape, or area of ​​obstacles. This represents the structured results of the communication-related layers, used to represent communication scenario categories, candidate low-blocking beam directions, or link blocking-related information.

[0161] In one alternative implementation, consistency verification of the semantic results of the communication-related layers can be performed based on the channel map. For example, when the visual language large model outputs the first... When a beam direction is a candidate low-blocking beam direction, the average channel gain of all effective communication nodes under that beam direction can be calculated:

[0162]

[0163] in, Indicates the first The average channel gain of each effective communication node at the grid location under each candidate transmit beam direction. This represents the set of grid positions participating in the consistency check. This indicates the number of grid positions in the set. If the candidate low-blocking beam direction output by the large visual language model corresponds to... A larger value indicates that the beam direction is largely consistent with the high channel gain region shown in the channel map; if its corresponding If the value is significantly lower, it indicates that the output result may be inconsistent with the physical propagation characteristics in the channel map.

[0164] Furthermore, a semantic consistency score can be constructed. This is used to represent the degree of consistency between the results generated by the large visual language model and the channel map. Taking the semantics of the communication-related layer as an example, if the set of candidate low-blocking beam directions output by the model is... Then we can define:

[0165]

[0166] in, Indicates the consistency score of the communication-related layers. This represents the set of candidate low-blocking beam directions output by the large visual language model. This indicates the number of beam directions output by the model. Indicates the first The average channel gain corresponding to each beam direction. This represents the maximum average channel gain across all candidate beam directions. Communication-related layer consensus score. The closer it is to 1, the more consistent the candidate beam direction output by the model is with the high-gain direction in the channel map; The smaller the value, the more likely there is a deviation in the model's output.

[0167] In another alternative implementation, consistency verification can also be performed on the semantics of the target attribute layer. For example, when the visual language large model outputs that an obstacle exists in a certain region, it can be determined whether the average channel strength corresponding to that region is lower than a preset threshold. Let the obstacle candidate region be... Then the obstacle area consistency score can be calculated:

[0168]

[0169] in, This indicates the consistency score of the target attribute layer. This represents the candidate obstacle region obtained from the analysis of environmental perception results. This indicates the number of grid positions in the set. This indicates the preset low channel strength threshold. Indicates an indicator function. When When the signal strength is low, it indicates that the average channel strength at that grid location is low, which may be related to obstruction or signal blockage.

[0170] Furthermore, the consistency scores of the communication-related layer and the target attribute layer can be fused to obtain the overall semantic consistency score:

[0171]

[0172] in, Indicates semantic consistency score, and These represent the weights of the consistency score at the communication-related layer and the consistency score at the target attribute layer, respectively, and satisfy the following conditions: In practical applications, adjustments can be made dynamically based on the environmental awareness task type corresponding to the current text prompt. and The value of .

[0173] Therefore, Example 2 can verify and enhance the generated results of the large visual language model without changing the overall model structure of Example 1, through simple channel statistical features and semantic consistency scoring. Compared with the method of only performing one-time text generation, this example can reduce unreasonable results in multi-level environmental semantic output, improve the credibility of candidate low-blockage beam direction, obstacle position and link blocking related semantics, thereby further improving the stability and interpretability of radio frequency-based environmental perception results in millimeter wave / terahertz communication scenarios.

[0174] Example 3

[0175] Please refer to Figure 5 , Figure 5 This is a structural block diagram of an electronic device 900 provided in Embodiment 3 of this application. The electronic device 900 in this application may include one or more of the following components: a processor 910, a memory 920, and one or more application programs, wherein the one or more application programs may be stored in the memory 920 and configured to be executed by one or more processors 910, and the one or more programs are configured to perform the methods as described in the foregoing method embodiments.

[0176] Processor 910 may include one or more processing cores. Processor 910 connects to various parts within the electronic device 900 using various interfaces and lines, and performs various functions and processes data of the electronic device 900 by running or executing instructions, programs, code sets, or instruction sets stored in memory 920, and by calling data stored in memory 920. Optionally, processor 910 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 910 may integrate one or more of a Central Processing Unit (CPU) and a modem. The CPU primarily handles the operating system and applications; the modem is used for wireless communication. It is understood that the modem may also not be integrated into processor 910 and may be implemented separately using a communication chip.

[0177] The memory 920 may include random access memory (RAM) or read-only memory (ROM). The memory 920 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data created by the electronic device 900 during use.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation, characterized in that, The method includes the following steps: Sample construction steps: Obtain millimeter-wave / terahertz channel gain observation data for different beam directions and positions in multiple training scenarios, and combine the real environmental semantic information of the training scenarios to construct text domain and radio frequency domain prompt information, as well as environmental semantic descriptions, to form a training sample set; Network architecture construction steps: Based on the pre-trained visual language large model, construct a radio frequency-based environment perception framework including a routing module and multiple semantic-level low-rank adaptation experts, and establish the parameter adaptation relationship between the routing module, multiple low-rank adaptation experts and the pre-trained visual language large model. Joint training steps: Use the training sample set to train the radio frequency-based environment awareness framework to obtain a trained routing module and multiple low-rank adaptation experts; Environmental perception generation steps: Obtain channel gain observation data and environmental perception requirements of the scene to be perceived, input them into the radio frequency-based environmental perception framework, and infer the multi-level environmental perception results of the scene.

2. The multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation as described in claim 1, characterized in that, The sample construction process includes the following steps: Step S111: Select multiple training scenarios with different spatial structures, number of obstacles, obstacle attributes and communication propagation characteristics, and obtain millimeter wave / terahertz channel gain observation data for different positions with different beam directions in each training scenario; Step S112: Arrange the channel gain observation data of each training scenario according to the spatial location of the communication node and the direction of the candidate transmission beam to obtain a sparse channel gain tensor. Input the sparse channel gain tensor into the channel gain reconstruction module to obtain a channel gain tensor. Visualize the channel gain tensor to obtain a training channel map. Step S113: Based on the real environment information of each training scenario, obtain the spatial layout layer environment semantics, the target attribute layer environment semantics, and the communication-related layer environment semantics; Step S114: Construct text prompts and corresponding real-world semantic text descriptions based on different environmental perception requirements; Step S115: Combine the training channel map, text prompts, and real-world semantic text descriptions to form a training sample set for model training.

3. The multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation as described in claim 1, characterized in that, The specific steps for constructing the radio frequency-based environment awareness framework in the network architecture construction process are as follows: A pre-trained visual language large model, including a visual encoding module, a text encoding module, and a text generation module, is selected as the base model. The visual encoding module is used to extract spatial distribution features in the channel map, the text encoding module is used to extract task requirement features in the text prompt information, and the text generation module is used to generate environmental semantic text descriptions. For different semantic levels, low-rank adaptation experts for spatial layout, target attribute, and communication-related semantics are constructed respectively, which are responsible for parameter adaptation functions for overall spatial structure, obstacle attributes, and communication-related semantics. A low-rank matrix group is set up for each low-rank adaptation expert, where the low-rank matrix group corresponding to the low-rank adaptation expert in the spatial layout layer is as follows. , This is called the low-rank input mapping matrix of the spatial layout layer. This is called the low-rank output mapping matrix of the spatial layout layer, and the parameter update matrix corresponding to the spatial layout layer. The low-rank matrix group corresponding to the low-rank adaptation expert in the target attribute layer is represented as follows: , This is called the low-rank input mapping matrix of the target attribute layer. This is called the low-rank input mapping matrix of the target attribute layer, and the parameter update matrix corresponding to the spatial layout layer. The low-rank matrix set corresponding to the low-rank adaptation expert in the communication-related layer is represented as follows: , This is called the low-rank input mapping matrix of the communication-related layer. This is called the low-rank output mapping matrix of the communication-related layer, and the parameter update matrix corresponding to the spatial layout layer. ; A routing module consisting of a multi-layered perceptron is configured. The weight vectors generated by the routing module will be used for dynamic combination of multiple low-rank adaptation experts during the training and inference phases.

4. The multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation as described in claim 1, characterized in that, In the network architecture construction steps, the parameter adaptation relationship between the routing module, multiple low-rank adaptation experts, and the pre-trained visual language large model includes: Text prompt message Input the pre-trained word embedding module or the text encoding module to obtain a cue embedding vector containing task requirements, semantic hierarchy, and generation target. ; Embed the prompt vector Input routing module The routing module generates a training route weight vector that includes multiple low-rank adaptation expert weights. ,in For routing module parameters; Based on the training route weight vector Update the set of expert parameters By performing a weighted combination, we obtain the overall low-rank parameter update: ,in, Indicates transpose; Update the comprehensive low-rank parameter The original parameters superimposed on the pre-trained visual language model This forms the parameters of the large visual language model that corresponds to the current training samples. ,in To adapt the scaling factor to the low rank, The rank of the low-rank matrix group used by each low-rank adaptation expert.

5. The multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation as described in claim 1, characterized in that, The specific steps for jointly training the routing module and multiple low-rank adaptation experts in the joint training step are as follows: Training channel map in the training sample set Text prompts and real-world semantic text description The radio frequency-based environment perception framework is input, and the adapted visual language large model generates a prediction lexical probability matrix in an autoregressive manner. The predicted word probability matrix consists of the predicted probability distribution corresponding to each word position, i.e. ,in, , Indicates the first The predicted probability distribution of each word position. Indicates the total number of lexical units. Indicates the first The sequence of lexical units already generated before each lexical unit, Represents a probability generating function; Using the cross-entropy loss between the predicted lexical probability matrix and the true lexical target matrix as the training objective, the routing module parameters and the low-rank matrix groups of multiple low-rank adaptation experts are jointly optimized. The training objective is: in, The number of training samples. To predict the cross-entropy loss between the lexical probability matrix and the true lexical target matrix, Indicates the first The predicted word probability matrix of each training sample. This is the set of parameters for the low-rank output mapping matrix. The set of parameters for the low-rank input mapping matrix; During joint training, the original parameters of the pre-trained visual language large model are maintained. Freeze, only update routing module parameters and the set of parameters for the low-rank output mapping matrix and the set of parameters of the low-rank input mapping matrix The process continues until the preset training termination conditions are met, resulting in a trained routing module and multiple low-rank adaptation experts.

6. The multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation as described in claim 1, characterized in that, The specific steps of the environmental perception generation step are as follows: Step S141: In the real-world communication scenario to be perceived, the millimeter-wave / terahertz base station communicates with several communication nodes using a set of candidate transmit beam directions to obtain channel gain observation data for different positions of different beam directions; the channel gain observation data is arranged according to spatial position and beam direction to obtain the sparse channel gain tensor to be perceived. Step S142: Reconstruct the sparse channel gain tensor to be perceived using the channel gain reconstruction module to obtain the channel gain tensor. Then, perform beam direction slicing, stitching, normalization, and image processing on the channel gain tensor to obtain the channel map to be perceived. ; Step S143: Obtain the current environmental perception requirement, determine the semantic level corresponding to the current environmental perception requirement, and construct inference text prompt information. ; Step S144: Provide reasoning text prompts Convert to inference cue embedding vector The trained routing module is then input to obtain the inference routing weight vector. ; Step S145: Based on the inference route weight vector The parameter update set for multiple low-rank fitting experts that have completed training. By performing a weighted combination, the overall low-rank parameter update is obtained. = ; Step S146: Update the comprehensive low-rank parameters This is applied to a pre-trained visual language model to form an adapted visual language model. The parameters of the adapted visual language model are: and the channel map to be perceived and inference text prompts Input the adapted visual language large model to obtain the predicted word probability matrix. ; Generate environment perception results based on the predicted word probability matrix ; Step S147: Perform structured analysis on the environmental perception results to obtain multi-level environmental perception results; Step S148: Output multi-level environmental perception results and use the multi-level environmental perception results for subsequent communication decisions or environmental modeling.

7. The multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation as described in claim 1, characterized in that, The channel gain tensor obtained by channel gain reconstruction is Then the average channel strength at each grid position can be calculated as follows: in, Indicates grid position Average channel strength at that location Indicates the number of candidate beam directions. Indicates the first Okay, number Ranked in Channel gain in each transmit beam direction.

8. The multi-level environment perception method based on a large visual language model and millimeter-wave / terahertz channel observation as described in claim 1, characterized in that, When the visual language large model outputs the first When a beam direction is a candidate low-blocking beam direction, the average channel gain of all effective communication nodes under that beam direction can be calculated: in, Indicates the first The average channel gain of each effective communication node at the grid location under each candidate transmit beam direction. This represents the set of grid positions participating in the consistency check. This indicates the number of grid positions in the set.

Images