A Method and System for Environmental Semantics Extraction and Beamforming Based on Multimodal Large Model
By using cloud-based multimodal large model environmental semantic extraction and beamforming methods, combined with multimodal sensing data and edge-end dynamic tracking, the problems of high pilot overhead, insufficient real-time performance, and poor robustness of environmental perception schemes in traditional methods are solved, achieving low-latency and high-reliability beam control.
Patent Information
- Application Number
- CN202610971444.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-01
- Publication Date
- 2026-11-17
AI Technical Summary
In high-frequency, large-array, and high-speed mobile scenarios, traditional channel state acquisition and beamforming methods suffer from high pilot overhead, insufficient real-time performance, poor generalization and robustness of existing environmental perception schemes, and the inability of direct deployment of multimodal large models to meet the millisecond-level control requirements of wireless channels.
The system employs a cloud-based multimodal large model for environmental semantic extraction and beamforming. Through cross-attention fusion of multimodal sensor data, channel relationship reasoning, and dynamic tracking at the edge, it achieves low-latency and high-reliability beam control.
It achieves low-latency, high-reliability beamforming, improves channel prediction accuracy and link stability, adapts to complex and ever-changing wireless environments, and reduces pilot overhead and exploration complexity.
Smart Images

Figure CN123122604A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of wireless communication technology, artificial intelligence and environmental perception communication, and specifically relates to an environmental semantic extraction and beamforming method and system based on a multimodal large model. Background Technology
[0002] The sixth-generation mobile communication system is evolving towards ultra-large-scale multiple-input multiple-output (UMIM), millimeter-wave / terahertz bands, vehicle-to-everything (V2X) networks, and open radio access networks. In high-frequency, large-array, and high-speed mobile scenarios, the multipath structure of wireless channels is complex, and coherence time is significantly shortened. This places stringent demands on the accuracy of channel state information acquisition, beamforming response delay, and time-frequency resource overhead. Achieving low-overhead, high-reliability, and low-latency channel prediction and beam control is a core problem that urgently needs to be solved in this field.
[0003] Traditional channel state acquisition and beamforming rely on pilot measurement and exhaustive beam scanning. Channel estimation is achieved by periodically sending pilot symbols, followed by traversing candidate beams to select the optimal configuration. As antenna array size increases, the number of candidate beams grows, and terminal mobility improves, the proportion of time-frequency resources occupied by pilots increases significantly, and beam scanning control delay rises accordingly. This not only squeezes out effective data transmission resources but also makes it difficult to match the rapidly changing characteristics of the channel, easily leading to problems such as beam mismatch and link quality fluctuations.
[0004] Environmental sensing communication technology can utilize the physical correlation between the propagation environment and the channel to identify scene scatterers through multimodal data such as camera images, LiDAR point clouds, and integrated communication and sensing echoes, and deduce multipath structures to assist channel prediction, reducing dependence on pilot resources. However, existing solutions are mostly based on small-scale neural networks or single-modal features, with limited cross-modal fusion and global relationship modeling capabilities. In scenarios such as occlusion, noise interference, and scene migration, the accuracy of scatterer identification and channel prediction decreases significantly, and the generalization and robustness are insufficient. Summary of the Invention
[0005] Therefore, the present invention provides an environmental semantic extraction and beamforming method and system based on a multimodal large model to solve or partially solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A first aspect provides a method for environmental semantic extraction and beamforming based on a multimodal large model, comprising:
[0007] Multimodal sensing data of the target wireless environment is collected in the cloud, and the multimodal sensing data is time-synchronized and extrinsic parameter calibrated to obtain environmental observation results in a unified coordinate system; multiple candidate scatterers are detected from the environmental observation results, and the multi-source heterogeneous features of each candidate scatterer are extracted. Guided by the reliability weights corresponding to the modes, cross-attention fusion operation is performed on the multi-source heterogeneous features of the candidate scatterers, and the kinematic encoding and semantic attribute encoding of the corresponding scatterers are superimposed to generate the instantiated semantic representation of each scatterer. The instantiated semantic representations of all scatterers and the spatial context information of the transceiver are input into the channel relationship reasoning model. The relative contribution of each scatterer to the multipath channel is inferred through the spatial conditional self-attention mechanism to obtain the global channel state representation and the higher-layer beam steering strategy. The higher-layer beam steering strategy is encapsulated into a structured semantic strategy message and the semantic strategy message is sent to the edge side. After receiving the semantic policy message, the edge side uses the semantic policy message as a kinematic prior and spatial range constraint input to the dynamic tracking unit. Through multi-scale temporal tracking, the trajectory, birth and death state and phase evolution of the scatterer are updated iteratively on a cycle-by-cycle basis, and the instantaneous channel state information is reconstructed based on the tracking results. Based on the reconstructed instantaneous channel state information, and combined with the angle space constraints and confidence information in the semantic policy message, a lightweight reinforcement learning model adaptively shrinks or expands the beam search action space, outputs the optimal beamforming parameters, and completes physical layer beam control.
[0008] As a preferred scheme for environmental semantic extraction and beamforming method based on multimodal large model, the multimodal sensing data includes at least one of visual image, depth map, lidar point cloud, and integrated communication and sensing echo; the extracted multi-source heterogeneous features include visual semantic features, point cloud geometric features, reflection intensity features, and kinematic features.
[0009] As a preferred scheme for environmental semantic extraction and beamforming methods based on multimodal large models, the instantiated semantic representation of the scatterer is generated through a reliability-aware cross-attention mechanism, and the calculation method is as follows:
[0010] In the formula, For the first k The instantiation semantic representation of each scatterer is used to input the channel relationship reasoning model; , These are the feature vectors output by the visual encoder and the point cloud / geometric encoder, respectively, which can be extended into depth maps or integrated communication and sensing echo features. , These are the reliability weights for the visual modality and the point cloud / geometric modality, respectively. , , For learnable query, key, and value projection matrices; This is a cross-attention fusion function used to adaptively adjust the contribution ratio of each modality feature according to the modality reliability weight; The kinematic coding function is used to derive and predict displacement from the velocity and acceleration of the scattering body. For semantic attribute encoding functions, As a scatterer category, For scattering material or reflective properties; This is a layer normalization operation used to stabilize the numerical scale of features in different modes.
[0011] As a preferred scheme for environmental semantic extraction and beamforming methods based on multimodal large models, the channel relationship inference model calculates the path length, time delay, and Doppler frequency shift based on the spatial position of the transmitter and receiver relative to the scatterer when calculating the geometric parameters of the propagation path corresponding to the scatterer. The calculation method is as follows:
[0012]
[0013] In the formula, Location of the transmitter. Location of the receiving end. For the first k The position of each scatterer; For the first k The total path length of the scattering paths, This represents the propagation delay along the corresponding path; At the speed of light, The carrier wavelength; For the first k The Doppler frequency shift of a path is determined by the scatterer velocity, the transmitter-receiver velocity, and the reflection geometry. The channel relationship reasoning model employs a spatial conditional self-attention mechanism to calculate the channel contribution weights of each scatterer and generate a global channel state representation. The calculation method is as follows:
[0014]
[0015] In the formula, This is the spatial context for the transceiver, including position, velocity, array orientation, and antenna configuration information. For the first k A vector of geometric parameters for each candidate path; For the first k The relative contribution weights of each scatterer to the overall channel state; , , For query, key, and value mapping functions in spatial conditional self-attention; The feature dimension is used for numerical scaling of attention weights; This is a learnable mapping matrix used to map the aggregated scatterer relationship representation to a global channel state representation; It is a global channel state representation used to characterize the multipath structure, occlusion relationships, and spatial correlation of the current environment.
[0016] As a preferred scheme for environmental semantic extraction and beamforming method based on multimodal large model, the structured semantic policy message includes at least the following fields: policy number, cloud generation timestamp, policy validity duration, angle space mask, candidate path parameter set, semantic confidence, modal reliability vector and fault protection flag; wherein the candidate path parameter set includes the time delay, Doppler frequency shift, path gain and angle information of each candidate path; The semantic confidence score is calculated jointly by channel state representation and environmental observation, specifically as follows:
[0017] In the formula, The total number of tokens generated for the strategy sequence of the model; To generate a sequence Global channel state representation Environmental observation With prior information Under the condition, generate the first l The probability of each strategy token.
[0018] As an optimal solution for environmental semantic extraction and beamforming based on a multimodal large model, the edge side calculates the semantic policy effectiveness weight by jointly using semantic confidence and policy timeliness to regulate the influence of semantic priors on beamforming decisions. The specific calculation method is as follows:
[0019] In the formula, The confidence level of the current semantic strategy; , These are the lower and upper thresholds for semantic confidence, respectively. This is a truncation function used to restrict the calculation result to the interval between 0 and 1; This is the moment when cloud-based policies are generated. For execution time at the edge; This is the effective time constant of the policy, used to characterize the decay rate of the policy over time.
[0020] As an optimal solution for environmental semantic extraction and beamforming based on a multimodal large model, the edge side adaptively adjusts the expansion radius of the angle space mask according to the effectiveness weight of the semantic strategy to determine the effective action space for beam search. The specific calculation method is as follows:
[0021]
[0022] In the formula, This represents the minimum angular extension range under high confidence. The maximum angular expansion range when the confidence level is low or the strategy is nearing expiration; For the entire beam action set, The effective beam action space at the current moment; For action a Corresponding beam direction; For sector expansion operations of angular space masks; An angular space mask distributed from the cloud.
[0023] As an optimal solution for environmental semantic extraction and beamforming based on a multimodal large model, the specific method for reconstructing instantaneous channel state information at the edge is as follows:
[0024] In the formula, For the first k The path at time t The probability of existence is used for soft gating of unreliable paths; For the first k The complex gain of a path is determined by the material's reflection coefficient, path loss, and antenna pattern gain. This is the array response vector at the receiving end. , These are the azimuth and elevation angles of the arrival path, respectively. This is the conjugate transpose of the transmitter array response vector. , These are the azimuth and elevation angles of the starting path, respectively. For path propagation delay, For Doppler frequency shift, For carrier frequency.
[0025] As a preferred scheme for environmental semantic extraction and beamforming methods based on multimodal large models, the lightweight reinforcement learning model adopts an Actor-Critic architecture. The reward function is based on instantaneous spectral efficiency and deducts beam switching costs. The specific calculation method is as follows:
[0026] In the formula, For transmit signal-to-noise ratio, The number of spatial data streams; The noise covariance matrix is... The instantaneous channel matrix; To simulate the phase shift matrix, For digital baseband precoding matrix; For the cost of beam switching, This is the switching penalty coefficient, used to suppress frequent beam switching.
[0027] As a preferred scheme for environmental semantic extraction and beamforming methods based on multimodal large models, it also includes value assessment and event-triggered screening of samples in the link interaction process at the edge, and reporting the high-value samples obtained to the cloud; the cloud uses the high-value samples to fine-tune the parameters of the multimodal large model, so as to realize continuous optimization and performance evolution of cloud-edge collaboration.
[0028] As an optimal solution for environmental semantic extraction and beamforming methods based on multimodal large models, when evaluating the value of link interaction samples at the edge, the sample value score is calculated jointly based on multiple dimensions of indicators. The calculation method is as follows:
[0029] In the formula, For the first i The sample value score of each interaction trajectory; This represents the temporal difference error of the corresponding trajectory; For trajectory rewards, The reward is a moving average. This is the beam mismatch flag; it takes a value of 1 when there is beam mismatch, and 0 otherwise. The semantic confidence score of the corresponding trajectory; This is a modal anomaly flag; it takes a value of 1 when a modal anomaly occurs, and 0 otherwise. to These are the weighting coefficients for each evaluation dimension.
[0030] As a preferred scheme for environmental semantic extraction and beamforming methods based on multimodal large models, after receiving the semantic policy message at the edge, position extrapolation is performed based on the policy timestamp and scatterer kinematic parameters to compensate for transmission delay. The calculation method is as follows:
[0031] In the formula, For edge devices at execution time t The predicted first k The location of each scatterer; , , They are respectively the cloud at the time of generation Given the scatterer's position, velocity, and acceleration; the edge updates the angle sector, latency, and Doppler prior based on the predicted position to compensate for the impact of cloud inference and delivery latency.
[0032] As a preferred scheme for environmental semantic extraction and beamforming methods based on multimodal large models, the dynamic tracking is implemented based on multi-scale feature pyramids, global matching, local 4D related volume, and direction-aware Transformer. The local 4D related volume extracts local discriminative features in terms of time delay, angle, time, and feature level to reduce the probability of mismatch of scatterers. The direction-aware Transformer introduces a time decay factor to perform time-series modeling of cross-time slot trajectories and outputs trajectory increments and existence probability increments.
[0033] As a preferred scheme for environmental semantic extraction and beamforming methods based on multimodal large models, a fault protection mode is set on the edge side, which is triggered when any of the following conditions are met: If the semantic confidence is lower than the minimum threshold for several consecutive control cycles, the policy age exceeds the maximum effective duration, the moving average spectral efficiency is lower than the preset threshold, the number of beam mismatches exceeds the threshold within the time window, the single-mode or multi-mode reliability is lower than the corresponding threshold, or the cloud-edge link is disconnected or the policy message verification fails. After triggering fault protection, the edge device removes the semantic prior constraints from the cloud and performs standard beam scanning, historical best beam holding, or local neighborhood beam search.
[0034] Secondly, the present invention provides an environment semantic extraction and beamforming system based on a multimodal large model, employing the environment semantic extraction and beamforming method based on a multimodal large model according to the first aspect or any possible implementation thereof, including: The scatterer feature extraction unit is used to collect multimodal sensing data of the target wireless environment in the cloud, perform time synchronization and extrinsic parameter calibration on the multimodal sensing data to obtain environmental observation results in a unified coordinate system; detect multiple candidate scatterers from the environmental observation results, and extract the multi-source heterogeneous features of each candidate scatterer. The scatterer semantic representation unit is used to perform cross-attention fusion operation on the multi-source heterogeneous features of the candidate scatterers, guided by the reliability weights corresponding to the modes, and superimpose the kinematic encoding and semantic attribute encoding of the corresponding scatterer to generate an instantiated semantic representation for each scatterer. The guidance strategy generation unit is used to input the instantiated semantic representations of all scatterers and the spatial context information of the transceiver into the channel relationship reasoning model, and to reason about the relative contribution of each scatterer to the multipath channel through the spatial conditional self-attention mechanism, so as to obtain the global channel state representation and the higher-layer beam guidance strategy. The semantic policy encapsulation unit is used to encapsulate the higher-layer beam steering policy into a structured semantic policy message and send the semantic policy message to the edge side. The instantaneous channel state reconstruction unit is used to receive the semantic policy message at the edge side, and then input the semantic policy message as a kinematic prior and spatial range constraint into the dynamic tracking unit. The unit performs cycle-by-cycle iterative updates on the trajectory, birth and death state and phase evolution of the scatterer through multi-scale temporal tracking, and reconstructs the instantaneous channel state information based on the tracking results. The optimal beamforming parameter output unit is used to output the optimal beamforming parameters and complete the physical layer beam control based on the reconstructed instantaneous channel state information, combined with the angle space constraints and confidence information in the semantic policy message, through a lightweight reinforcement learning model to adaptively shrink or expand the beam search action space.
[0035] Thirdly, the present invention provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the processor executes the program or instructions, it implements the first aspect or any possible implementation thereof, the environmental semantic extraction and beamforming method based on a multimodal large model.
[0036] Fourthly, the present invention provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the first aspect or any possible implementation thereof, the environmental semantic extraction and beamforming method based on a multimodal large model.
[0037] The present invention has the following advantages: This invention employs a dual-timescale architecture that decouples cloud-based semantic inference from edge real-time control. The cloud performs environmental semantic extraction, channel relationship inference, and high-level policy generation at a low frequency, fully leveraging the strong semantic understanding and global relationship modeling capabilities of the multimodal large model. The edge performs lightweight tracking and beam control at millisecond or even sub-millisecond intervals, without bearing the inference load of a large model. This architecture overcomes the limitations of edge computing power on large model deployment and avoids the fundamental conflict between the long inference latency of large models and the short coherence time of wireless channels, introducing semantic-level environmental awareness while ensuring real-time beam control.
[0038] This invention employs a reliability-aware cross-attention mechanism to achieve multimodal feature fusion. It dynamically adjusts feature contribution weights based on the real-time reliability of each modality, such as vision, point cloud, and echo. When a modality fails due to occlusion, rain, fog, or noise, the fusion weights of the remaining effective modalities are automatically increased, preventing a sharp drop in perception performance caused by a single modality failure. Compared to traditional single-modal or simple stitching fusion schemes, this mechanism can still stably identify scatterers under occlusion, scene changes, and distribution migration scenarios, effectively reducing prediction errors of channel state information.
[0039] This invention designs a dynamic scatterer tracking mechanism at the edge, incorporating local 4D correlated volume and a direction-aware Transformer. This mechanism can iteratively update the trajectory, birth and death states, and phase evolution of the scatterer on a cycle-by-cycle basis. The 4D correlated volume can extract local discriminative features in complex multipath environments with high noise levels, reducing scatterer mismatches. The direction-aware Transformer introduces a time decay factor to achieve temporal modeling, ensuring the smoothness and accuracy of trajectory prediction under high-speed movement. The reconstructed instantaneous channel state information more closely matches the real channel, effectively reducing beam mismatch and improving link stability.
[0040] This invention uses the angle space mask output from the cloud as a priori constraint to guide the edge reinforcement learning agent to shrink the beam search space, pruning a large number of invalid beam sectors, significantly reducing the exploration complexity of reinforcement learning and accelerating training convergence. Simultaneously, by combining semantic confidence and policy timeliness to adaptively expand or shrink the action space, it avoids performance losses caused by invalid exploration and prevents local optima caused by semantic prior bias, ultimately achieving higher instantaneous spectral efficiency and more stable link quality.
[0041] This invention employs an event-triggered sample selection mechanism, using multi-dimensional indicators such as temporal difference error, reward mutation, beam mismatch, and modal anomalies to filter high-value feedback samples. Only the interaction trajectories with the highest information content are reported to the cloud, maximizing the sample utility of cloud-based fine-tuning within limited uplink bandwidth. The cloud continuously optimizes the multimodal large model through efficient parameter fine-tuning, ensuring long-term alignment between cloud semantic understanding and edge control strategies. System performance can continuously iterate and improve over deployment time.
[0042] This invention establishes a hierarchical control mechanism that jointly gates semantic confidence and policy timeliness, which can dynamically adjust the weight of the semantic policy's influence on beam decision based on the effectiveness of the semantic policy. At the same time, it designs multi-dimensional fault protection trigger conditions. When anomalies such as sensor failure, cloud-edge link interruption, policy expiration, or sudden drop in link quality occur, the edge end can automatically decouple the cloud semantic prior and fall back to the standard beam scanning or the historical best beam-keeping policy, ensuring the continuity and reliability of the communication link and improving the availability of the system in complex real-world environments. Attached Figure Description
[0043] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.
[0044] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.
[0045] Figure 1 This is a schematic diagram of the process of environmental semantic extraction and beamforming method based on multimodal large model provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the cloud-edge collaborative intelligent network architecture provided in an embodiment of the present invention; Figure 3 This is a diagram of the cloud-based multimodal large model architecture for semantic environment understanding and channel prediction provided in this embodiment of the invention; Figure 4 This is a schematic diagram of the edge-end dynamic scatterer cluster tracking and channel reconstruction module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the edge beamforming control closed loop based on MLLM-RL provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the system architecture for environmental semantic extraction and beamforming based on a multimodal large model provided in this embodiment of the invention; Figure 7 This is an electronic device architecture diagram provided in an embodiment of the present invention. Detailed Implementation
[0046] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0047] The sixth-generation (6G) mobile communication system is evolving towards ultra-large-scale multiple-input multiple-output (MIMO), millimeter-wave / terahertz bands, vehicle-to-everything (V2X) networks, and open radio access networks (O-RAN), placing more stringent demands on the accuracy of channel state information (CSI) acquisition, beamforming response latency, and resource overhead. In high-frequency, large-array, and high-speed mobile scenarios, the multipath structure of wireless channels is complex, and the coherence time is significantly shortened. Achieving accurate and low-latency channel prediction and beam control within limited time and frequency resources is one of the core technical challenges in the current wireless communication field.
[0048] Traditional channel state acquisition and beamforming methods mainly rely on pilot measurement and exhaustive beam scanning. Channel estimation is achieved by periodically transmitting pilot signals at the transceiver end, and then the optimal configuration is selected by traversing the candidate beam set. As the antenna array size increases, the number of candidate beams increases, and the terminal mobility speed increases, the time-frequency resources occupied by pilot symbols will increase significantly, and the control delay of beam scanning will also increase accordingly. This not only reduces the effective utilization of spectrum resources, but also makes it difficult to match the rapidly changing characteristics of the channel in high-speed mobile scenarios, and easily leads to problems such as beam mismatch and link quality fluctuations.
[0049] To reduce pilot overhead and beam search costs, environmental sensing communication technology has emerged. This technology leverages the strong correlation between communication channels and the physical propagation environment, using multimodal sensor data such as camera images, depth maps, LiDAR point clouds, and ISAC echoes to identify scatterers in a scene, including buildings, vehicles, pedestrians, and roadside facilities. This allows for the derivation of multipath propagation structures and beam directions, aiding in channel prediction and beamforming. However, most existing environmental sensing solutions are based on small-scale neural networks or single-modal features, resulting in limited feature extraction and relationship modeling capabilities. In scenarios involving occlusion, noise interference, scene switching, and data distribution migration, the accuracy of scatterer identification and channel prediction significantly decreases, exhibiting insufficient generalization ability and robustness, making it difficult to adapt to complex and ever-changing real-world wireless environments.
[0050] Multimodal large models possess cross-modal semantic understanding, global relational reasoning, and complex scene modeling capabilities, which theoretically can significantly improve the semantic depth of environmental perception and the accuracy of channel prediction. However, deploying multimodal large models directly at the edge and performing inference every transmission time interval (TTI) is subject to strict limitations in the computing power and memory of edge devices. Furthermore, the hundreds of milliseconds of inference latency of large models are fundamentally contradictory to the millisecond or even sub-millisecond control requirements of wireless channels, failing to meet the real-time requirements of physical layer beamforming. Currently, there is no mature technical solution that can effectively combine the strong semantic reasoning capabilities of cloud-based large models with the real-time control requirements of the edge, achieving both semantic-level environmental perception and channel reasoning while ensuring low latency and high reliability of beamforming.
[0051] In view of this, and based on the shortcomings of the existing technology, there is an urgent need to develop an environmental semantic extraction and beamforming method and system based on a multimodal large model. This would address the technical problems of high pilot overhead and insufficient real-time performance in traditional beamforming, poor generalization and robustness of existing environmental perception schemes, and the inability of direct deployment of multimodal large models to adapt to the millisecond-level control requirements of wireless channels. The following are the specific contents of the embodiments of the present invention.
[0052] See Figure 1This invention provides a method for environmental semantic extraction and beamforming based on a multimodal large model, comprising the following steps: S1. Collect multimodal sensing data of the target wireless environment in the cloud, synchronize the time and calibrate the external parameters of the multimodal sensing data to obtain environmental observation results in a unified coordinate system; detect multiple candidate scatterers from the environmental observation results and extract the multi-source heterogeneous features of each candidate scatterer. S2. Guided by the reliability weights corresponding to the modes, perform cross-attention fusion operation on the multi-source heterogeneous features of the candidate scatterers, and superimpose the kinematic encoding and semantic attribute encoding of the corresponding scatterers to generate the instantiated semantic representation of each scatterer. S3. Input the instantiated semantic representations of all scatterers and the spatial context information of the transceiver into the channel relationship reasoning model, and infer the relative contribution of each scatterer to the multipath channel through the spatial conditional self-attention mechanism to obtain the global channel state representation and the higher-layer beam steering strategy; encapsulate the higher-layer beam steering strategy into a structured semantic strategy message, and send the semantic strategy message to the edge side. S4. After receiving the semantic policy message, the edge side uses the semantic policy message as a kinematic prior and spatial range constraint input to the dynamic tracking unit. Through multi-scale temporal tracking, the trajectory, birth and death state and phase evolution of the scatterer are iteratively updated cycle by cycle, and the instantaneous channel state information is reconstructed based on the tracking results. S5. Based on the reconstructed instantaneous channel state information, and combined with the angle space constraints and confidence information in the semantic policy message, the optimal beamforming parameters are output and physical layer beam control is completed by adaptively shrinking or expanding the beam search action space through a lightweight reinforcement learning model.
[0053] In this embodiment, in step S1, the multimodal sensing data includes at least one of visual images, depth maps, LiDAR point clouds, and integrated communication-sensing echoes. The extracted multi-source heterogeneous features include visual semantic features, point cloud geometric features, reflection intensity features, and kinematic features. Visual images are rich in category, texture, and material information, and can identify scatterer types and reflection attributes. Depth maps and LiDAR point clouds provide high-precision three-dimensional geometric structures and spatial coordinates, supporting geometric calculations of the propagation path. Integrated communication-sensing echoes directly reflect the signal's reflection characteristics and time delay information, and have the highest correlation with channel physical characteristics. The corresponding features cover the four dimensions of semantics, geometry, reflection, and motion, respectively. When a single mode is blocked or fails due to noise interference, the other modes can still provide effective information, avoiding the collapse of sensing performance due to single-mode failure.
[0054] In this embodiment, in step S2, traditional feature fusion often uses fixed weight concatenation, which cannot adapt to dynamically changing modal quality. Reliability weight-guided cross-attention fusion dynamically allocates attention weights based on the real-time quality of each modality, giving higher contribution ratios to high-quality modalities and automatically reducing the weights of damaged modalities, achieving adaptive robust fusion. Superimposed kinematic coding injects the motion trend of the scatterer into the features, enabling the representation to have temporal predictive capabilities; superimposed semantic attribute coding maps discrete semantic information such as category and material into continuous features, achieving spatial alignment with perceptual features. The final generated instantiated semantic representation combines current features, motion patterns, and semantic attributes, providing comprehensive input for channel inference.
[0055] The instantiation semantic representation of the scatterer is generated through a reliability-aware cross-attention mechanism, and the calculation method is as follows:
[0056] In the formula, For the first k The instantiation semantic representation of each scatterer is used to input the channel relationship reasoning model; , These are the feature vectors output by the visual encoder and the point cloud / geometric encoder, respectively, which can be extended into depth maps or integrated communication and sensing echo features. , These are the reliability weights for the visual modality and the point cloud / geometric modality, respectively. , , For learnable query, key, and value projection matrices; This is a cross-attention fusion function used to adaptively adjust the contribution ratio of each modality feature according to the modality reliability weight; The kinematic coding function is used to derive and predict displacement from the velocity and acceleration of the scattering body. For semantic attribute encoding functions, As a scatterer category, For scattering material or reflective properties; This is a layer normalization operation used to stabilize the numerical scale of features in different modes.
[0057] Specifically, when visual modalities are affected by rain, fog, backlighting, or occlusion, the reliability weight... As the number of points decreases, the contribution of visual features to attention computation also decreases, and the model automatically relies more on point cloud geometric features; conversely, when the point cloud is sparse, the weight of visual semantic features increases. Kinematic encoding predicts displacement based on a uniformly accelerated motion model, mapping temporal motion information into feature vectors, enabling the representation to have temporal predictive capabilities; semantic attribute encoding transforms discrete category and material information into continuous vectors, achieving spatial alignment of multi-source features. Layer normalization is used to eliminate numerical scale differences between features of different modalities, preventing features with large magnitudes from dominating the fusion result and ensuring training stability.
[0058] In this embodiment, in step S3, when calculating the geometric parameters of the propagation path corresponding to the scatterer, the channel relationship inference model calculates the path length, time delay, and Doppler frequency shift based on the spatial position of the transceiver and the scatterer. The calculation method is as follows:
[0059]
[0060] In the formula, Location of the transmitter. Location of the receiving end. For the first k The position of each scatterer; For the first k The total path length of the scattering paths, This represents the propagation delay along the corresponding path; At the speed of light, The carrier wavelength; For the first k The Doppler frequency shift of a single scattering path is determined by the scatterer velocity, the transmitter-receiver velocity, and the reflection geometry. The propagation distance of a single scattering path is equal to the sum of the distances from the transmitter to the scatterer and from the scatterer to the receiver. The propagation delay is determined by the ratio of the path length to the speed of light and is the physical source of channel multipath delay spread. The Doppler frequency shift originates from the rate of change of path length caused by the relative motion between the transmitter / receiver and the scatterer. After wavelength normalization, the frequency offset is obtained and is a determining factor of the time-varying characteristics of the channel in high-speed moving scenarios. Introducing geometric priors to calculate path parameters can reduce the inference difficulty of large models while ensuring the physical consistency of channel prediction and avoiding results that violate propagation laws.
[0061] The channel relationship reasoning model employs a spatial conditional self-attention mechanism to calculate the channel contribution weights of each scatterer and generate a global channel state representation. The calculation method is as follows:
[0062]
[0063] In the formula, This is the spatial context for the transceiver, including position, velocity, array orientation, and antenna configuration information. For the first k A vector of geometric parameters for each candidate path; For the first k The relative contribution weights of each scatterer to the overall channel state; , , For query, key, and value mapping functions in spatial conditional self-attention; The feature dimension is used for numerical scaling of attention weights; This is a learnable mapping matrix used to map the aggregated scatterer relationship representation to a global channel state representation; It is a global channel state representation used to characterize the multipath structure, occlusion relationships, and spatial correlation of the current environment. q The function encodes the transmitter and receiver locations, speeds, and array configurations into query features. K The function integrates the semantic representation of the scatterer with its geometric parameters into key features. The dot product similarity between the two features reflects the importance of the scatterer to the current transmit / receive link. After softmax normalization, the contribution weights are obtained. Dividing by a scaling factor is used to avoid excessively large dot product values under high-dimensional features, which could lead to softmax gradient saturation. A global channel state representation is obtained through weighted summation and linear mapping, achieving the aggregation from discrete scatterers to global channel characteristics. This preserves the physical meaning of each path while forming a compact global feature representation.
[0064] In this embodiment, in step S3, the structured semantic policy message includes at least the following fields: policy number, cloud generation timestamp, policy validity period, angle space mask, candidate path parameter set, semantic confidence, modal reliability vector, and fault protection flag; wherein the candidate path parameter set includes the delay, Doppler shift, path gain, and angle information of each candidate path. The structured message is a standardized carrier for transmitting cloud semantic capabilities to the edge, realizing the transformation of unstructured large model output into executable parameters. The policy number is used for cloud-edge data alignment and feedback sample tracing; the timestamp and validity period together constitute the policy's timeliness attribute, supporting the edge end to calculate the policy aging degree; the angle space mask is used to constrain the beam search range of the edge end, serving as a carrier to reduce exploration overhead; the candidate path parameter set provides coarse-grained priors for multipath channels, providing initial values for edge channel reconstruction; the semantic confidence and modal reliability vector provide quantitative indicators of policy credibility for the edge end, supporting hierarchical control and adaptive adjustment; the fault protection flag is used for rapid degradation in abnormal scenarios, ensuring link reliability.
[0065] The semantic strategy output from the cloud can be written as follows:
[0066] In the formula, For angular space mask, This represents the number of valid candidate paths selected from the cloud. For target type, For semantic confidence, The number of tokens generated for the policy sequence is determined by the edge Actor-Critic agent. The agent combines these policies with local CSI statistics to form the state. and select action ,in To simulate the phase shift matrix, This is the digital baseband precoding matrix.
[0067] The semantic confidence level is calculated jointly by channel state representation and environmental observation, specifically as follows:
[0068] In the formula, The total number of tokens generated for the strategy sequence of the model; To generate a sequence Global channel state representation Environmental observation With prior information Under the condition, generate the first l The probability of each strategy token is calculated based on the autoregressive probability of each generated token, using the average probability of all generated tokens as the overall confidence level. The more certain the model is about the generation result, the higher the conditional probability of each token, and the higher the confidence level. When environmental observations are blurry or scatterer occlusion is severe, the uncertainty of generated tokens increases, and the confidence level decreases accordingly. This confidence level provides a quantifiable prior quality indicator for the edge device, allowing it to dynamically adjust its dependence on cloud strategies based on the confidence level, avoiding low-quality semantic priors from misleading beamforming decisions.
[0069] Specifically, the edge side calculates the semantic policy effectiveness weight by jointly using semantic confidence and policy timeliness to regulate the influence of semantic priors on beam decision-making. The specific calculation method is as follows:
[0070] In the formula, The confidence level of the current semantic strategy; , These are the lower and upper thresholds for semantic confidence, respectively. This is a truncation function used to restrict the calculation result to the interval between 0 and 1; This is the moment when cloud-based policies are generated. For execution time at the edge; The effective time constant of the policy is used to characterize the rate of policy decay over time. The confidence component is normalized to the 0-1 interval through linear mapping and truncation: below the lower limit... Time weight is 0, cloud strategies are not relied upon at all; exceeding the upper limit. The time-based weight is 1, indicating complete acceptance. The time-based component adopts an exponential decay model, where the weight decreases as the policy is generated further back, consistent with the physical laws governing environmental changes and policy gradual failure in high-speed mobile scenarios. The combined weight obtained by multiplying the two components achieves an adaptive adjustment logic where the impact of cloud-based policies is minimized for policies with lower quality and longer duration.
[0071] In this process, the edge side adaptively adjusts the expansion radius of the angle space mask according to the semantic strategy effectiveness weight to determine the effective action space for beam search. The specific calculation method is as follows:
[0072]
[0073] In the formula, This represents the minimum angular extension range under high confidence. The maximum angular expansion range when the confidence level is low or the strategy is nearing expiration; For the entire beam action set, The effective beam action space at the current moment; For action a Corresponding beam direction; For sector expansion operations of angular space masks; This is the angle space mask distributed from the cloud. When the validity weight... When the value approaches 1, the extended radius is close to its minimum, and the beam search is strictly limited to the narrow sector recommended by the cloud, minimizing exploration overhead; when During descent, the expansion radius increases linearly, and the search sector gradually widens, allowing for exploration in more directions at the edge and correcting biases from prior information in the cloud. Expanding the mask's sectors through an expansion operation ensures the continuity of the search space, avoids angular gaps, and ultimately ensures that the effective action space always revolves around the cloud-recommended range while preserving a fault-tolerant exploration space.
[0074] In this embodiment, the specific method for reconstructing the instantaneous channel state information on the edge side in step S4 is as follows:
[0075] In the formula, For the first k The path at time t The probability of existence is used for soft gating of unreliable paths; For the firstk The complex gain of a path is determined by the material's reflection coefficient, path loss, and antenna pattern gain. This is the array response vector at the receiving end. , These are the azimuth and elevation angles of the arrival path, respectively. This is the conjugate transpose of the transmitter array response vector. , These are the azimuth and elevation angles of the starting path, respectively. For path propagation delay, For Doppler frequency shift, Let be the carrier frequency. This formula, based on a geometric channel model, decomposes the MIMO channel matrix into a superposition of multiple independent scattering paths. Each path is composed of the array response, complex gain, time delay phase, and Doppler phase. The array response vector is determined by the antenna array's geometry and direction of arrival / departure, reflecting the spatial selectivity of the antenna array. The complex gain includes path loss and reflection loss, determining the path's power contribution. The time delay and Doppler terms characterize the path's time delay spread and time-varying phase characteristics, respectively. An existence probability is introduced. Soft gating is implemented to reduce the contribution weight of paths with low tracking confidence, thereby avoiding the pollution of channel reconstruction results by erroneous tracking paths and improving the robustness of CSI reconstruction under non-ideal tracking.
[0076] In this embodiment, traditional beamforming often relies on closed-loop computation based on perfect CSI, which leads to significant performance degradation in non-ideal CSI scenarios. In step S5, reinforcement learning is employed to continuously optimize the beam strategy through interaction in a dynamically time-varying channel. The lightweight network structure ensures that the inference latency at the edge meets real-time requirements. Introducing angle space constraints and confidence information to constrain the action space significantly reduces the exploration dimension of reinforcement learning, accelerates convergence, and improves the search efficiency for the optimal strategy. The adaptive action space mechanism balances exploration and utilization, quickly converging to the optimal solution when the semantic prior is reliable, and retaining exploration capabilities to avoid getting trapped in local optima when the prior is unreliable.
[0077] The lightweight reinforcement learning model adopts an Actor-Critic architecture, and its reward function is based on instantaneous spectral efficiency minus beam switching cost. The specific calculation method is as follows:
[0078] In the formula, For transmit signal-to-noise ratio, The number of spatial data streams; The noise covariance matrix is... The instantaneous channel matrix; To simulate the phase shift matrix, For digital baseband precoding matrix; For the cost of beam switching, A switching penalty coefficient is used to suppress frequent beam switching. The Actor-Critic architecture balances policy optimization and value assessment. The Actor network outputs beamforming actions, and the Critic network evaluates the value of the actions and provides feedback, making it suitable for optimization problems in continuous action spaces. The main body of the reward function is the instantaneous spectral efficiency of the MIMO system, derived based on Shannon's formula, directly guiding the agent to optimize towards increasing the transmission rate. A beam switching penalty term is added because frequent beam switching in real systems can lead to control signaling overhead, phase calibration errors, and link fluctuations. The penalty coefficient is used to mitigate these effects. By adjusting the switching frequency, a balance can be achieved between spectral efficiency and system stability, avoiding frequent beam switching by the agent in pursuit of instantaneous speed.
[0079] In one possible embodiment, it also includes S6, where the edge side performs value assessment and event-triggered screening of samples during the link interaction process, and reports the high-value samples obtained to the cloud; the cloud uses the high-value samples to fine-tune the parameters of the multimodal large model, so as to realize continuous optimization and performance evolution of cloud-edge collaboration.
[0080] Specifically, reporting all interactive samples consumes a significant amount of uplink bandwidth, and most regular samples contribute little to model optimization. Event-triggered filtering, by quantifying sample value and reporting only high-information-density samples, maximizes the benefits of cloud-based fine-tuning under bandwidth constraints. Continuous fine-tuning in the cloud based on real-world interactive samples continuously corrects cognitive biases of the large model regarding specific scenarios, ensuring that cloud semantic understanding continuously aligns with actual edge channel characteristics. This forms an evolutionary mechanism of cloud-based generation strategy – edge execution feedback – cloud-based optimization iteration, resulting in continuous improvement in system performance over deployment time.
[0081] In one possible embodiment, when the edge side evaluates the value of link interaction samples, it calculates the sample value score based on multiple dimensions of indicators. The calculation method is as follows:
[0082] In the formula, For the first i The sample value score of each interaction trajectory; This represents the temporal difference error of the corresponding trajectory; For trajectory rewards, The reward is a moving average. This is the beam mismatch flag; it takes a value of 1 when there is beam mismatch, and 0 otherwise. The semantic confidence score of the corresponding trajectory; This is a modal anomaly flag; it takes a value of 1 when a modal anomaly occurs, and 0 otherwise. to These are the weighting coefficients for each evaluation dimension. TD error reflects the prediction bias of the current value model; the larger the error, the more valuable the sample for correction. Reward deviations from the average level represent abnormal events, containing more unknown information. Beam mismatch samples directly correspond to policy failure scenarios and are extremely valuable for correcting beam strategies. Low-confidence samples correspond to scenarios where the model's understanding is weak, supplementing the model's knowledge gaps. Modal anomaly samples correspond to extreme conditions, improving the model's robustness. A comprehensive score is obtained by weighting and summing multiple dimensions, ensuring that the selected samples cover various high-value scenarios and improving the comprehensiveness and efficiency of fine-tuning.
[0083] In one possible embodiment, after receiving the semantic policy message at the edge, the position is extrapolated based on the policy timestamp and the kinematic parameters of the scatterer to compensate for the transmission delay. The calculation method is as follows:
[0084] In the formula, For edge devices at execution time t The predicted first k The location of each scatterer; , , They are respectively the cloud at the time of generation Given the scatterer's position, velocity, and acceleration, the edge device updates the angle sector, latency, and Doppler prior based on the predicted position to compensate for the impact of cloud inference and transmission latency. The cloud policy's generation and execution at the edge involve inference time and transmission latency. In high-speed moving scenarios, this latency can cause significant deviations in the scatterer's position, leading to beam pointing errors. This formula, based on a uniformly accelerated motion model, uses the velocity and acceleration parameters provided by the cloud to extrapolate the scatterer's position from the cloud generation time to the edge execution time, geometrically compensating for the positional error caused by latency. This computationally inexpensive approach can be completed quickly at the edge without additional computing power, making it an effective and cost-efficient means to improve the accuracy of policies in high-speed scenarios.
[0085] In one possible embodiment, the dynamic tracking is achieved based on a multi-scale feature pyramid, global matching, local 4D correlated volume, and a direction-aware Transformer. The local 4D correlated volume extracts local discriminative features in terms of time delay, angle, time, and feature level to reduce the probability of mismatches with scatterers. The direction-aware Transformer introduces a time decay factor to perform temporal modeling of cross-time slot trajectories and outputs trajectory increments and existence probability increments. The multi-scale feature pyramid can extract features at different resolutions, balancing the tracking requirements of large-scale motion and fine displacement. Global matching completes the coarse correlation of trajectories, solving the initial matching problem of scatterers across frames. The local 4D correlated volume calculates feature similarity in a four-dimensional space of time delay, angle, time, and feature, which can distinguish highly similar scattering paths in complex multipath environments and solve the mismatch problem caused by multipath interference. The direction-aware Transformer models the motion law of the trajectory through temporal attention, introduces a time decay factor so that the influence of recent time slots on the current prediction is greater than that of distant time slots, which conforms to the law of motion continuity. At the same time, it models the changes in birth and death states, realizing dynamic adaptation to the appearance and disappearance of scatterers, improving the tracking accuracy and stability in complex dynamic scenes.
[0086] In one possible embodiment, a fault protection mode is set at the edge, which is triggered when any of the following conditions are met: semantic confidence is lower than a minimum threshold for several consecutive control cycles, policy age exceeds the maximum effective duration, moving average spectral efficiency is lower than a preset threshold, beam mismatch count exceeds a threshold within a time window, single-mode or multi-mode reliability is lower than the corresponding threshold, cloud-edge link is disconnected, or policy message verification fails. After triggering fault protection, the edge device removes the cloud-based semantic prior constraints and performs standard beam scanning, historical best beam holding, or local neighborhood beam search. The fault protection mechanism is a safety net for system reliability, covering six types of abnormal scenarios: semantic quality degradation, policy expiration, link performance degradation, beam mismatch, sensor failure, and communication interruption, achieving full-dimensional anomaly detection. After triggering, the cloud-based semantic constraints are removed, and the system reverts to a traditional beam scheme that does not rely on the cloud, avoiding the continuous deterioration of communication performance due to invalid semantic priors and ensuring basic link connectivity. Multiple fallback schemes can be flexibly selected according to the degree of anomaly, balancing performance and overhead, ensuring that the system can maintain basic communication capabilities under any abnormal conditions, achieving degraded availability.
[0087] See Figure 2In one deployment scenario of the method of the present invention, the cloud-based multimodal large model is deployed as an rApp in the Service Management and Orchestration Layer (SMO) of O-RAN or the Non-Real-Time RAN Intelligent Controller (Non-RT RIC), which is responsible for environmental semantic understanding, channel relationship reasoning and policy generation; the edge-end dynamic tracker and Actor-Critic agent are deployed as xApps in the Near-Real-Time RAN Intelligent Controller (Near-RT RIC), and interact with O-CU / O-DU or the base station physical layer via the E2 interface.
[0088] Cameras, LiDAR, or ISAC sensing nodes upload compressed environmental observation data to the cloud via the O1 interface or a dedicated sensing interface. Instead of performing large model inference at every transmission interval, the cloud generates or updates semantic policies when significant environmental changes are detected, edge rewards decrease, or a preset period is reached. Preferably, the cloud inference frequency is 0.5 to 2 Hz, suitable for semantic updates at the hundreds of millisecond level; the edge caches this policy and performs beamforming actions in conjunction with local CSI within each near real-time control cycle.
[0089] This deployment method concentrates the computational load of large models in the cloud, while the edge only needs to perform lightweight tracking and reinforcement learning inference. For example, the cloud can use multimodal large models with more than one billion parameters, while the edge Actor-Critic network can use a three-layer multilayer perceptron with about 0.2 million parameters, thereby meeting the sub-millisecond inference requirements of edge hardware.
[0090] like Figure 3 As shown, the cloud first performs time synchronization, coordinate transformation, and instance segmentation on camera images, depth maps, and LiDAR point clouds. For road intersection scenes, the instance segmentation model can identify scattering objects such as vehicles, pedestrians, traffic light poles, building walls, road signs, and guardrails, and calculate their 3D bounding boxes, centroid coordinates, velocity, acceleration, and visibility.
[0091] The multimodal scatterer fusion module constructs a scatterer cluster instance token. Modal reliability is affected when visual images are impacted by rain, fog, backlighting, or occlusion. As the threshold decreases, cross-attention automatically increases the weight of LiDAR geometric features or ISAC echo features; when the point cloud is sparse but the visual categories are clear, it increases the contribution of visual semantics to material properties and reflectance properties.
[0092] The channel relationship modeling module calculates the candidate propagation path parameters for each scatterer and obtains the channel state token through spatial conditional self-attention. The cloud-based multimodal large model generates a structured policy message based on the channel state token. This message contains at least: an angle space mask. Candidate path delay Candidate path Doppler Path gain Target Category Semantic confidence And the effective timestamp of the strategy.
[0093] When the strategy is deployed, the cloud transforms the unstructured environment understanding results into a set of parameters that can be resolved at the edge. For example, when the target vehicle is located northeast of the roadside unit and the main reflection path comes from the vehicle body and building walls, the cloud strategy provides the corresponding angular sector, nominal time delay, and Doppler range, which the edge device uses to narrow the beam search space.
[0094] like Figure 4 As shown, the edge device treats the cloud strategy as a coarse-grained semantic prior, rather than an immutable physical layer instruction. The edge device first maps the time-varying channel sampling into a multi-scale feature pyramid, and then performs global matching on the candidate positions of the query scatterer in subsequent time slots to obtain the initial trajectory and existence probability.
[0095] To address the issue of local mismatches caused by complex multipath propagation and noise, a local 4D correlation volume is constructed within the neighborhood of the initial trajectory at the edge. This 4D correlation volume calculates the similarity between the query trajectory point and the neighborhood candidate points using time delay, angle, time, and feature level as dimensions, and extracts local discriminative features through multi-layer 2D convolution, group normalization, and ReLU activation function.
[0096] Subsequently, the orientation-aware Transformer performs global temporal modeling of the cross-time-slot trajectory. Its relative position offset distinguishes between historical time slots and future predicted time slots, and a time decay factor is introduced to gradually reduce the influence of distant time slots on the current prediction. The Transformer outputs trajectory increments and existence probability increments, and after multiple iterations, a smooth scatterer trajectory, birth and death states, and phase evolution are obtained.
[0097] The instantaneous CSI is reconstructed at the edge. The trajectory delay change is used to estimate the geometric distance, the angle change is used to estimate the departure angle and arrival angle, the existence probability and power characteristics jointly determine the path amplitude, and the Doppler and reflection properties are used to correct the time-varying phase.
[0098] like Figure 5 As shown, the edge-side Actor-Critic agent will state... Defined as the prediction channel matrix Angular space mask Candidate Doppler and delay, semantic confidence A concatenated vector of received signal-to-noise ratio, historical spectral efficiency, and recent beam switching information. Action Including simulated phase shift matrix Digital baseband precoding matrix And the Top-k candidate beam set.
[0099] Angle space mask during action selection Clearly ineffective beam sectors are relegated to low priority or made unselectable, allowing the Actor network to focus on exploring directions relevant to effective scatterers. Semantic confidence. Used to adjust the weight of cloud guidance in the reward function: when At higher elevations, the edge edge more strongly follows the cloud angle prior. Expand the exploration range at the edge when the reward is low or the link reward continues to decline.
[0100] After beamforming is performed on the physical link, the edge calculates the instantaneous spectral efficiency reward, and the Critic network calculates the TD error. The Actor network updates the policy distribution according to the TD error, and the Critic network updates by minimizing the value estimation error. The edge also maintains an experience replay memory, and selects representative samples such as high reward, low reward, occlusion changes, and beam mismatch according to an event-triggered mechanism to upload to the cloud.
[0101] When a serious sensor failure, cloud-edge link interruption, or expired semantic policy is detected, the edge device enters fault protection mode, temporarily decoupling the cloud semantic module and reverting to standard beam scanning or the historical best beam-keeping strategy to ensure communication continuity. After normal operation is restored, the edge device reconnects to the cloud semantic policy and continues closed-loop learning.
[0102] In a preferred embodiment, each time the cloud sends a semantic policy message, it carries a policy number, generation timestamp, validity period, angle space mask, candidate path parameters, semantic confidence, and modal reliability vector. After receiving the policy, the edge device first calculates the semantic policy validity weight. Then determine the radius of the angle mask expansion.
[0103] For example, when the confidence level in the cloud is high and the policy has just been issued, Approaching 1, the edge selects candidate beams only within a narrow-angle sector indicated by the cloud; when rapid vehicle movement leads to increased policy age, or when visual modal confidence decreases due to occlusion, As the trend declines, the edge automatically expands the candidate sectors and relies more on local CSI statistics and historical link quality. If... If the efficiency remains below the threshold for an extended period or the moving average spectral efficiency continues to decline, the edge device will enter fault protection mode.
[0104] In fault protection mode, the edge no longer strictly adheres to cloud-based semantic masks. Instead, it employs standard beam scanning, historical best beam holding, or local neighborhood beam search to ensure link continuity. Trajectory samples generated during fault protection are marked as high-value feedback samples and prioritized for uploading to the cloud for model fine-tuning and policy correction.
[0105] To illustrate the technical effects of this invention, an embodiment based on a publicly available simulation scenario is provided. This embodiment uses the Urban Intersection scenario from the SynthSoM dataset, with Roadside Units (RSUs) configured as an 8×8 uniform planar array, operating in the 28 GHz millimeter-wave band. Inputs include RSU-view RGB images, depth maps, LiDAR point clouds, and corresponding real channel information. The perception module is trained for 100 epochs, followed by 450 episodes of shared edge RL agent interaction with the environment. Comparison methods include a single-modal LLM baseline using only text CSI cues, and an LSTM baseline based solely on historical signal time series predictions. The values in Table 1 below represent approximate simulation results corresponding to curve readings, used to illustrate performance trends.
[0106] Table 1. Approximate simulation results corresponding to curve readings
[0107] As shown in Table 1, at a low signal-to-noise ratio of -10 dB, the NMSE of this invention is approximately 0.75, lower than the single-modal LLM baseline of approximately 0.95 and the LSTM baseline of approximately 1.20; at a high signal-to-noise ratio of 20 dB, the NMSE of this invention is approximately 0.08, still lower than both baselines. This indicates that multimodal environmental semantics can provide stable out-of-band information even when historical CSI is contaminated by noise.
[0108] Table 2. Convergence Results of Spectral Efficiency
[0109] Referring to Table 2, in terms of spectral efficiency convergence, this invention achieves approximately 15 bps / Hz in the early stages of training, while the two baselines are approximately 4 bps / Hz; after approximately 150 episodes, this invention tends to stabilize, with an average spectral efficiency of approximately 21 to 22 bps / Hz. This is because the cloud-based angle space mask can prune a large number of invalid beam sectors, reducing the exploration complexity of the edge RL agent.
[0110] Table 3 Ablation Results
[0111] Referring to Table 3, the ablation results show that the complete scheme achieves approximately 32.5 bps / Hz at 40 dB, which is higher than approximately 29.8 bps / Hz with the channel relationship modeling module removed, approximately 26.0 bps / Hz with simple splicing fusion, and approximately 23.0 bps / Hz with pure edge baseline. This demonstrates that both the cross-attention fusion and channel relationship modeling modules in the multimodal scatterer fusion module substantially contribute to improving array gain and avoiding beam mismatch.
[0112] It should be noted that the method of this embodiment can also be applied to distributed scenarios, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the environmental semantic extraction and beamforming method based on a multimodal large model.
[0113] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0114] See Figure 6 This invention also provides an environment semantic extraction and beamforming system based on a multimodal large model, employing the above embodiments or any possible implementation thereof, including: The scatterer feature extraction unit 100 is used to collect multimodal sensing data of the target wireless environment in the cloud, perform time synchronization and extrinsic parameter calibration on the multimodal sensing data, and obtain environmental observation results in a unified coordinate system; detect multiple candidate scatterers from the environmental observation results, and extract the multi-source heterogeneous features of each candidate scatterer. The scatterer semantic representation unit 200 is used to perform cross-attention fusion operation on the multi-source heterogeneous features of the candidate scatterers guided by the reliability weights corresponding to the modes, and superimpose the kinematic encoding and semantic attribute encoding of the corresponding scatterer to generate an instantiated semantic representation for each scatterer. The guidance strategy generation unit 300 is used to input the instantiated semantic representations of all scatterers and the spatial context information of the transceiver into the channel relationship reasoning model, and to reason about the relative contribution of each scatterer to the multipath channel through the spatial conditional self-attention mechanism, so as to obtain the global channel state representation and the higher-layer beam guidance strategy. The semantic policy encapsulation unit 400 is used to encapsulate the higher-layer beam steering policy into a structured semantic policy message and send the semantic policy message to the edge side. The instantaneous channel state reconstruction unit 500 is used to receive the semantic policy message at the edge side, input the semantic policy message as kinematic prior and spatial range constraint into the dynamic tracking unit, and perform cycle-by-cycle iterative updates on the trajectory, birth and death state and phase evolution of the scatterer through multi-scale temporal tracking, and reconstruct the instantaneous channel state information based on the tracking results. The optimal beamforming parameter output unit 600 is used to output the optimal beamforming parameters and complete the physical layer beam control based on the reconstructed instantaneous channel state information, combined with the angle space constraints and confidence information in the semantic policy message, through a lightweight reinforcement learning model to adaptively shrink or expand the beam search action space.
[0115] The system described above is used to implement the environmental semantic extraction and beamforming method based on the multimodal large model in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0116] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the environmental semantic extraction and beamforming method based on a multimodal large model as described in any of the above embodiments.
[0117] Figure 7 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 710, a memory 720, an input / output interface 730, a communication interface 740, and a bus 750. The processor 710, memory 720, input / output interface 730, and communication interface 740 are interconnected internally via the bus 750.
[0118] The processor 710 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0119] The memory 720 can be implemented in the form of ROM (Read-Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.
[0120] The input / output interface 730 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.
[0121] The communication interface 740 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0122] Bus 750 includes a pathway for transmitting information between various components of the device, such as processor 710, memory 720, input / output interface 730, and communication interface 740.
[0123] It should be noted that although the above-described device only shows the processor 710, memory 720, input / output interface 730, communication interface 740, and bus 750, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.
[0124] The electronic devices described above are used to implement the environmental semantic extraction and beamforming methods based on multimodal large models in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0125] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the environmental semantic extraction and beamforming method based on a multimodal large model as described in any of the above embodiments.
[0126] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0127] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the environmental semantic extraction and beamforming method based on a multimodal large model as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0128] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity.
[0129] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of the invention, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of the invention, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of the invention will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of the invention, it will be apparent to those skilled in the art that the embodiments of the invention may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.
[0130] Although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., DRAM) may use the embodiments discussed.
[0131] The embodiments of this invention are intended to cover all such substitutions, modifications, and variations falling within the scope of the claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this invention should be included within the scope of protection of this invention.
Claims
1. A method for environmental semantic extraction and beamforming based on a multimodal large model, characterized in that, include: Multimodal sensing data of the target wireless environment is collected in the cloud, and the multimodal sensing data is time-synchronized and extrinsic parameter calibrated to obtain environmental observation results in a unified coordinate system; multiple candidate scatterers are detected from the environmental observation results, and the multi-source heterogeneous features of each candidate scatterer are extracted. Guided by the reliability weights corresponding to the modes, cross-attention fusion operation is performed on the multi-source heterogeneous features of the candidate scatterers, and the kinematic encoding and semantic attribute encoding of the corresponding scatterers are superimposed to generate the instantiated semantic representation of each scatterer. The instantiated semantic representations of all scatterers and the spatial context information of the transceiver are input into the channel relationship reasoning model. The relative contribution of each scatterer to the multipath channel is inferred through the spatial conditional self-attention mechanism to obtain the global channel state representation and the higher-layer beam steering strategy. The higher-layer beam steering strategy is encapsulated into a structured semantic strategy message and the semantic strategy message is sent to the edge side. After receiving the semantic policy message, the edge side uses the semantic policy message as a kinematic prior and spatial range constraint input to the dynamic tracking unit. Through multi-scale temporal tracking, the trajectory, birth and death state and phase evolution of the scatterer are updated iteratively on a cycle-by-cycle basis, and the instantaneous channel state information is reconstructed based on the tracking results. Based on the reconstructed instantaneous channel state information, and combined with the angle space constraints and confidence information in the semantic policy message, a lightweight reinforcement learning model adaptively shrinks or expands the beam search action space, outputs the optimal beamforming parameters, and completes physical layer beam control.
2. The environmental semantic extraction and beamforming method based on a multimodal large model according to claim 1, characterized in that, The multimodal sensing data includes at least one of visual images, depth maps, lidar point clouds, and integrated communication and sensing echoes; the extracted multi-source heterogeneous features include visual semantic features, point cloud geometric features, reflection intensity features, and kinematic features.
3. The environmental semantic extraction and beamforming method based on a multimodal large model according to claim 1, characterized in that, The instantiation semantic representation of the scatterer is generated by fusing through a reliability-aware cross-attention mechanism, and the calculation method is as follows: ; In the formula, For the first k The instantiation semantic representation of each scatterer is used to input the channel relationship reasoning model; , These are the feature vectors output by the visual encoder and the point cloud / geometric encoder, respectively, which can be extended into depth maps or integrated communication and sensing echo features. , These are the reliability weights for the visual modality and the point cloud / geometric modality, respectively. , , For learnable query, key, and value projection matrices; This is a cross-attention fusion function used to adaptively adjust the contribution ratio of each modality feature according to the modality reliability weight; The kinematic coding function is used to derive and predict displacement from the velocity and acceleration of the scattering body. For semantic attribute encoding functions, As a scatterer category, For scattering material or reflective properties; This is a layer normalization operation used to stabilize the numerical scale of features in different modes.
4. The environmental semantic extraction and beamforming method based on a multimodal large model according to claim 1, characterized in that, When calculating the geometric parameters of the propagation path corresponding to the scatterer, the channel relationship inference model calculates the path length, time delay, and Doppler frequency shift based on the spatial position of the transceiver and the scatterer. The calculation method is as follows: ; ; In the formula, Location of the transmitter. Location of the receiving end. For the first k The position of each scatterer; For the first k The total path length of the scattering paths, This represents the propagation delay along the corresponding path; At the speed of light, The carrier wavelength; For the first k The Doppler frequency shift of a path is determined by the scatterer velocity, the transmitter-receiver velocity, and the reflection geometry. The channel relationship reasoning model employs a spatial conditional self-attention mechanism to calculate the channel contribution weights of each scatterer and generate a global channel state representation. The calculation method is as follows: ; ; In the formula, This is the spatial context for the transceiver, including position, velocity, array orientation, and antenna configuration information. For the first k A vector of geometric parameters for each candidate path; For the first k The relative contribution weights of each scatterer to the overall channel state; , , For query, key, and value mapping functions in spatial conditional self-attention; The feature dimension is used for numerical scaling of attention weights; This is a learnable mapping matrix used to map the aggregated scatterer relationship representation to a global channel state representation; It is a global channel state representation used to characterize the multipath structure, occlusion relationships, and spatial correlation of the current environment.
5. The environmental semantic extraction and beamforming method based on a multimodal large model according to claim 1, characterized in that, The structured semantic policy message includes at least the following fields: policy number, cloud generation timestamp, policy validity period, angle space mask, candidate path parameter set, semantic confidence, modal reliability vector, and fault protection flag. The candidate path parameter set includes the time delay, Doppler frequency shift, path gain, and angle information of each candidate path; The semantic confidence score is calculated jointly by channel state representation and environmental observation, specifically as follows: ; In the formula, The total number of tokens generated for the strategy sequence of the model; To generate a sequence Global channel state representation Environmental observation With prior information Under the condition, generate the first l The probability of each strategy token; On the edge side, the semantic policy effectiveness weight is calculated jointly by semantic confidence and policy timeliness to regulate the influence of semantic prior on beam decision. The specific calculation method is as follows: ; In the formula, The confidence level of the current semantic strategy; , These are the lower and upper thresholds for semantic confidence, respectively. This is a truncation function used to restrict the calculation result to the interval between 0 and 1; This is the moment when cloud-based policies are generated. For execution time at the edge; The effective time constant of the policy is used to characterize the decay rate of the policy over time. The edge side adaptively adjusts the expansion radius of the angle space mask according to the semantic strategy effectiveness weight to determine the effective action space of beam search. The specific calculation method is as follows: ; ; In the formula, This represents the minimum angular extension range under high confidence. The maximum angular expansion range when the confidence level is low or the strategy is nearing expiration; For the entire beam action set, The effective beam action space at the current moment; For action a Corresponding beam direction; For sector expansion operations of angular space masks; An angular space mask distributed from the cloud.
6. The environmental semantic extraction and beamforming method based on a multimodal large model according to claim 1, characterized in that, The specific method for reconstructing instantaneous channel state information at the edge is as follows: ; In the formula, For the first k The path at time t The probability of existence is used for soft gating of unreliable paths; For the first k The complex gain of a path is determined by the material's reflection coefficient, path loss, and antenna pattern gain. This is the array response vector at the receiving end. , These are the azimuth and elevation angles of the arrival path, respectively. This is the conjugate transpose of the transmitter array response vector. , These are the azimuth and elevation angles of the starting path, respectively. For path propagation delay, For Doppler frequency shift, For carrier frequency.
7. The environmental semantic extraction and beamforming method based on a multimodal large model according to claim 1, characterized in that, The lightweight reinforcement learning model adopts an Actor-Critic architecture. The reward function is based on instantaneous spectral efficiency and deducts beam switching costs. The specific calculation method is as follows: ; In the formula, For transmit signal-to-noise ratio, The number of spatial data streams; The noise covariance matrix is... The instantaneous channel matrix; To simulate the phase shift matrix, For digital baseband precoding matrix; For the cost of beam switching, This is the switching penalty coefficient, used to suppress frequent beam switching.
8. The environmental semantic extraction and beamforming method based on a multimodal large model according to claim 1, characterized in that, It also includes value assessment and event-triggered screening of samples during the link interaction process at the edge, and reporting the high-value samples to the cloud; the cloud uses the high-value samples to fine-tune the parameters of the multimodal large model, so as to realize continuous optimization and performance evolution of cloud-edge collaboration; When evaluating the value of link interaction samples at the edge, the sample value score is calculated jointly based on multiple dimensions of indicators. The calculation method is as follows: ; In the formula, For the first i The sample value score of each interaction trajectory; This represents the temporal difference error of the corresponding trajectory; For trajectory rewards, The reward is a moving average. This is the beam mismatch flag; it takes a value of 1 when there is beam mismatch, and 0 otherwise. The semantic confidence score of the corresponding trajectory; This is a modal anomaly flag; it takes a value of 1 when a modal anomaly occurs, and 0 otherwise. to These are the weighting coefficients for each evaluation dimension; After receiving the semantic policy message, the edge side extrapolates the position based on the policy timestamp and the kinematic parameters of the scatterer to compensate for the transmission delay. The calculation method is as follows: ; In the formula, For edge devices at execution time t The predicted first k The location of each scatterer; , , They are respectively the cloud at the time of generation Given the scatterer's position, velocity, and acceleration; the edge updates the angle sector, latency, and Doppler prior based on the predicted position to compensate for the impact of cloud inference and delivery latency.
9. The environmental semantic extraction and beamforming method based on a multimodal large model according to claim 1, characterized in that, The dynamic tracking is based on a multi-scale feature pyramid, global matching, local 4D related volume, and orientation-aware Transformer. The local 4D related volume extracts local discriminative features in terms of time delay, angle, time, and feature level to reduce the probability of mismatch with scatterers. The orientation-aware Transformer introduces a time decay factor to perform time-series modeling of cross-time slot trajectories and outputs trajectory increments and existence probability increments. The edge side is configured with a fault protection mode, which is triggered when any of the following conditions are met: If the semantic confidence is lower than the minimum threshold for several consecutive control cycles, the policy age exceeds the maximum effective duration, the moving average spectral efficiency is lower than the preset threshold, the number of beam mismatches exceeds the threshold within the time window, the single-mode or multi-mode reliability is lower than the corresponding threshold, or the cloud-edge link is disconnected or the policy message verification fails. After triggering fault protection, the edge device removes the semantic prior constraints from the cloud and performs standard beam scanning, historical best beam holding, or local neighborhood beam search.
10. An environment semantic extraction and beamforming system based on a multimodal large model, employing the environment semantic extraction and beamforming method based on a multimodal large model as described in any one of claims 1 to 9, characterized in that, include: The scatterer feature extraction unit is used to collect multimodal sensing data of the target wireless environment in the cloud, perform time synchronization and extrinsic parameter calibration on the multimodal sensing data to obtain environmental observation results in a unified coordinate system; detect multiple candidate scatterers from the environmental observation results, and extract the multi-source heterogeneous features of each candidate scatterer. The scatterer semantic representation unit is used to perform cross-attention fusion operation on the multi-source heterogeneous features of the candidate scatterers, guided by the reliability weights corresponding to the modes, and superimpose the kinematic encoding and semantic attribute encoding of the corresponding scatterer to generate an instantiated semantic representation for each scatterer. The guidance strategy generation unit is used to input the instantiated semantic representations of all scatterers and the spatial context information of the transceiver into the channel relationship reasoning model, and to reason about the relative contribution of each scatterer to the multipath channel through the spatial conditional self-attention mechanism, so as to obtain the global channel state representation and the higher-layer beam guidance strategy. The semantic policy encapsulation unit is used to encapsulate the high-layer beam steering policy into a structured semantic policy message and send the semantic policy message to the edge side. The instantaneous channel state reconstruction unit is used to receive the semantic policy message at the edge side, and then input the semantic policy message as a kinematic prior and spatial range constraint into the dynamic tracking unit. The unit performs cycle-by-cycle iterative updates on the trajectory, birth and death state and phase evolution of the scatterer through multi-scale temporal tracking, and reconstructs the instantaneous channel state information based on the tracking results. The optimal beamforming parameter output unit is used to output the optimal beamforming parameters and complete the physical layer beam control based on the reconstructed instantaneous channel state information, combined with the angle space constraints and confidence information in the semantic policy message, through a lightweight reinforcement learning model to adaptively shrink or expand the beam search action space.
