Multi-modal cognitive integration method based on energy constraint in high mobility network

Abstract

This invention belongs to the field of wireless communication and sensing integration, specifically relating to an energy-constrained multimodal sensing integration method in high mobility networks. This invention introduces a virtual energy deficit queue, which transforms the long-term average energy budget constraint into a queue stability problem that can be handled online. This allows the system to automatically reduce the visual activation frequency or relax the visual update requirements of non-critical time slots when the energy budget is tight, and enhance visual calibration when the energy deficit is small, thereby achieving long-term sustainable operation. This invention employs a heterogeneous hybrid expert structure to separate long-term time scheduling from instantaneous spatial phase mapping. LSTM experts are adept at handling AoI and queue evolution, while MLP experts are adept at handling non-convex constant-mode beam mapping. A strict gradient isolation mechanism reduces gradient conflicts between sensing accuracy, energy saving, and spatial gain objectives in a single network.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication and sensing integration, specifically relating to a multimodal sensing integration method based on energy constraints in high mobility networks. Background Technology

[0002] With the development of sixth-generation mobile communication networks, intelligent transportation, and vehicle-road cooperative systems, millimeter-wave and terahertz frequency bands can provide ultra-high data transmission rates. However, they suffer from high path loss and weak diffraction capabilities, requiring base stations to be equipped with large-scale antenna arrays to form highly directional narrow beams. In high-speed vehicle scenarios, the vehicle's position and angle change rapidly, and narrow beams are extremely sensitive to spatial misalignment. Frequent beam training and beam recovery significantly increase system overhead.

[0003] Multimodal sensing integration technology, by combining millimeter-wave radar, RGB cameras, and communication links, can proactively acquire road environment and vehicle spatial status during communication. The radar can continuously provide low-power kinematic estimates such as angle of arrival and distance, while the RGB camera can provide high-precision visual semantic context and vehicle position calibration. However, visual data processing typically requires GPUs or dedicated accelerators for target detection and localization, resulting in high computational power consumption and edge processing latency.

[0004] Existing methods often only consider whether to activate the visual sensor or only optimize beamforming, making it difficult to simultaneously address three interdependent problems: First, queuing of visual tasks leads to an increase in the age of semantic information, which in turn causes angle uncertainty and beam misalignment; second, edge computing capabilities are limited, and continuous visual processing can cause instability in the computation queue; third, millimeter-wave analog phase shifters have strict constant mode constraints, and beamforming variables exhibit a non-convex structure. Traditional single-network reinforcement learning or static fusion methods are prone to gradient conflicts between long-term temporal scheduling and instantaneous spatial mapping, leading to energy consumption exceeding budget, queue divergence, or decreased perception and tracking accuracy.

[0005] Therefore, it is necessary to propose a multimodal synesthetic optimization method that can explicitly characterize the relationship between semantic information age, edge computing queue, long-term energy budget and physical layer angle tracking error, and solve the long-term scheduling and instantaneous non-convex phase mapping problems through heterogeneous network structures. Summary of the Invention

[0006] This invention aims to address the challenges faced by multimodal sensory integration systems in highly mobile V2I networks, including high energy consumption in visual processing, information staleness due to visual task queuing, difficulty in guaranteeing long-term energy budgets, and the challenge of real-time solution of hybrid discrete-continuous action spaces. This invention constructs a physical coupling relationship between vectorized semantic information age and the posterior Cramer-Rao bound of angles, introduces a real computing queue and a virtual energy deficit queue, and designs an LD-H-MoE heterogeneous hybrid expert network. This enables the system to achieve lower tracking PCRB and stronger radio frequency robustness while ensuring the stability of the edge computing queue and long-term energy budget.

[0007] The technical solution of the present invention is as follows:

[0008] A multimodal sensing integration method based on energy constraints in high mobility networks, wherein the high mobility network consists of a base station and K high mobility users, and the base station includes an M-element antenna array, a millimeter-wave radar module, an RGB camera module, an edge computing server, and a multimodal scheduling controller. The base station operates in discrete time slots. Simultaneously, downlink communication, radar detection, vision scheduling, and edge computing resource allocation are performed. The system model is divided into a communication link and a sensing link, where the communication link is used to complete downlink data transmission from the base station to the highly mobile user. The corresponding millimeter-wave channel is defined as the radial distance to the highly mobile user. Angle of arrival or angle of departure Path loss Propagation delay and array guide vector Related functions 'n' refers to the nth time slot, and the subscript 'k' refers to the kth high-mobility user. The perception link is used to acquire and update distance, angle, speed, and visual semantic information of highly mobile users. The perception link includes an active perception link based on a millimeter-wave radar module and a passive perception link based on an RGB camera module. The active perception link operates continuously with low power consumption, acquiring radial distance, radial speed, and coarse angle estimates. The passive perception link needs to be activated by a multimodal scheduling controller, setting scheduling variables. The vision sensor is activated to acquire tangential position, angle calibration, and semantic context. The two types of perception results are fused in the edge computing server to form vectorized semantic information such as age, spatial uncertainty, angle PCRB, and beam misalignment probability. The coupling relationship between the communication link and the perception link is as follows: the distance, angle, and velocity estimates of highly mobile users output by the perception link are used to update the channel state and array steering vector in the communication link, thereby determining the phase of the beamforming vector; the beam misalignment probability and angle PCRB of the communication link serve as reliability feedback for vision scheduling and computing resource allocation.

[0009] The method includes:

[0010] In the nth time slot, the system state of the current time slot is obtained. The system state includes the vectorized semantic information age, edge computing task queue, virtual energy deficit queue, high-mobility user distance, and historical channel observation data. The vectorized semantic information age consists of radial semantic information age and tangential semantic information age. The radial semantic information age corresponds to continuous millimeter-wave radar observations, while the tangential semantic information age is updated to the corresponding queuing processing delay when the RGB camera module is activated and completes the visual task, and increments with time slots when the task is not completed. The virtual energy deficit queue is defined as... The update rule is as follows: ,in Let n be the total system energy consumption in the nth time slot. This is the preset energy budget for a single time slot;

[0011] The system state is input into a pre-trained LD-H-MoE network. The hierarchical execution architecture of the LD-H-MoE network includes a state input layer, an internal heterogeneous policy network, an action output layer, an experience replay buffer, a Lyapunov optimization loop, and external physical environment feedback. The state input layer receives the system state input and forms the current reinforcement learning state. The internal heterogeneous policy network comprises an LSTM time expert and an MLP spatial expert. The LSTM time expert processes the vectorized semantic information (age), edge computing task queue, and virtual energy deficit queue, and outputs scheduling variables. and CPU frequency allocation strategy The MLP spatial expert process handles distance and historical channel observation data for highly mobile users, outputs a beamforming phase matrix, and normalizes the beamforming phase matrix based on the constant mode constraint of the analog phase shifter to generate beamforming vectors. The Lyapunov optimization loop is based on the action. Post-execution state transition Construct a single time slot optimization objective Among them, actions include , and beamforming vector ; The instantaneous reward for the nth time slot is obtained by taking the inverse of the single time slot optimization objective. To adjust the non-negative weighting coefficients that balance the relationship between angle tracking accuracy, queue stability, and system energy consumption. For the k-th high-mobility user, the posterior Cramer-Rao boundary is given. The size of the visual computing task generated when the k-th highly mobile user activates the visual sensor in the n-th time slot. for The corresponding computation queue, The duration of a single time slot, This represents a virtual energy deficit; the experience replay cache is used to store state transition samples, which are used to update the time expert and the space expert respectively; the external physical environment feedback is used to provide feedback on new radar observations, visual task completion status, calculation queue length, energy consumption, and communication link quality.

[0012] The multimodal scheduling controller receives the output of the LD-H-MoE network, controls the activation and sleep of the RGB camera according to the sensor scheduling decision, allocates edge computing resources according to the CPU frequency allocation strategy, and controls the antenna array to transmit integrated sensing signals according to the beamforming vector.

[0013] After transmitting the integrated sensor signal, the virtual energy deficit queue is updated based on the difference between the instantaneous system energy consumption and the preset energy budget, and the edge computing task queue is updated based on the visual task processing status.

[0014] Specifically, the M-element antenna array adopts a uniform linear array or a uniform area array, and each antenna element is connected to an analog phase shifter to generate a directional millimeter-wave beam toward K highly mobile users.

[0015] Specifically, the millimeter-wave radar module works in conjunction with the antenna array or radio frequency front end to obtain low-power kinematic estimates of the angle of arrival, radial distance, and radial velocity of highly mobile users through broadband detection signals; the RGB camera module is mounted on the base station mast or roadside bracket to cover the road field of view and obtains the tangential position, angle calibration information, and semantic context of highly mobile users through a target detection network or visual positioning algorithm; the RGB camera is only activated under the instruction of the scheduling controller and is in a dormant state when not activated to save energy.

[0016] Specifically, the edge computing server is used to process visual tasks, allocate CPU or GPU computing frequencies, maintain computing task queues corresponding to each highly mobile user, and run the LD-H-MoE network for online decision-making.

[0017] Specifically, Including calculated energy consumption based on a dynamic voltage and frequency regulation model And the probability of beam misalignment in the communication link Triggered radio frequency recovery power consumption ,in, The effective switching capacitor coefficient of the processor chip. The processing frequency allocated to the visual computing task of the kth high-mobility user. The duration of a single time slot, The total energy consumption for radio frequency beam scanning and recovery triggered by beam misalignment in the nth time slot is... The energy consumed to perform a single radio frequency beam scan and recovery operation.

[0018] A multimodal sensing device based on energy constraints in high-mobility networks, used in the method of the present invention, includes:

[0019] The status acquisition module is used to acquire the system status of the current time slot. The system status includes vectorized semantic information age, edge computing task queue, virtual energy deficit queue, high mobile user distance, and historical channel observation data.

[0020] The network inference module has a built-in pre-trained LD-H-MoE network, which includes an LSTM time expert and an MLP spatial expert. The network inference module is used to process the vectorized semantic information age, edge computing task queue, and virtual energy deficit queue through the LSTM time expert, and output sensor scheduling decisions and CPU frequency allocation strategies. It also processes high mobile user distance and historical channel observation data through the MLP spatial expert, and outputs beamforming phase matrix.

[0021] The beam generation module is used to normalize the beamforming phase matrix based on the constant mode constraint of the analog phase shifter to generate a constant mode beamforming vector.

[0022] A multimodal scheduling controller is used to control the activation and hibernation of RGB cameras according to the sensor scheduling decision, and to allocate edge computing resources according to the CPU frequency allocation strategy;

[0023] Antenna array module, used to transmit integrated sensing signals according to the constant mode beamforming vector;

[0024] The queue update module is used to update the virtual energy deficit queue based on the difference between the instantaneous system energy consumption and the preset energy budget after the integrated sensor signal transmission is executed, and to update the edge computing task queue based on the visual task processing status.

[0025] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in this invention.

[0026] The present invention has the following beneficial effects: Compared with existing schemes that are based solely on continuous visual activation or solely on radar tracking, the present invention couples the network layer queuing delay with the physical layer spatial uncertainty by vectorizing the semantic information age, so that visual scheduling is no longer triggered by a fixed period, but by an event-triggered strategy jointly determined by the tracking PCRB, the computation queue and the energy budget.

[0027] This invention introduces a virtual energy deficit queue, which can transform long-term average energy budget constraints into a queue stability problem that can be handled online. This allows the system to automatically reduce the visual activation frequency or relax the visual update requirements of non-critical time slots when the energy budget is tight, and enhance visual calibration when the energy deficit is small, thereby achieving long-term sustainable operation.

[0028] This invention employs a heterogeneous hybrid expert structure to separate long-term time scheduling from instantaneous spatial phase mapping. LSTM experts excel at handling AoI and queue evolution, while MLP experts excel at handling non-convex constant-mode beam mapping. A strict gradient isolation mechanism reduces gradient conflicts between sensing accuracy, energy saving, and spatial gain objectives within a single network.

[0029] This invention can stabilize the edge computing queue while maintaining a low-angle PCRB and a low beam misalignment probability, and ensure that the long-term average energy consumption meets the preset budget. It is suitable for 6G vehicle-to-everything (V2X) networks, roadside smart base stations, unmanned vehicle cooperative perception, and high-mobility millimeter-wave communication systems. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the multimodal sensory integration V2I scenario to which this invention is applicable.

[0031] Figure 2 This is a schematic diagram of the LD-H-MoE hierarchical execution architecture proposed in this invention, illustrating the relationship between the internal heterogeneous policy network and the external Lyapunov optimization loop.

[0032] Figure 3 This is a comparison chart of the time-averaged PCRB tracking performance of different strategies at a signal-to-noise ratio of 10 dB.

[0033] Figure 4 This is a comparison chart of the time-averaged system energy consumption and the preset energy budget constraint for different strategies.

[0034] Figure 5 This diagram illustrates how the length of the edge computing queue changes over time under different strategies.

[0035] Figure 6 A comparison of the steady-state PCRB robustness of different strategies under different radio frequency signal-to-noise ratios. Detailed Implementation

[0036] The technical solution of the present invention will be further described below with reference to the accompanying drawings and simulation examples.

[0037] like Figure 1As shown, the highly mobile user is defined as a vehicle, and the scenario is a vehicle-to-everything (V2I) scenario. A multimodal sensing integrated base station is deployed on the roadside. The base station includes an M-element antenna array, a millimeter-wave radar module, an RGB camera module, an edge computing server, and a multimodal scheduling controller. The base station operates in discrete time slots. It simultaneously performs downlink communication, radar detection, visual scheduling, and edge computing resource allocation.

[0038] The M-element antenna array can be a uniform linear array or a uniform area array. Each antenna element is connected to an analog phase shifter to generate a directional millimeter-wave beam for K high-speed vehicle users. The vehicle side can be configured with a single antenna or a small number of antenna receiving modules to receive downlink communication data from the base station. Spatial leakage between adjacent beams causes multi-user interference; therefore, the base station needs to continuously update the beamforming vector based on the vehicle's spatial conditions.

[0039] Millimeter-wave radar modules work in conjunction with antenna arrays or RF front-ends to obtain low-power kinematic estimates of vehicle arrival angle, radial distance, and radial velocity using FMCW or other broadband detection signals. RGB cameras, mounted on base station masts or roadside brackets, cover the road field of view and obtain vehicle tangential position, angle calibration information, and semantic context through target detection networks or visual localization algorithms. The cameras are activated only upon command from the dispatch controller and remain in sleep mode when not in use to conserve energy.

[0040] Edge computing servers handle vision tasks, allocate CPU or GPU computing frequencies, maintain computing task queues for each vehicle, and run the LD-H-MoE network for online decision-making. The multimodal scheduling controller receives LD-H-MoE outputs and executes actions related to camera power supply, image acquisition, visual inference task submission, and antenna phase control.

[0041] In terms of the communication link, the base station acts as the transmitter, using an M-element array to generate K directional millimeter-wave beams and simultaneously transmitting downlink data to K vehicles using space division multiple access. The vehicles, acting as receivers, receive their own data streams. Due to the high-speed movement of the vehicles... Rapid changes mean that communication links need to rely on spatial state estimates provided by sensing links to update channels and beams; otherwise, they will suffer from increased multi-user interference, reduced effective channel gain, and the risk of link interruption.

[0042] In terms of the perception link, the base station performs active radar detection and passive visual observation of road vehicles. The radar link operates continuously with low power consumption, mainly providing radial distance, radial velocity, and coarse angle estimates; the visual link is activated on demand by the scheduling controller, mainly providing high-precision tangential position, angle calibration, and semantic context. The two types of perception results are fused in the edge server to form vectorized semantic information such as age, spatial uncertainty, angle PCRB, and beam misalignment probability.

[0043] The aforementioned communication and sensing links do not operate in isolation: the reliability constraints of the communication link determine when higher-precision visual calibration is needed, while the energy consumption and queuing status of the sensing link limit the frequency of visual calibration. This invention utilizes LD-H-MoE to perform joint decision-making between the two links, achieving coordinated optimization of sensor scheduling, computing resource allocation, and constant-mode beamforming.

[0044] The method proposed in this invention includes the following steps S1-S8.

[0045] S1: Construct the communication link model and the perception link model respectively.

[0046] This step explicitly divides the system model into communication links and perception links, and establishes the coupling relationship between the two based on vehicle spatial state, semantic information age, and beamforming. The communication link is used to complete downlink data transmission from the base station to the vehicle, while the perception link is used to acquire and update vehicle distance, angle, speed, and visual semantic information.

[0047] ① Communication link modeling: The base station simultaneously serves K highly mobile vehicles through an M-element antenna array in the nth time slot. Let... This represents the downlink beamforming vector for the k-th vehicle. This represents the data symbol sent to the k-th vehicle. Let represent the millimeter-wave channel vector corresponding to the k-th vehicle. Then, the received signal of the k-th vehicle includes the desired signal. Multi-user interference caused by beam leakage from other vehicles and additive white Gaussian noise The signal flow of the communication link is as follows: the base station baseband processor generates data symbols, which are then phase-shifted by a beamformer and an analog phase shifter to apply phase to the M-channel radio frequency signals. The array antenna forms a directional millimeter-wave beam, and the electromagnetic waves are demodulated by the vehicle receiving module after propagating through the space channel.

[0048] The geometric millimeter-wave channel can be modeled as a radial distance from the vehicle. Angle of arrival or angle of departure Path loss Propagation delay and array guide vector Related functions In line-of-sight-dominated scenarios, it can be represented as Because narrow-beam communication is extremely sensitive to angle errors, the reliability of the communication link mainly depends on the accuracy of vehicle angle estimation and beam phase update provided by the sensing link.

[0049] ② Perception Link Modeling: The perception link includes a millimeter-wave radar active perception link and an RGB vision passive perception link. In the radar perception link, the base station radar module transmits FMCW or broadband detection signals. After the detection signals are reflected by the vehicle body, the echo is received by the base station array. Through matched filtering, range estimation, speed estimation, and angle estimation algorithms such as MUSIC and ESPRIT, the arrival angle estimate, radial distance estimate, and speed estimate of the k-th vehicle are obtained. The radar operates continuously and has low power consumption and low latency characteristics, but its angle and tangential position estimation accuracy is affected by aperture, noise, and multipath.

[0050] In the RGB visual perception chain, the camera schedules variables. Road images are acquired in real time, and edge computing servers run object detection, vehicle localization, or semantic segmentation networks to obtain vehicle bounding boxes, azimuth calibration, tangential position, and semantic context; when At this time, the camera remains in sleep mode and does not generate new visual computing tasks. Visual link technology offers high accuracy but consumes a lot of power and creates a queue of computing tasks on the edge server. Therefore, its observation results have an information age determined by queuing and processing delays.

[0051] ③ Sensory link coupling relationship: The vehicle distance, angle, and speed estimates output by the sensor link are used to update the channel state and array steering vector in the communication link, thereby determining the beamforming vector. The phase; the beam misalignment probability and angle PCRB of the communication link, in turn, serve as reliability feedback for visual scheduling and computational resource allocation. When the visual task is not completed in time, the tangential semantic information age The increase in angle prior uncertainty leads to an increase in channel error variance and an increase in the probability of beam misalignment, ultimately triggering higher RF beam recovery energy consumption.

[0052] S2: Establish an edge computing task queue and dynamic energy consumption model.

[0053] Based on the aforementioned perception link, when the k-th vehicle activates its vision sensor in the n-th time slot, it generates a value of... Visual computing tasks. Corresponding computing queues. The evolution, following the pattern of "the remaining amount in the previous queue after CPU service plus the amount of newly arriving visual tasks," can be represented as follows: Total system energy consumption Including calculated energy consumption based on a dynamic voltage and frequency regulation model And the probability of beam misalignment in the communication link Triggered radio frequency recovery power consumption .

[0054] S3: Establish a geometrically coupled vectorized semantic information age model.

[0055] Unlike using a single scalar AoI to describe information staleness, this invention defines vectorized semantic information age. The radial AoI corresponds to continuous radar observation, while the tangential AoI corresponds to the completion status of the visual task. Since radar typically operates continuously, the radial AoI remains at a low value; the tangential AoI is updated to the corresponding queuing processing delay when the visual task is completed, and increases with time slots when it is not completed.

[0056] S4: Map semantic information age to spatial uncertainty, beam misalignment probability, and angle PCRB.

[0057] The vehicle position estimation at the base station follows a two-dimensional Gaussian uncertainty, with radial variance related to radial velocity and radial AoI, and tangential variance related to tangential velocity and tangential AoI. This is achieved by analyzing the channel function... A first-order multivariate Taylor expansion is performed to map the spatial uncertainty caused by AoI (Aspect-Oriented Inaccuracy) to the channel error variance. Furthermore, beam misalignment occurs when the angle error exceeds half the half-power beamwidth, and the misalignment probability can be calculated using the standard Q-function. To evaluate tracking reliability, a Bayesian FIM (Fischer Information Model) is constructed, consisting of a data Fisher information matrix and a priori Fisher information matrix, with the angle dimension PCRB (Polygonal Probability of Gain) used as the primary sensing accuracy indicator.

[0058] S5: Construct a long-term stochastic optimization problem.

[0059] This invention aims to minimize the long-term expected angle PCRB and jointly optimize sensor scheduling. CPU frequency and beamforming vector This problem also satisfies the following constraint: constant mode constraint of the simulated phase shifter. Total calculation frequency limit The real-world computation queue exhibits strong stability constraints, ensuring that the long-term average system energy consumption does not exceed the preset energy budget. And the visual scheduling variable is a binary variable.

[0060] S6: Use the Lyapunov method to transform the problem.

[0061] To address the stability constraints of the real computation queue and the long-term energy budget constraints, this invention introduces a virtual energy deficit queue. Its update rules are as follows When instantaneous energy consumption exceeds the budget, Cumulative increase; when instantaneous energy consumption is lower than the budget, Release. Construct a quadratic Lyapunov function. Furthermore, by employing a drift-plus-penalty criterion, the long-term stochastic problem is transformed into a single-slot alternative objective: Where V is a non-negative weighting coefficient used to adjust the trade-off between angle tracking accuracy, queue stability, and system energy consumption.

[0062] S7: Construct the LD-H-MoE reinforcement learning network and define the state, action, and reward.

[0063] System status Includes vectorized semantic information about age Real computation queue Virtual energy deficit queue Vehicle distance and historical channel observation .action Including discrete sensor scheduling Continuous CPU frequency allocation and complex beamforming vector The immediate reward is set as the inverse of the single-slot substitution objective, such that maximizing the cumulative reward of reinforcement learning is equivalent to minimizing the long-term angle PCRB under queue stability and energy budget constraints.

[0064] S8: Train and deploy using strict gradient isolation.

[0065] LD-H-MoE comprises an LSTM time expert and an MLP spatial expert. The LSTM time expert handles long-term sequence states such as AoI, queues, and energy deficits, outputting visual scheduling strategies and calculating frequency allocations; the MLP spatial expert handles instantaneous spatial channel characteristics, outputting a phase matrix. , and by Reconstruct the constant-mode beam. During training, the time expert is updated by the scheduling advantage function corresponding to the Lyapunov reward, and the spatial expert is updated by the physical channel gain, angle PCRB, or spatial refinement target. The gradient paths of the two experts are isolated from each other, thus avoiding negative transfer. After the system executes an action, it updates the real computation queue and the virtual energy deficit queue and moves to the next time slot.

[0066] like Figure 2 As shown, the LD-H-MoE hierarchical execution architecture of this invention includes a state input layer, an internal heterogeneous policy network, an action output layer, an experience replay buffer, a Lyapunov optimization loop, and external physical environment feedback. The state input layer receives multimodal sensor input and a computation queue. Virtual energy deficit queue Semantic information: age Based on historical channel observations, the current reinforcement learning state is formed. .

[0067] like Figure 2 As shown, the internal heterogeneous policy network consists of an LSTM temporal expert and an MLP spatial expert. The LSTM temporal expert is used to handle sequence states with long-term evolutionary characteristics, such as AoI, computation queues, and energy deficits, and outputs visual sensor scheduling actions. It provides a temporal decision-making basis for computing resource allocation; the MLP spatial expert is used to process instantaneous spatial characteristics such as vehicle distance, angle, array steering vector, and channel history, and outputs beamforming vectors. or phase matrix Then, the constant mode constraint of the analog phase shifter is satisfied through exponential mapping.

[0068] like Figure 2 As shown, the Lyapunov optimization loop is based on the action. Post-execution state transition Construct a single-slot optimization objective, which includes an angle PCRB penalty term and a real computation queue drift term. and virtual energy deficit The state transition samples, after being written to the experience replay buffer, are used to update the time expert and spatial expert, respectively. The gradient of the time expert comes from the scheduling-related loss, and the gradient of the spatial expert comes from the channel gain, PCRB, or phase refinement loss, thereby achieving... Figure 2 The strict gradient isolation is shown. Here, R(n) is the instantaneous reward obtained in the nth time slot based on the inverse of the single-time-slot substitute target, used to evaluate the combined impact of the current action on tracking accuracy, computation queue stability, and energy budget constraints.

[0069] like Figure 2 As shown, the actions output by LD-H-MoE are sent to the physical environment for execution: Control whether the RGB camera captures and submits visual tasks. The downlink beam phase of the control array antenna is controlled. The physical environment then provides feedback on new radar observations, visual mission completion status, calculated queue length, energy consumption, and communication link quality, forming a closed-loop online optimization process of "state input - expert decision-making - physical execution - Lyapunov feedback".

[0070] The present invention also provides an energy-constrained multimodal sensor integrated device based on Lyapunov-driven heterogeneous hybrid experts in high mobility networks. The device includes a multimodal sensor group, an edge computing server, a state buffer and feature extraction module, a Lyapunov queue update module, an LD-H-MoE deep learning network, a multimodal scheduling controller, an analog phase shifter and an array antenna.

[0071] A multimodal sensor array is used to acquire millimeter-wave radar echoes, visual semantic context, vehicle angle of arrival, radial distance, and speed information of high-mobility vehicles; an edge computing server is used to perform visual inference, maintain the real computing task queue, and allocate CPU or GPU computing frequencies; a state buffer and feature extraction module is used to form reinforcement learning states that include AoI, queues, energy deficit, and channel history; a Lyapunov queue update module is used to update the real computing queue and the virtual energy deficit queue; an LD-H-MoE deep learning network is used to jointly output sensor scheduling, computing resource allocation, and constant-mode beam phase; a multimodal scheduling controller is used to perform visual activation and sleep; and an analog phase shifter and array antenna are used to implement physical layer beamforming and integrated inductive transmission.

[0072] Simulation example:

[0073] Figure 2 The architecture diagram shown illustrates how LD-H-MoE receives states, generates schedules and beam actions, calculates Lyapunov rewards, and performs feedback training within each time slot.

[0074] System initialization. Base station configuration. Each antenna array can be set to a carrier frequency of 100000. Or higher millimeter-wave frequencies. The system serves K highly mobile vehicles, with a time slot length of τ. Initialize the computation queue for each vehicle. Initialize the virtual energy deficit queue Initialize vehicle distance, angle, and historical channel observation buffers.

[0075] The communication and sensing links operate simultaneously. Within each time slot, the communication link transmits downlink data from the base station to each vehicle based on the current beamforming vector. Vehicles receive the signals and provide feedback on necessary link quality or synchronization status. In the sensing link, the radar module continuously receives vehicle echoes and outputs angle of arrival, radial distance, and speed estimates. LD-H-MoE outputs visual scheduling based on the current status. .when At that time, the RGB camera captures images and submits the visual task to the edge computing server; when At this time, the camera goes into sleep mode, and the system relies on radar observation and historical visual information to predict the vehicle's spatial status.

[0076] Queue and AoI updates. If the vision task is completed, the tangential AoI is updated to reflect the actual queue processing latency. If the visual task is not completed, the tangent AoI increments with time slots. The real computing queue is updated based on the number of visual tasks arriving and the CPU frequency service load. The system updates the virtual energy deficit queue based on the difference between the current total energy consumption and the energy budget.

[0077] PCRB and reward calculation. The base station calculates the angle PCRB based on the current beamforming vector, array steering vector, and prior uncertainty corresponding to the vectorized AoI. This is combined with the actual queue term. and energy deficit We obtain the Lyapunov single-slot substitute target and take its negative as the reinforcement learning reward.

[0078] LD-H-MoE training: The LSTM temporal expert takes the historical state sequence as input and outputs the visual scheduling probability and CPU frequency parameters; the MLP spatial expert takes the current spatial state as input and outputs the phase matrix and reconstructs the constant mode beam through exponential mapping. After each training round, the state transition is sampled from the experience replay buffer, the temporal expert loss and spatial expert loss are calculated respectively, and only the corresponding expert parameters are updated.

[0079] Online deployment. After training, the base station performs forward inference once in each time slot to obtain sensor scheduling, computation frequency, and beam phase. If the virtual energy deficit queue is large, the strategy tends to reduce visual activation and high-frequency computing resource allocation; if PCRB or tangential AoI rises rapidly, the strategy triggers visual calibration to suppress angle uncertainty.

[0080] like Figure 3 As shown, under the condition of 10 dB radio frequency signal-to-noise ratio, the time-averaged PCRB of the strategy of the present invention is compared with that of different comparative strategies to characterize the long-term tracking accuracy of each strategy for the angular state of highly moving vehicles.

[0081] like Figure 4 As shown, the time-averaged system energy consumption of different strategies is compared with the preset energy budget to verify the moderating effect of the virtual energy deficit queue on the long-term average energy consumption constraint.

[0082] like Figure 5 As shown, the changes in the length of the edge computing task queue over time under different strategies are compared to verify the ability of the present invention to maintain the stability of the computing queue under the condition of random arrival of visual tasks.

[0083] like Figure 6 As shown, the steady-state PCRB of each strategy is compared under different radio frequency signal-to-noise ratio conditions to verify the angular tracking robustness of the present invention when the quality of the communication and sensing links changes.

Claims

1. A multimodal sensing integration method based on energy constraints in a high mobility network, wherein the high mobility network consists of a base station and K high mobility users, and the base station includes an M-element antenna array, a millimeter-wave radar module, an RGB camera module, an edge computing server, and a multimodal scheduling controller, and the base station operates in discrete time slots. It simultaneously performs downlink communication, radar detection, visual scheduling, and edge computing resource allocation; its characteristic is that... The system model is divided into a communication link and a sensing link. The communication link is used to complete downlink data transmission from the base station to the high-mobility user. The corresponding millimeter-wave channel is defined as the radial distance to the high-mobility user. Angle of arrival or angle of departure Path loss Propagation delay and array guide vector Related functions 'n' refers to the nth time slot, and the subscript 'k' refers to the kth high-mobility user. The perception link is used to acquire and update distance, angle, speed, and visual semantic information of highly mobile users. The perception link includes an active perception link based on a millimeter-wave radar module and a passive perception link based on an RGB camera module. The active perception link operates continuously with low power consumption, acquiring radial distance, radial speed, and coarse angle estimates. The passive perception link needs to be activated by a multimodal scheduling controller, setting scheduling variables. The vision sensor is activated to acquire tangential position, angle calibration, and semantic context. The two types of perception results are fused in the edge computing server to form vectorized semantic information such as age, spatial uncertainty, angle PCRB, and beam misalignment probability. The coupling relationship between the communication link and the perception link is as follows: the distance, angle, and speed estimates of high-mobility users output by the perception link are used to update the channel state and array steering vector in the communication link, thereby determining the phase of the beamforming vector. The beam misalignment probability and angle PCRB of the communication link serve as reliability feedback for visual scheduling and computational resource allocation. The method includes: In the nth time slot, the system state of the current time slot is obtained. The system state includes the vectorized semantic information age, edge computing task queue, virtual energy deficit queue, high-mobility user distance, and historical channel observation data. The vectorized semantic information age consists of radial semantic information age and tangential semantic information age. The radial semantic information age corresponds to continuous millimeter-wave radar observations, while the tangential semantic information age is updated to the corresponding queuing processing delay when the RGB camera module is activated and completes the visual task, and increments with time slots when the task is not completed. The virtual energy deficit queue is defined as... The update rule is as follows ,in Let n be the total system energy consumption in the nth time slot. This is the preset energy budget for a single time slot; The system state is input into a pre-trained LD-H-MoE network. The hierarchical execution architecture of the LD-H-MoE network includes a state input layer, an internal heterogeneous policy network, an action output layer, an experience replay buffer, a Lyapunov optimization loop, and external physical environment feedback. The state input layer receives the system state input and forms the current reinforcement learning state. The internal heterogeneous policy network comprises an LSTM time expert and an MLP spatial expert. The LSTM time expert processes the vectorized semantic information (age), edge computing task queue, and virtual energy deficit queue, and outputs scheduling variables. and CPU frequency allocation strategy The MLP spatial expert process handles distance and historical channel observation data for highly mobile users, outputs a beamforming phase matrix, and normalizes the beamforming phase matrix based on the constant mode constraint of the analog phase shifter to generate beamforming vectors. The Lyapunov optimization loop is based on the action. Post-execution state transition Construct a single time slot optimization objective Among them, actions include , and beamforming vector ; The instantaneous reward for the nth time slot is obtained by taking the negative of the single-time-slot optimization objective. To adjust the non-negative weighting coefficients that balance the relationship between angle tracking accuracy, queue stability, and system energy consumption. For the k-th high-mobility user, the posterior Cramer-Rao boundary is given. The size of the visual computing task generated when the k-th highly mobile user activates the visual sensor in the n-th time slot. for The corresponding computation queue, The duration of a single time slot, This represents a virtual energy deficit; the experience replay cache is used to store state transition samples, which are used to update the time expert and the space expert respectively; the external physical environment feedback is used to provide feedback on new radar observations, visual task completion status, calculation queue length, energy consumption, and communication link quality. The multimodal scheduling controller receives the output of the LD-H-MoE network, controls the activation and sleep of the RGB camera according to the sensor scheduling decision, allocates edge computing resources according to the CPU frequency allocation strategy, and controls the antenna array to transmit integrated sensing signals according to the beamforming vector. After transmitting the integrated sensor signal, the virtual energy deficit queue is updated based on the difference between the instantaneous system energy consumption and the preset energy budget, and the edge computing task queue is updated based on the visual task processing status.

2. The energy-constrained multimodal sensing integration method in high-mobility networks according to claim 1, characterized in that, The M-element antenna array adopts a uniform linear array or a uniform area array, and each antenna element is connected to an analog phase shifter to generate a directional millimeter-wave beam toward K highly mobile users.

3. The energy-constrained multimodal sensing integration method in high-mobility networks according to claim 1, characterized in that, The millimeter-wave radar module works in conjunction with the antenna array or radio frequency front end to obtain low-power kinematic estimates of the angle of arrival, radial distance, and radial velocity of highly mobile users through broadband detection signals; the RGB camera module is mounted on the base station mast or roadside bracket to cover the road field of view and obtains the tangential position, angle calibration information, and semantic context of highly mobile users through a target detection network or visual positioning algorithm. The RGB camera is only activated by the command of the scheduling controller; when not activated, it is in a sleep state to save energy.

4. The energy-constrained multimodal sensing integration method in high-mobility networks according to claim 1, characterized in that, The edge computing server is used to process visual tasks, allocate CPU or GPU computing frequencies, maintain computing task queues corresponding to each high-mobility user, and run the LD-H-MoE network for online decision-making.

5. The energy-constrained multimodal sensing integration method in high-mobility networks according to claim 1, characterized in that, Including calculated energy consumption based on a dynamic voltage and frequency regulation model and the probability of beam misalignment in the communication link Triggered radio frequency recovery power consumption ,in, The effective switching capacitor coefficient of the processor chip. The processing frequency allocated to the visual computing task of the kth high-mobility user. The total energy consumption for radio frequency beam scanning and recovery triggered by beam misalignment in the nth time slot is... The energy consumed to perform one radio frequency beam scan and recovery operation is 91CF.

6. A multimodal sensing integrated device based on energy constraints in high-mobility networks, used to perform the method according to any one of claims 1-5, characterized in that, include: The status acquisition module is used to acquire the system status of the current time slot. The system status includes vectorized semantic information age, edge computing task queue, virtual energy deficit queue, high mobile user distance, and historical channel observation data. The network inference module has a built-in pre-trained LD-H-MoE network, which includes an LSTM time expert and an MLP spatial expert. The network inference module is used to process the vectorized semantic information age, edge computing task queue, and virtual energy deficit queue through the LSTM time expert, and output sensor scheduling decisions and CPU frequency allocation strategies. It also processes high mobile user distance and historical channel observation data through the MLP spatial expert, and outputs beamforming phase matrix. The beam generation module is used to normalize the beamforming phase matrix based on the constant mode constraint of the analog phase shifter to generate a constant mode beamforming vector. A multimodal scheduling controller is used to control the activation and hibernation of RGB cameras according to the sensor scheduling decision, and to allocate edge computing resources according to the CPU frequency allocation strategy; Antenna array module, used to transmit integrated sensing signals according to the constant mode beamforming vector; The queue update module is used to update the virtual energy deficit queue based on the difference between the instantaneous system energy consumption and the preset energy budget after the integrated sensor signal transmission is executed, and to update the edge computing task queue based on the visual task processing status.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images