Multi-modal semantic communication resource optimization method for air-ground collaborative traffic perception

Abstract

The application discloses a multi-modal semantic communication resource optimization method for air-ground collaborative traffic perception, constructs an air-ground collaborative traffic perception network model composed of a base station, a roadside unit and a UAV, constructs a multi-modal semantic communication model of the roadside unit and the UAV and the base station, constructs a quality of experience model according to semantic transmission accuracy and transmission time, determines a resource optimization target equation with the maximum system quality of experience as the target, constructs a multi-agent reinforcement learning model according to the target equation, and optimizes a UAV trajectory and a communication resource configuration strategy. The application can realize collaborative semantic transmission of high-altitude overhead information and ground perspective information, and improve the accuracy and timeliness of traffic perception information transmission under a resource limited condition.

Description

Technical Field

[0001] This invention belongs to the field of semantic communication technology, specifically relating to a multimodal semantic communication resource optimization method for air-ground cooperative traffic perception. Background Technology

[0002] With the development of low-altitude intelligence and smart transportation, urban traffic perception is evolving towards air-ground collaborative perception. Roadside units, limited by coverage and building obstructions, struggle to meet the demands of large-scale traffic situational awareness in complex urban environments. Unmanned aerial vehicles (UAVs), with their advantages of flexible deployment and dynamic coverage, are widely used for information collection, target perception, and communication assistance tasks in low-altitude scenarios. Air-ground collaborative perception combines the high-altitude viewpoint information from UAVs with the close-range observation information from ground nodes, compensating for the shortcomings of single ground perception and obtaining more comprehensive traffic environment information. Therefore, collaboratively utilizing the high-altitude perception capabilities of UAVs, the ground observation capabilities of roadside units, and the processing capabilities of base stations can provide support for urban traffic situational awareness.

[0003] However, while enhancing perception capabilities, air-to-ground collaborative perception also brings the pressure of large-scale multimodal data transmission. Directly employing traditional bit-level communication methods would consume significant communication resources and increase transmission latency, making it difficult to adapt to the resource-constrained characteristics of UAV communication networks. Semantic communication, which focuses on extracting and transmitting effective semantic information for the task, rather than pursuing the complete recovery of original bits, can reduce irrelevant data transmission, lower communication overhead and transmission latency, and improve information transmission efficiency in resource-constrained scenarios. Therefore, introducing a semantic communication mechanism into air-to-ground collaborative urban traffic perception scenarios helps improve the real-time transmission of perceived information to base stations.

[0004] Preliminary research has been conducted on resource optimization for semantic communication in air-to-ground networks. E. Yang et al., in their paper "Joint Semantic Extraction and Resource Optimization in Communication-Efficient UAV Crowd Sensing" (IEEE Transactions on Network and Service Management, vol. 22, no. 6, pp. 5900-5914, Dec. 2025), constructed a semantic feature extraction module and improved the transmission efficiency and task performance of sensing data by jointly optimizing UAV trajectories, bandwidth allocation, and semantic compression ratio. Building on this, with the development of multimodal semantic communication technology, S. Liu et al., in their paper "Multi-UAV-Assisted MEC in Internet of Vehicles With Combined Multi-Modal Semantic Communication Under Jamming Attacks" (IEEE Transactions on Mobile Computing, vol. 24, no. 8, pp. 7600-7614, Aug. 2025), combined multimodal semantic communication with UAV-assisted mobile edge computing. By jointly optimizing UAV trajectory, user association, and channel selection, they reduced communication and computation latency while meeting the requirements for semantic transmission accuracy.

[0005] In summary, current research on air-to-ground collaborative semantic communication still has the following shortcomings. Most studies treat UAVs as high-altitude sensing nodes to collect and transmit images, or to receive and forward ground information, without sufficient consideration for the collaborative transmission and joint utilization of UAV high-altitude view information and local ground observation information. Different semantic modalities differ in transmission overhead and resource requirements. Existing research mostly focuses on transmission and resource optimization under a single modality, with insufficient research on resource optimization in multimodal semantic communication scenarios, making it difficult to achieve a balance between semantic transmission performance and resource utilization efficiency across different modalities. Existing research on semantic communication resource optimization often uses single performance indicators such as transmission latency, semantic rate, or resource cost as optimization targets. Since the performance of semantic communication depends on whether the user can complete the task, its evaluation criteria should not only consider the transmission process itself but also focus on the user's subjective experience. Therefore, it is necessary to introduce experience quality to comprehensively evaluate the task performance and transmission timeliness of semantic communication. Summary of the Invention

[0006] To address the aforementioned issues, this invention discloses a multimodal semantic communication resource optimization method for air-ground cooperative traffic perception. By constructing a two-stage air-ground multimodal semantic communication model, it jointly optimizes the UAV trajectory, communication bandwidth, transmission power, and number of transmitted semantic symbols. Furthermore, it utilizes a multi-agent reinforcement learning method to obtain UAV trajectory and communication resource allocation strategies, thereby achieving collaborative semantic transmission of high-altitude top-down information and ground-view information. This improves the accuracy and timeliness of traffic perception information transmission under resource-constrained conditions.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A multimodal semantic communication resource optimization method for air-ground cooperative traffic perception includes the following steps:

[0009] Step 1: Construct an air-ground cooperative traffic perception network model consisting of base stations, roadside units, and drones. This step enables the collaborative collection and transmission of aerial view information and ground traffic information, providing a network foundation for subsequent semantic communication.

[0010] Step 2: Construct a multimodal semantic communication model between roadside units and drones, and between drones and base stations. This step enables the joint transmission of semantic information from both air and ground perspectives, and adaptively selects the image-to-image or image-to-text semantic transmission mode based on the communication resource status.

[0011] Step 3: Construct an experience quality model based on semantic transmission accuracy and transmission time. This step comprehensively evaluates semantic transmission accuracy and transmission timeliness, providing an evaluation basis for system performance optimization.

[0012] Step 4: Determine the resource optimization objective equation with the goal of maximizing system experience quality. This step involves joint optimization of the UAV trajectory, spectrum bandwidth, transmit power, and number of transmitted semantic symbols to achieve coordinated allocation of communication resources and semantic transmission performance.

[0013] Step 5: Construct a multi-agent reinforcement learning model based on the objective equation to optimize the UAV trajectory and communication resource allocation strategy. This step utilizes multi-agent reinforcement learning to achieve collaborative decision-making between UAV trajectory planning and communication resource allocation.

[0014] The beneficial effects of this invention are as follows:

[0015] First, this invention constructs a two-stage air-ground multimodal semantic communication model to collaboratively process UAV aerial view information and roadside unit ground observation information, and selects the semantic transmission mode according to the communication resource status, thereby taking into account both the accuracy and timeliness of traffic perception information transmission.

[0016] Second, this invention jointly optimizes the UAV trajectory, communication bandwidth, transmission power, and number of transmitted semantic symbols, and uses a multi-agent reinforcement learning method to obtain UAV trajectory and communication resource allocation strategies, which can improve the resource allocation adaptability in a dynamic air-ground cooperative traffic perception environment. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an air-ground cooperative urban traffic situational awareness network scenario according to the present invention.

[0018] Figure 2 This is a flowchart of the method of the present invention;

[0019] Figure 3 This is a performance comparison chart of the joint optimization algorithm proposed in this invention and existing benchmark algorithms under different maximum available transmit powers in one embodiment.

[0020] Figure 4 This is a performance comparison chart of the joint optimization algorithm proposed in this invention and existing benchmark algorithms under different maximum available transmission bandwidths in one embodiment. Detailed Implementation

[0021] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] like Figure 1 As shown, this invention describes a multimodal semantic communication resource optimization method for air-ground cooperative traffic perception. In this method, roadside units collect ground traffic images of their target perception area and transmit them wirelessly to drones entering the target perception area. Simultaneously, the drones collect aerial images of the target area and transmit air-ground cooperative perception information to a base station via a wireless link. The base station receives semantic information uploaded by each drone to obtain traffic perception information for the target area. In this network, the roadside units and drones form a first-stage communication link, and the drones and the base station form a second-stage communication link, thus forming a two-stage air-ground cooperative semantic communication process.

[0023] The method of the present invention includes the following steps:

[0024] Step 1: Construct an air-ground cooperative traffic perception network model consisting of base stations, roadside units, and drones.

[0025] The air-ground cooperative traffic sensing network includes a base station, a roadside unit and A drone, among which, Set up within the traffic sensing area A circular target perception area with radius R is defined, centered on a roadside unit. The roadside units within the target perception area are used to acquire ground traffic images of the corresponding area. The UAV visits each target perception area at a fixed altitude. Upon entering the target perception area, it receives ground traffic images transmitted from the corresponding roadside units, simultaneously acquires aerial view images of the target perception area, and transmits the air-to-ground cooperative perception information to the base station. The mission cycle is divided into... There are several equal-length time slots, and the set of time slots is represented as... The duration of each time slot is The roadside unit set and the drone set are respectively represented as , Roadside unit The position is represented as , All drones fly at a fixed altitude H. In the time slot The position is represented as Setting up drones In the time slot Horizontal flight distance is Flight direction angle is Therefore, drones The horizontal position update can be represented as:

[0026] ;

[0027] ;

[0028] set up For the maximum horizontal flight distance of the drone within a single time slot, then the drone In the time slot Horizontal flight distance within Should meet .

[0029] The drone selects the nearest unfinished area as its current target and updates the target after completing the task in that area. When the drone completes its task within the area... After one service slot, the task for that area is considered complete, and completed areas will no longer participate in subsequent target selection. Define area communication indicator variables. .in Indicates drone In the time slot Enter the target area Perform air-to-ground coordinated semantic transmission; otherwise, The region indicator variable must then satisfy the following distance constraint:

[0030] ;

[0031] The drone will only perform air-to-ground collaborative perception and semantic transmission tasks in the area if it meets the geometric constraints.

[0032] Step 2: Construct a multimodal semantic communication model between roadside units and drones, and between drones and base stations.

[0033] The roadside unit transmits the acquired images to the drone via image-to-image semantic transmission. (The image information to be transmitted...) The roadside unit first extracts task-related semantic features using a semantic encoder, and then maps them into semantic symbols suitable for wireless channel transmission using a channel encoder. Let... and The parameters are respectively The semantic encoder and parameters are For a channel encoder, the semantic symbols generated by the transmitter can be represented as:

[0034] ;

[0035] Let the average mapping of each image patch be... There are semantic symbols, among which... , The roadside unit divides the acquired images into a set of optional semantic symbols representing the semantic transmission of images. Block, then there is After transmission via wireless channel, the receiving end receives the signal as follows:

[0036] ;

[0037] in For channel gain, It is additive white Gaussian noise, and the receiver uses the corresponding channel decoder. and semantic decoder Recover target information Then the output of the receiving end can be expressed as

[0038] ;

[0039] roadside unit With drones In the time slot The communication distance is expressed as:

[0040] ;

[0041] Therefore, the free space path loss is expressed as:

[0042] ;

[0043] in For carrier frequency, The speed is the speed of light. Assume the emission power of all roadside units is constant. The bandwidth of the link between the roadside unit and the drone is The noise power spectral density is Then the signal-to-noise ratio of the link from the roadside unit to the drone is

[0044] ;

[0045] This invention uses semantic transmission accuracy to measure the difference between the target information and the restored result. Image-to-image transmission and image-to-text transmission are evaluated using structural similarity and bilingual evaluation surrogate metrics, respectively. Semantic transmission accuracy is mainly determined by the signal-to-noise ratio. and the number of selected semantic symbols The semantic accuracy function is determined but difficult to obtain explicitly. Therefore, curve fitting is used to obtain the semantic accuracy function and modality. The accuracy function is as follows

[0046] ;

[0047] in, and Let represent the upper and lower asymptotes of the fitted curve, respectively. This indicates the growth rate of the fitted curve. This indicates the location of the midpoint of the fitted curve.

[0048] Image modal semantic entropy is This represents the minimum average number of semantic symbols required to meet task performance requirements, ignoring channel effects. Roadside Unit To drones The semantic transmission rate is

[0049] ;

[0050] The size of the ground traffic image information that the roadside unit needs to transmit to the drone in the time slot is [size missing]. Therefore, roadside units Transmit image information to the drone The required time is

[0051] ;

[0052] Set up drones In the time slot The collected aerial view image information is Aerial view images from UAVs can reflect the overall scene structure of the target area, while ground-view information from roadside units can supplement the detailed features of local targets. To leverage the complementary role of these two types of view information, the UAV-based semantic encoder, based on the extracted task-related features from the aerial image, introduces the restored ground-view semantic information to jointly encode semantically related target features from both the air and ground perspectives. This results in a collaborative semantic representation that simultaneously includes global scene information and local details. This process can be represented as follows:

[0053] ;

[0054] in The parameter is semantic encoder, The target information mode that the base station expects to recover is Then it is necessary to convert the information of the original image modality into the target modality. The semantic information. Then, through the parameters... Channel encoder Map it to semantic symbols suitable for wireless channel transmission.

[0055] ;

[0056] Let the average mapping of each image patch be... The drone will divide the collected images into semantic symbols. Block, then there is After the semantic symbols are transmitted through the wireless channel, the base station receives the signal as...

[0057] ;

[0058] in For channel gain, It is additive white Gaussian noise, and the receiver uses the corresponding channel decoder. and semantic decoder Recover the target information of mode J Then the output of the receiving end can be expressed as

[0059] ;

[0060] The communication link between the UAV and the base station adopts the same free-space path loss model as the communication link between the roadside unit and the UAV. The location of the base station is denoted as... The distance between the base station and the drone in the time slot is

[0061] ;

[0062] The path loss of the communication link between the drone and the base station is

[0063] ;

[0064] set up and They represent drones In the time slot Given the transmit power during semantic transmission to the base station and the bandwidth allocated for this transmission process, the signal-to-noise ratio (SNR) of the signal received at the base station can be expressed as:

[0065] ;

[0066] The UAV selects between two semantic transmission modes, image-to-image and image-to-text, based on the communication resource status. Let the UAV... The number of semantic symbols selected is Its satisfaction , ,in This represents the set of optional semantic symbols corresponding to text semantic transmission. The semantic transmission mode of the communication link between the UAV and the base station is determined by the number of semantic symbols selected by the UAV. When an image-to-image semantic transmission mode is adopted, the corresponding approximate semantic entropy and semantic accuracy functions are respectively... , ,when When using an image-to-text semantic transmission mode, the corresponding approximate semantic entropy and semantic accuracy functions are respectively... , .

[0067] Then in The achievable semantic transmission rate between the drone and the base station is

[0068] ;

[0069] When drones In the Within a given area, the UAV adaptively switches semantic modalities based on ground traffic image semantics from roadside units and its own aerial view images, forming semantic information to be transmitted to the base station. The amount of semantic information is [missing information]. Therefore, the time required for the drone to upload the image data collected in a coordinated air-to-ground manner to the base station is...

[0070] ;

[0071] Therefore, the total transmission time of a complete air-to-ground coordinated service is the sum of the transmission times of the two phases, i.e.

[0072] ;

[0073] Step 3: Construct an experience quality model based on semantic transmission accuracy and transmission time.

[0074] Based on the semantic transmission accuracy of the UAV-to-base station phase and the total transmission time of the two transmission phases (from roadside unit to UAV and from UAV to base station), a system experience quality evaluation function for the target perception area is constructed; when the UAV... In the When communicating with a target area, the quality of experience is defined as follows:

[0075] ;

[0076] in Indicates drone The accuracy of semantic transmission to the base station This indicates the total transmission time for completing the information collection and uploading for this area. and These represent the minimum accuracy and transmission time required to achieve a 50% score, respectively. and This represents the growth rate coefficient. This is the weighting factor between accuracy gains and latency gains.

[0077] Step 4: Determine the objective equation for resource optimization with the goal of maximizing system experience quality.

[0078] By jointly optimizing UAV trajectories, UAV transmission power, roadside units, and the spectrum bandwidth and transmitted semantic symbols of the UAVs, the overall experience quality of the urban traffic situational awareness system is maximized. This achieves synergistic optimization of communication resources and semantic transmission performance under resource-constrained conditions. Therefore, the optimization problem can be expressed as follows:

[0079] P1 ;

[0080] stC1: ;

[0081] C2: ;

[0082] C3: ;

[0083] C4: ;

[0084] C5: ;

[0085] C6: ;

[0086] C7: ;

[0087] C8: ; ;

[0088] C9: ;

[0089] Among them, constraints C1 and C2 are UAV flight constraints, constraint C3 gives the binary value of the area communication indicator variable, constraint C4 ensures that the UAV can only trigger the communication task after entering the target area, constraint C5 restricts the number of semantic symbols selected by the roadside unit and the UAV, constraint C6 restricts the transmission power of the UAV, constraint C7 represents the total bandwidth constraint of the system, and constraints C8 and C9 respectively ensure that the semantic accuracy is greater than the minimum threshold requirement and the transmission delay does not exceed the preset threshold.

[0090] Step 5: Construct a multi-agent reinforcement learning model based on the objective equation to optimize the UAV trajectory and communication resource allocation strategy.

[0091] The joint optimization problem of UAV trajectory and communication resources is decomposed into UAV trajectory planning sub-problem and communication resource management sub-problem, and trajectory planning model and resource management model based on multi-agent near-end policy optimization algorithm are constructed respectively.

[0092] No. A drone in a time slot The trajectory planning state is defined as

[0093] ;

[0094] in, Indicates the current location of the drone. Indicates the current target location. Indicates drone Distance to the current target Indicates drone Has the area been completed? The collection and transmission of air-ground collaborative sensing data. This indicates that the task has been completed. A region indicator variable that indicates the range within which a drone has entered the target area. This represents the set of locations of all roadside units. Indicates drone The finish line Indicates drone Distance to the base station.

[0095] Action Space: Drones The trajectory planning action is defined as .

[0096] The trajectory planning reward function is designed to guide the UAV through a three-stage task. The first stage involves flying towards the currently bound roadside unit. The second stage involves maintaining effective service for multiple consecutive time slots after entering the roadside unit's service range, completing one full two-stage communication in each time slot. The third stage involves flying to the preset destination after completing service with all roadside units. Therefore, the UAV... In the time slot The instant reward is defined as:

[0097] ;

[0098] Flight phase rewards When the drone has not yet entered the target area, the reward function guides it to approach the target.

[0099] ;

[0100] in The reward coefficient for getting closer to the target. Rewards for approaching the target area. To approach the reward threshold, This is an indicator function.

[0101] Communication phase rewards When the drone entered the When defining a target area, drones are encouraged to move closer to the base station during communication. The reward coefficient for proximity to the base station during the communication phase.

[0102] ;

[0103] Return phase rewards Once all roadside units have completed their service, the drone switches its target to the destination, and a reward function guides it back to base. The reward for the return phase is defined as follows:

[0104] ;

[0105] in, The reward coefficient for getting closer to the finish line. As a reward for reaching the finish line, For drones The distance to the destination.

[0106] In the UAV trajectory planning phase, a multi-agent proximal policy optimization network based on a centralized training and decentralized execution mechanism is constructed. This includes an Actor network that selects trajectory planning actions based on the local observations of each UAV, and a centralized Critic network that estimates the state value based on the global state of all UAVs. The parameters of the trajectory planning Actor network are then initialized. Critic network parameters Discount Factor Generalized dominance estimation parameters and strategy clipping parameters .

[0107] drones Local observations based on its trajectory planning Using Actor networks to sample trajectories and plan actions The trajectory planning actions of each UAV are combined into a joint action input environment to obtain local observations in the next time slot. global state and rewards The interaction experience is stored in the experience cache, and the temporal difference error is calculated based on the Critic network.

[0108] ;

[0109] in Based on the aforementioned time-series difference error, the dominance function for time slot t is calculated using the generalized dominance estimation method:

[0110] ;

[0111] And calculate the return estimate of the value network:

[0112] ;

[0113] The value loss function of the trajectory planning Critic network is constructed using mean square error:

[0114] ;

[0115] For drone n, the probability ratio between the current policy and the old policy before the update is defined as .

[0116] ;

[0117] The objective function of the pruning strategy in the trajectory planning Actor network is:

[0118] ;

[0119] Update the Critic network and Actor network

[0120] ;

[0121] ;

[0122] Based on the obtained UAV access trajectories, the communication events of each UAV entering the target perception area and performing air-to-ground cooperative semantic transmission are determined. Communication events occurring within the same time slot are grouped into a communication event set. ,in, Let τ represent the τ-th communication event group. In the resource management phase, the event group is used as the decision step, and each event group is regarded as an MDP step size. The MAPPO-MSC algorithm is used to optimize the communication resources of multiple UAVs.

[0123] The state space is: for the τ-th communication event group, the UAV Resource management local observation is defined as

[0124] ;

[0125] in, and These represent the distance from the roadside unit to the drone and the distance from the drone to the base station, respectively. and These represent the channel gain from the roadside unit to the drone and from the drone to the base station, respectively. and These represent the data size of the image captured by the drone and the data size of the image transmitted from the roadside unit to the drone, respectively. Indicates drone Remaining communication energy before the event is executed.

[0126] Action space includes continuous action space and discrete action space .

[0127] The reward function consists of two parts: the first is the experience quality reward, the second is the penalty when the semantic transmission accuracy of the communication between the drone and the base station does not meet the expected standard, and the third is the penalty when the semantic transmission accuracy of the communication between the roadside unit and the drone does not meet the expected standard.

[0128] ;

[0129] Using a communication event group as a resource management decision step, construct a resource management Actor network and a centralized Critic network, and initialize the Actor network parameters. Critic network parameters Discount Factor Generalized dominance estimation parameters and strategy clipping parameters .

[0130] Actor networks consist of continuous action branches and discrete action branches. The continuous action branches are based on local observations. Sampling spectrum bandwidth and transmit power:

[0131] ;

[0132] Discrete action branch output semantic symbol number selects the corresponding logical variable The Gumbel-Softmax method is used to generate discrete semantic resource actions.

[0133] ;

[0134] The complete resource management actions of UAV n consist of continuous and discrete actions: Input environment to obtain local observations in the next time slot global state and rewards And store the interactive experience in the experience cache area;

[0135] Calculate the timing difference error of the τth communication event group based on the centralized Critic network:

[0136] ;

[0137] Based on the aforementioned time-series differential error, calculate the generalized advantage estimate and return estimate for the resource management phase:

[0138] ;

[0139] ;

[0140] Define the probability ratio between the current hybrid action policy and the old policy before the update as:

[0141] ;

[0142] The objective function of the pruning strategy in the resource management Actor network is:

[0143] ;

[0144] The value loss function of the resource management Critic network is:

[0145] ;

[0146] Update the Critic network and Actor network

[0147] ;

[0148] ;

[0149] The effects of the present invention will be further explained below with reference to simulation experiments.

[0150] 1. Simulation parameter settings and benchmark method settings:

[0151] Consider a 1500 m × 1500 m urban traffic sensing area, in which M = 8 roadside units, N = 2 drones, and one fixed base station are deployed. The drones fly at a fixed altitude of H = 100 m, and the maximum flight distance of the drones is... Time slot length =1s, each target area The total time required is Total system bandwidth =8MHz, maximum transmit power of the drone =1W, the transmission power of the roadside unit . Optional set of image semantic symbols Optional set of text semantic symbols The target area is defined by a radius centered on the roadside unit. The circular area. Image information size collected by roadside units and drones. and It is randomly selected from the range [200, 400]. The weighting factor between accuracy gain and latency gain. , , =15, =30, =0.8, =0.25 noise power spectral density is , , .

[0152] Both stages of the algorithm use MAPPO. In the first stage, the hidden layers for Actor and Critic have dimensions of 256 and 128, respectively, and the learning rate is set to 3 x 10. -5 and 1x10 -4 The pruning factor was 0.15, the discount factor was 0.995, the GAE coefficient was 0.97, and the training run was 3000 times. In the second stage, the learning rates for the resource management actors and criters were 8x10. -5 and 4x10 -4 The pruning factor is 0.2, the discount factor is 0.5, the GAE factor is 0.95, and the training is performed 1500 times.

[0153] To verify the effectiveness of MAPPO-MSC, this invention compares it with three benchmark schemes:

[0154] 1. MAPPO-ISC: It adopts the same MAPPO framework as the proposed method, but the communication from the UAV to the base station is only allowed to use the image-to-image semantic transmission mode. It does not adaptively select the image-to-image and image-to-text semantic transmission modes according to the communication resource status.

[0155] 2. MAPPO-MSCFS: Allows UAV-to-base station communication to adaptively select between two semantic transmission modes, image-to-image and image-to-text, based on the communication resource status. However, the number of semantic symbols selected for each transmission mode is fixed. The number of semantic symbols for image-to-image and image-to-text semantic transmission modes are fixed at 256 and 48, respectively, and only the spectrum bandwidth and transmission power are optimized.

[0156] 3. DQN-MADDPG-MSC: This method employs DQN to handle discrete decisions such as semantic modality and the number of semantic symbols, and MADDPG to handle continuous decisions such as bandwidth and transmit power. It is used to compare the resource management performance of different discrete and continuous hybrid action reinforcement learning frameworks.

[0157] 2. Simulation content:

[0158] Figure 3 The results show the overall QoE of the system as a function of the maximum transmit power of the UAV. As the maximum transmit power increases, the QoE of each scheme gradually improves; this is because increasing the transmit power enhances the signal-to-noise ratio of the communication link between the UAV and the BS, thereby improving the accuracy of semantic transmission and reducing transmission latency. MAPPO-MSC achieves optimal performance across the entire power range, demonstrating its ability to adaptively select the semantic transmission mode based on resource status and jointly optimize the transmission of semantic symbols, bandwidth, and transmit power. DQN-MADDPG-MSC also optimizes the aforementioned discrete and continuous variables, thus outperforming the two benchmark schemes. However, since discrete semantic decision-making and continuous resource allocation are updated using different algorithms, their synergistic effect is weaker than that of MAPPO-MSC. The performance difference between MAPPO-MSCFS and MAPPO-MSC further indicates that optimizing the number of transmitted semantic symbols can improve the system QoE. Furthermore, MAPPO-MSC outperforms MAPPO-ISC, verifying the performance-enhancing effect of multimodal semantic communication on the system.

[0159] Figure 4 The results show that the total QoE of the system varies with the maximum available bandwidth. As the bandwidth increases, the QoE of all schemes shows an upward trend; this is because more sufficient bandwidth can reduce transmission latency and allow the transmission of more semantic symbols, thereby improving semantic accuracy and latency gains. As bandwidth increases further, the curve gradually flattens out, indicating that transmission latency and semantic accuracy have approached their upper limits, at which point system performance is more limited by factors such as transmit power. MAPPO-MSC maintains optimal performance across the entire bandwidth range, demonstrating its ability to adapt to different resource conditions by jointly adjusting semantic transmission modes, the number of semantic symbols, bandwidth, and transmit power. DQN-MADDPG-MSC also possesses semantic transmission mode and semantic symbol adaptation capabilities, but the resource allocation for discrete and continuous variables is handled by DQN and MADDPG respectively. Insufficient coordination during their updates can easily occur, resulting in lower performance than MAPPO-MSC with unified decision-making. MAPPO-MSCFS is lower than MAPPO-MSC, indicating that optimizing the number of transmitted semantic symbols can further improve system performance. MAPPO-ISC is lower than MAPPO-MSC, demonstrating that multimodal semantic communication can further improve system QoE by reducing the amount of transmitted data and transmission time, with its advantages being more pronounced under bandwidth-constrained conditions.

[0160] In summary, this invention constructs a two-stage air-ground collaborative multimodal semantic communication model from roadside units to UAVs and from UAVs to base stations, enabling collaborative semantic transmission of UAV aerial view information and roadside unit ground observation information. By jointly optimizing UAV trajectories and communication resource allocation, it balances the accuracy and timeliness of traffic perception information transmission under resource-constrained conditions.

[0161] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A multimodal semantic communication resource optimization method for air-ground cooperative traffic perception, characterized by: Includes the following steps: (1): Construct an air-ground cooperative traffic perception network model consisting of base stations, roadside units, and drones; (2): Construct a multimodal semantic communication model between roadside units and UAVs and between UAVs and base stations; (3): Construct an experience quality model based on semantic transmission accuracy and transmission time; (4): Determine the resource optimization objective equation with the goal of maximizing the system experience quality; (5): Construct a multi-agent reinforcement learning model based on the objective equation to optimize the UAV trajectory and communication resource allocation strategy.

2. The multimodal semantic communication resource optimization method for air-ground cooperative traffic perception according to claim 1, characterized in that, Step (1) includes: The air-ground cooperative traffic sensing network includes a base station, a roadside unit and A drone, among which, Set up within the traffic sensing area A circular target perception area with radius R is defined, centered on a roadside unit. The roadside units within the target perception area are used to collect ground traffic images of the corresponding area. The UAV visits each target perception area at a fixed altitude. Upon entering the target perception area, it receives ground traffic images transmitted from the corresponding roadside units, simultaneously collects aerial view images of the target perception area, and transmits the air-ground cooperative perception information to the base station. The task cycle is divided into... There are several equal-length time slots, and the set of time slots is represented as... The duration of each time slot is The roadside unit set and the drone set are respectively represented as , Roadside unit The position is represented as , All drones fly at a fixed altitude H. In the time slot The position is represented as ; Set up drones In the time slot Horizontal flight distance is Flight direction angle is Therefore, drones The horizontal position update is represented as: ； ； set up For the maximum horizontal flight distance of the drone within a single time slot, then the drone In the time slot Horizontal flight distance within Should meet ; The drone selects the nearest unfinished area as its current target and updates the target after completing the task in that area; when the drone completes the task within the area... After one service slot, the task in that area is considered complete, and completed areas will no longer participate in subsequent target selection; define area communication indicator variables. ;in Indicates drone In the time slot Enter the target area Perform air-to-ground coordinated semantic transmission; otherwise, The region indicator variable must then satisfy the following distance constraint: ； The drone will only perform air-to-ground collaborative perception and semantic transmission tasks in the area if it meets the geometric constraints.

3. The multimodal semantic communication resource optimization method for air-ground cooperative traffic perception according to claim 1, characterized in that, Step (2) includes: The roadside unit transmits the acquired images to the drone via image-to-image semantic transmission; for the image information to be transmitted... The roadside unit first extracts task-related semantic features using a semantic encoder, and then maps them into semantic symbols suitable for wireless channel transmission using a channel encoder; Let... and The parameters are respectively The semantic encoder and parameters are The channel encoder, then the semantic symbols generated by the transmitter are represented as ； Let the average mapping of each image patch be... There are semantic symbols, among which... , The roadside unit represents the set of optional semantic symbols for image semantic transmission; the roadside unit divides the acquired images into... Block, then there is After transmission via wireless channel, the receiving end receives the signal as ； in For channel gain, It is additive white Gaussian noise; the receiver uses the corresponding channel decoder. and semantic decoder Recover target information The output of the receiving end is then expressed as ； roadside unit With drones In the time slot The communication distance is expressed as: ； Therefore, the free space path loss is expressed as: ； in For carrier frequency, Assume the speed of light; assume the emission power of all roadside units is fixed. The bandwidth of the link between the roadside unit and the drone is The noise power spectral density is Then the signal-to-noise ratio of the link from the roadside unit to the drone is ； Curve fitting is used to obtain the semantic accuracy function and modality. The accuracy function is as follows ； in, and Let represent the upper and lower asymptotes of the fitted curve, respectively. This indicates the growth rate of the fitted curve. This indicates the location of the midpoint of the fitted curve; Image modal semantic entropy is It represents the minimum average number of semantic symbols required to meet task performance requirements, ignoring channel effects; roadside unit To drones The semantic transmission rate is ； The size of the ground traffic image information that the roadside unit needs to transmit to the drone in the time slot is [size missing]. Therefore, roadside units Transmit image information to the drone The required time is ； Set up drones In the time slot The collected aerial view image information is The UAV-based semantic encoder, based on the extracted mission-related features from high-altitude images, incorporates restored ground-view semantic information to jointly encode semantically related target features from both air and ground perspectives, resulting in a collaborative semantic representation. This process is represented as follows: ； in The parameter is semantic encoder, The target information mode that the base station expects to recover is Then it is necessary to convert the information of the original image modality into the target modality. Semantic information; then through parameters Channel encoder Map it to semantic symbols suitable for wireless channel transmission. ； Let the average mapping of each image patch be... The drone will divide the collected images into semantic symbols. Block, then there is After the semantic symbols are transmitted through the wireless channel, the base station receives the signal as... ； in For channel gain, It is additive white Gaussian noise; the receiver uses the corresponding channel decoder. and semantic decoder Recover the target information of mode J The output of the receiving end is then expressed as ； The communication link between the UAV and the base station adopts the same free-space path loss model as the communication link between the roadside unit and the UAV; the location of the base station is denoted as... The distance between the base station and the drone in the time slot is ； The path loss of the communication link between the drone and the base station is ； set up and They represent drones In the time slot Given the transmit power during semantic transmission to the base station and the bandwidth allocated for this transmission process, the signal-to-noise ratio (SNR) of the signal received at the base station is expressed as: ； The UAV selects between two semantic transmission modes, image-to-image and image-to-text, based on the communication resource status; assuming the UAV... The number of semantic symbols selected is Its satisfaction , ,in This represents the set of optional semantic symbols corresponding to text semantic transmission; the semantic transmission mode of the communication link between the UAV and the base station is determined by the number of semantic symbols selected by the UAV; when When an image-to-image semantic transmission mode is adopted, the corresponding approximate semantic entropy and semantic accuracy functions are respectively... , ;when When using an image-to-text semantic transmission mode, the corresponding approximate semantic entropy and semantic accuracy functions are respectively... , ; Then in The achievable semantic transmission rate between the drone and the base station is ； When drones In the Within a given area, the UAV adaptively switches semantic modalities based on ground traffic image semantics from roadside units and its own aerial view images, forming semantic information to be transmitted to the base station. The amount of semantic information is [missing information]. Therefore, the time required for the drone to upload the image data collected in a coordinated air-to-ground manner to the base station is... ； Therefore, the total transmission time of a complete air-to-ground coordinated service is the sum of the transmission times of the two phases, i.e. 。 4. The multimodal semantic communication resource optimization method for air-ground cooperative traffic perception according to claim 1, characterized in that, Step (3) includes: Based on the semantic transmission accuracy of the UAV-to-base station phase and the total transmission time of the two transmission phases (from roadside unit to UAV and from UAV to base station), a system experience quality evaluation function for the target perception area is constructed; when the UAV... In the When communicating with a target area, the quality of experience is defined as follows: ； in Indicates drone The accuracy of semantic transmission to the base station This indicates the total transmission time for completing the information collection and uploading for this area; and These represent the minimum accuracy and transmission time required to achieve a 50% score, respectively. and This is the growth rate coefficient; This is the weighting factor between accuracy gains and latency gains.

5. The multimodal semantic communication resource optimization method for air-ground cooperative traffic perception according to claim 1, characterized in that, Step (4) includes: By jointly optimizing drone trajectories, drone transmission power, roadside units, and the drone's spectrum bandwidth and transmitted semantic symbols, the goal is to maximize the overall experience quality of the urban traffic situational awareness system; therefore, the optimization problem is expressed as follows: P1 ； s.t.C1: ; C2: ； C3： ； C4： ； C5: ； C6： ； C7: ； C8： ； ； C9： ； Among them, constraints C1 and C2 are UAV flight constraints, constraint C3 gives the binary value of the area communication indicator variable, constraint C4 ensures that the UAV can only trigger the communication task after entering the target area, constraint C5 restricts the number of semantic symbols selected by the roadside unit and the UAV, constraint C6 restricts the transmission power of the UAV, constraint C7 represents the total bandwidth constraint of the system, and constraints C8 and C9 respectively ensure semantic accuracy and transmission delay requirements.

6. The multimodal semantic communication resource optimization method for air-ground cooperative traffic perception according to claim 1, characterized in that, Step (5) includes: The joint optimization problem of UAV trajectory and communication resources is decomposed into UAV trajectory planning sub-problem and communication resource management sub-problem, and trajectory planning model and resource management model based on multi-agent near-end policy optimization algorithm are constructed respectively. No. A drone in a time slot The trajectory planning state is defined as ； in, Indicates the current location of the drone. Indicates the current target location. Indicates drone Distance to the current target Indicates drone Has the area been completed? The collection and transmission of air-ground collaborative sensing data. This indicates that the task has been completed; A region indicator variable that indicates the range within which a drone has entered the target area. This represents the set of locations of all roadside units. Indicates drone The finish line Indicates drone Distance to the base station; Action Space: Drones The trajectory planning action is defined as ; The trajectory planning reward function is designed to guide the drone through three phases of a task: Phase 1, flying towards the currently bound roadside unit; Phase 2, maintaining effective service for multiple consecutive time slots after entering the roadside unit's service range, completing one full two-phase communication in each time slot; Phase 3, after completing service with all roadside units, flying towards the preset destination; therefore, the drone... In the time slot The instant reward is defined as: ； Flight phase rewards When the drone has not yet entered the target area, the reward function guides it to approach the target. ； in The reward coefficient for getting closer to the target. Rewards for approaching the target area. To approach the reward threshold, For indicator functions; Communication phase rewards When the drone entered the When defining a target area, drones are encouraged to move closer to the base station during communication. The reward coefficient for proximity to the base station during the communication phase. ； Return phase rewards Once all roadside units have completed their service, the drone switches its target to the destination, and a reward function guides it back to base; the reward for the return phase is defined as follows: ； in, The reward coefficient for getting closer to the finish line. As a reward for reaching the finish line, For drones Distance to the destination; In the UAV trajectory planning phase, a multi-agent proximal policy optimization network based on a centralized training and decentralized execution mechanism is constructed. This includes an Actor network that selects trajectory planning actions based on the local observations of each UAV, and a centralized Critic network that estimates the state value based on the global state of all UAVs. The parameters of the trajectory planning Actor network are initialized. Critic network parameters Discount Factor Generalized dominance estimation parameters and strategy clipping parameters ; drones Local observations based on its trajectory planning Using Actor networks to sample trajectories and plan actions The trajectory planning actions of each UAV are combined into a joint action input environment to obtain local observations in the next time slot. global state and rewards The interaction experience is stored in the experience cache; the temporal difference error is calculated based on the Critic network. ； in Based on the aforementioned time-series difference error, the dominance function for time slot t is calculated using the generalized dominance estimation method: ； And calculate the return estimate of the value network: ； The value loss function of the trajectory planning Critic network is constructed using mean square error: ； For drone n, the probability ratio between the current policy and the old policy before the update is defined as . ； The objective function of the pruning strategy in the trajectory planning Actor network is: ； Update the Critic network and Actor network ； ； Based on the obtained UAV access trajectories, the communication events of each UAV entering the target perception area and performing air-to-ground cooperative semantic transmission are determined. Communication events occurring within the same time slot are grouped into a communication event set. ;in, Let τ represent the τth communication event group; in the resource management phase, the event group is used as the decision step, and each event group is regarded as an MDP step size. The MAPPO-MSC algorithm is used to optimize the communication resources of multiple UAVs. The state space is: for the τ-th communication event group, the UAV Resource management local observation is defined as ； in, and These represent the distance from the roadside unit to the drone and the distance from the drone to the base station, respectively. and These represent the channel gain from the roadside unit to the drone and from the drone to the base station, respectively. and These represent the data size of the image captured by the drone and the data size of the image transmitted from the roadside unit to the drone, respectively. Indicates drone Remaining communication energy before the event is executed; Action space includes continuous action space and discrete action space ; The reward function consists of two parts: the first is the experience quality reward, the second is the penalty when the semantic transmission accuracy of the communication between the drone and the base station does not meet the expected standard, and the third is the penalty when the semantic transmission accuracy of the communication between the roadside unit and the drone does not meet the expected standard. ； Using a communication event group as a resource management decision step, construct a resource management Actor network and a centralized Critic network, and initialize the Actor network parameters. Critic network parameters Discount Factor Generalized dominance estimation parameters and strategy clipping parameters ; The Actor network consists of continuous action branches and discrete action branches; the continuous action branches are based on local observations. Sampling spectrum bandwidth and transmit power: ； Discrete action branch output semantic symbol number selects the corresponding logical variable The Gumbel-Softmax method is used to generate discrete semantic resource actions. ； The complete resource management actions of UAV n consist of continuous and discrete actions: Input environment to obtain local observations in the next time slot global state and rewards And store the interactive experience in the experience cache area; Calculate the timing difference error of the τth communication event group based on the centralized Critic network: ； Based on the aforementioned time-series differential error, calculate the generalized advantage estimate and return estimate for the resource management phase: ； ； Define the probability ratio between the current hybrid action policy and the old policy before the update as: ； The objective function of the pruning strategy in the resource management Actor network is: ； The value loss function of the resource management Critic network is: ； Update the Critic network and Actor network ； 。

Images