A semantic communication multi-agent reinforcement learning collaborative control method based on a visual language model drive

Abstract

The application discloses a kind of based on visual language model driven semantic communication multi-agent reinforcement learning collaborative control method, comprising: S1.Semantic perception: through knowledge distillation constructs light semantic encoder, real-time extraction first semantic feature;S2.Semantic communication: broadcast first semantic feature, receiving end according to channel quality adaptive degradation processing, generate second semantic feature and communication mask;S3.Cross-modal fusion: through bidirectional cross attention mechanism first, second semantic feature and dynamics state are fused, and output joint feature;S4.Policy optimization: based on first semantic feature constructs internal reward, combined with external reward to form total reward, using centralized training, decentralized execution paradigm, through heterogeneous agent near-end policy optimization algorithm carries out end-to-end training.The application solves the collaborative perception, semantic communication and decision optimization problem in the scene of limited edge computing power and extremely low communication bandwidth, and is suitable for heterogeneous multi-agent collaborative task.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence, multi-agent systems, and semantic communication, specifically to a multi-agent reinforcement learning collaborative control method for semantic communication based on a visual language model-driven approach. Background Technology

[0002] Multi-agent reinforcement learning (MARL) has broad application prospects in fields such as cooperative navigation, search and rescue, and formation control. However, existing technologies suffer from the following four main drawbacks: First, there are limitations in perception methods. Existing MARL systems mainly rely on manually designed low-dimensional features, such as grid maps, coordinate vectors, and sensor distance values. These features only contain geometric information and cannot express the high-level semantics of the scene. For example, the system cannot understand semantic information such as "there is a dead end ahead and you need to turn around" or "the left passage leads to the exit." The lack of semantic understanding leads to low policy learning efficiency, and the agent needs a lot of trial and error to learn basic navigation behaviors.

[0003] Second, the communication mechanism is limited. In classic multi-agent communication methods such as CommNet, TarMAC, and QMIX, agents transmit uninterpretable hidden state vectors or complete observation data. This presents two problems: first, bandwidth increases linearly with the observation dimension, which is impractical in real-world deployment scenarios with low bandwidth and high packet loss; second, it transmits raw numerical values ​​rather than semantic concepts, making it difficult for the receiver to effectively utilize this information for collaborative decision-making.

[0004] Third, the sparse reward problem is severe. In complex navigation scenarios, an agent may need to take hundreds of steps to reach the goal, receiving a positive reward only upon reaching the destination. Existing potential field reward shaping methods only utilize Euclidean distance or path distance, without associating them with the semantics of the scene. For example, "approaching a wall in a dead end" and "approaching a passage leading to an exit" may have the same distance metric as "approaching a passage leading to an exit," but their semantic meanings are completely different, and existing methods cannot distinguish between these two cases.

[0005] Fourth, there is a lack of effective means for multimodal fusion. The semantic tokens output by visual language models are high-dimensional, discrete representations, while the robot's dynamic states are low-dimensional, continuous vectors. These two modalities differ significantly in dimensionality, distribution, and physical meaning. Simple vector concatenation will lose the interaction between modalities, making it impossible for semantic information to truly guide motion decisions.

[0006] Therefore, there is an urgent need for an integrated multi-agent collaborative control scheme that can achieve semantic-level perception, efficient communication, deep information fusion, and intelligent decision-making under resource-constrained conditions. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a semantic communication multi-agent reinforcement learning cooperative control method based on visual language model-driven method. This method can achieve efficient cooperative perception, adaptive semantic communication and end-to-end cooperative decision optimization in heterogeneous multi-agent systems with limited edge computing power and extremely low communication bandwidth, thereby overcoming the problems of perception limitations, inefficient communication, sparse rewards and difficulties in fusion in the background technology.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows.

[0009] A multi-agent reinforcement learning cooperative control method for semantic communication based on a visual language model includes the following steps: S1. Semantic perception: Based on the visual language model VLM, a lightweight semantic encoder is constructed and deployed on the edge of each agent through knowledge distillation, so that each agent can extract the first semantic feature in real time based on its own visual observation; S2. Semantic Communication: Each agent broadcasts its first semantic feature to other agents via a wireless channel; the receiving agent performs adaptive downgrading processing on the received semantic feature according to the channel quality to obtain the second semantic feature and generates a communication mask that characterizes the communication quality. S3. Cross-modal fusion: Each agent inputs its own first semantic feature, the received second semantic feature, and its own dynamic state into the cross-modal fusion module. Through a bidirectional cross-attention mechanism, the semantic modality and the dynamic state modality are exchanged and aligned, and the fused joint features are output. S4. Policy Learning and Optimization: Construct semantically driven intrinsic rewards based on the first semantic features, combine them with extrinsic rewards provided by the environment to form a total reward signal; take the total reward as the optimization objective, adopt a centralized training and decentralized execution paradigm, and use joint features and communication masks to train the multi-agent policy network end-to-end through a heterogeneous agent proximal policy optimization algorithm.

[0010] Preferably, step S1 specifically includes: S11. Encode the observed images and structured text descriptions of the training scene using a large teacher VLM, extract high-dimensional teacher semantic features, and generate corresponding scene text description embeddings; S12. Construct a student-side semantic projection head network with fewer parameters than the teacher's VLM; S13. Train the student-side semantic projection head network using a joint distillation loss function; the joint distillation loss function includes at least: a feature regression loss to make the output of the student-side semantic projection head network approximate the dimensionality reduction result of the teacher's semantic features, a KL divergence soft label loss to align the classification-level behavior, and a cross-modal contrast loss to align the output of the student-side semantic projection head network with the scene text description embedded in the feature space. S14. After training, the student-side semantic projection head network is deployed as a lightweight semantic encoder on each agent to extract low-dimensional, fixed-length first semantic features in real time.

[0011] Preferably, the student-end semantic projection head network is configured to output full semantic features and coarse-grained semantic features of partial dimensions extracted from the full semantic features; in step S2, the sending agent selects to broadcast either the full semantic features or the coarse-grained semantic features based on the channel quality.

[0012] Preferably, in step S2, adaptive degradation processing based on channel quality specifically includes: S21. Calculate the packet loss probability of the current channel based on the distance between the sender and receiver agents; S22. Sample a random number, and based on the random number, the packet loss probability, and a preset coarse-grained switching threshold, determine one of the following three transmission modes: If the random number is less than or equal to the packet loss probability, it is determined to be a packet loss mode; If the random number is greater than the packet loss probability and the packet loss probability is less than the coarse-grained switching threshold, then the decision is to use full transmission mode. If the random number is greater than the packet loss probability, and the packet loss probability is greater than or equal to the coarse-grained switching threshold, then the decision is to switch to coarse-grained transmission mode. Furthermore, in full-volume transmission mode, the complete first semantic feature is transmitted and channel noise is superimposed; in coarse-grained transmission mode, only a preset portion of the first semantic feature is transmitted, the remaining dimensions are set to zero, and channel noise is superimposed; in packet loss mode, the receiver obtains a vector of all zeros. S23. The values ​​of the communication mask are mapped one-to-one with the transmission modes to identify the communication quality of the current channel.

[0013] Preferably, in step S3, the cross-modal fusion module employs a bidirectional cross-modal attention mechanism, specifically including: S31. Modal projection: Projecting the semantic feature set and the dynamic state vector onto a unified feature dimension respectively; S32. Bidirectional Attention Calculation: Perform attention computation for semantically guided dynamics, using the dynamic state vector as the query vector and the set of semantic features as the key and value vectors; Perform attention computation guided by dynamics, using semantic sub-tokens in the semantic feature set as query vectors and dynamic state vectors as key and value vectors; S33. Gated Fusion: The semantic features and dynamic features calculated by bidirectional attention are pooled separately to obtain semantic pooling vectors and dynamic pooling vectors; the semantic pooling vectors and dynamic pooling vectors are weighted and summed by a learnable gating vector to generate joint features; the bias term of the gating vector is initialized to a negative value so that the joint features can automatically and smoothly tend to depend on the dynamic pooling vector in the early stage of training or when the semantic signal fails due to communication degradation or loss.

[0014] Preferably, in step S4, the semantically driven intrinsic reward includes at least one of the following: Target alignment reward: obtained by calculating the similarity between the current first semantic feature after mapping and the predefined task target text embedding corresponding to the agent type; Semantic novelty reward: calculated through a random network distillation model; the random network distillation model includes a target network with fixed parameters and a trainable prediction network, and the semantic novelty reward is the error between the prediction output of the prediction network for the current first semantic feature and the output of the target network; Information gain reward: obtained by calculating the reduction in information entropy of unexplored areas caused by the reconnaissance agent during the exploration process.

[0015] Preferably, in step S4, the optimization process of the heterogeneous agent proximal policy optimization algorithm includes: Each agent uses an independent actor network, whose input is the joint features output from step S3; A centralized network of commentators is used, whose inputs are the observation information of all agents, the second semantic features received from other agents, and the global state formed by concatenating the corresponding communication masks; When updating the policy, each agent proceeds in a preset order. When updating the current agent, the importance sampling weight correction factor is calculated using the new policy of the updated agent.

[0016] Preferably, the total loss function for policy optimization in step S4 includes a communication sparsity regularization term to encourage the student-side semantic projection head network to output sparse first semantic features in order to reduce the effective communication load.

[0017] Preferably, the training process in step S4 further includes a course learning mechanism: multiple training stages with different environmental parameter configurations are preset; during the training process, the task success rate of the agent is monitored, and when the success rate meets the preset conditions, the training environment is automatically switched to the next stage with higher difficulty.

[0018] Due to the adoption of the above technical solutions, the technical progress achieved by this invention is as follows.

[0019] This invention enhances the semantic level of perception and decision-making: through knowledge distillation, a lightweight edge encoder achieves semantic understanding capabilities approaching those of a large-scale Virtual Machine (VLM). The agent can extract high-level semantic features such as "passable areas" and "potential collision risks," significantly improving the sample efficiency of policy learning and the rationality of the agent's decisions in complex environments.

[0020] This invention achieves high bandwidth efficiency and communication robustness: the proposed layered semantic communication protocol transmits only low-dimensional, interpretable semantic features, resulting in extremely low communication load. Adaptive transmission degradation and packet loss handling mechanisms enable the system to maintain cooperative performance in real-world wireless channels with fluctuating quality, achieving a balance between communication efficiency and robustness.

[0021] This invention effectively alleviates the sparse reward problem: by utilizing multiple intrinsic rewards (target alignment, semantic novelty, and information gain) constructed using VLM semantics, it provides the agent with rich and immediate learning signals before obtaining the final task reward. This guides the agent to conduct more purposeful exploration, significantly accelerates training convergence, and improves the performance of the final policy.

[0022] This invention achieves deep cross-modal information fusion and secure rollback: a bidirectional cross-modal attention mechanism enables deep interaction and alignment of semantic and dynamic information, allowing high-level semantics to accurately guide low-level motion control. In particular, the negative initialization design of the bias term in the gating fusion mechanism ensures that in the early stages of training, during communication interruptions, or when semantic features fail, the system can automatically and smoothly roll back to relying on stable dynamic features for decision-making, without requiring additional fault detection logic, greatly enhancing the system's practicality and security.

[0023] This invention has strong generalization ability: the course learning mechanism enables the agent to be trained step by step from simple to complex, and automatically adjusts the environmental difficulty by monitoring the task success rate, thereby improving the agent's adaptability and generalization performance in unknown environments.

[0024] This invention achieves end-to-end collaborative optimization: it employs a heterogeneous agent near-end policy optimization algorithm for centralized training and decentralized execution, with each agent updating its policy sequentially, making full use of the new policy information of the updated agents to achieve efficient collaborative optimization of heterogeneous multi-agent systems. Attached Figure Description

[0025] Figure 1 This is the process of the present invention. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0027] A multi-agent reinforcement learning cooperative control method based on visual language model-driven semantic communication, combined with... Figure 1 As shown, it includes the following steps: S1. Semantic perception: Based on the visual language model (VLM), a lightweight semantic encoder is constructed and deployed on the edge of each agent through knowledge distillation, enabling each agent to extract the first semantic feature in real time based on its own visual observation.

[0028] This step is specifically implemented based on the Teacher-Student knowledge distillation framework, aiming to address the issue of limited edge computing power preventing the direct execution of large-scale VLMs. In heterogeneous multi-agent systems, each agent is equipped with a visual sensor (such as a camera), but limited edge computing power prevents the direct execution of large-scale VLMs. Therefore, the semantic extraction capabilities of large-scale VLMs need to be migrated to a lightweight projection head network through offline distillation, enabling edge devices to output semantic representations in real-time at a high frame rate. Specifically, this includes: S11. Encode the observed images and structured text descriptions of the training scene using a large teacher VLM, extract high-dimensional teacher semantic features, and generate corresponding scene text description embeddings.

[0029] First, a large teacher VLM is deployed on the GPU server. In this embodiment, the Qwen3-VL-8B-Instruct model is used. The inference process of the teacher VLM is as follows: Step 1: Constructing Multimodal Inputs. For each training scene state, construct a multimodal input pair containing an image and text. The image comes from the agent's visual sensor RGB image; the text is a structured scene description prompt, including agent type, current position coordinates, orientation angle, visible obstacle information, and task objective.

[0030] Step 2: VLM Multimodal Encoding. The image and prompt are fed into the teacher VLM. The visual encoder inside the model performs patch segmentation and feature extraction on the image, and the text encoder tokenizes the prompt. Then, the visual and text information are fused through multi-layer Transformer cross-attention.

[0031] Step 3: Extract semantic features. Take the last hidden state of the teacher's VLM and perform attention-mask-based average pooling on the sequence dimension to obtain the teacher's semantic features. This vector encapsulates the scene's spatial layout, obstacle distribution, and navigation-related semantic information.

[0032] Step 4: Generate Text Description Embedding. The teacher VLM simultaneously generates a natural language description of the scene, which is then encoded into a text embedding using a pre-trained sentence encoder. This is used for subsequent cross-modal contrastive learning.

[0033] S12. Construct a student-side semantic projection head network with significantly fewer parameters than the teacher's VLM.

[0034] The structure of the student-side semantic projection head network is as follows: in, d s =64, which is the dimension of the output semantic token. The total number of projection head parameters is approximately 4096×512+512×64≈2.1M, which can be deployed on an embedded platform.

[0035] The student-side semantic projection head network is configured to output semantic features at two granularities: ① Full semantic features: 64 dimensions, containing complete semantic information, denoted as .

[0036] ② Coarse-grained semantic features: These are partial dimensions extracted from the full set of semantic features. Specifically, the first 16 dimensions of the full set of semantic features are taken, denoted as... Transmission occurs when communication quality is poor.

[0037] S13. The student-side semantic projection head network is trained using the joint distillation loss function.

[0038] The joint distillation loss function includes at least: feature regression loss, KL divergence soft label loss, and cross-modal contrast loss. The feature regression loss is used to make the output of the student semantic projection head network approximate the dimensionality reduction result of the teacher's semantic features; the KL divergence soft label loss is used to align the classification level behavior; and the cross-modal contrast loss is used to align the output of the student semantic projection head network with the scene text description embedded in the feature space.

[0039] Specifically, the distillation stage employs a three-term joint loss to train the student-side semantic projection head network: in, λ1 represents the total distillation loss; λ1 represents the feature regression loss weight. The feature regression loss is λ; the weights for the λ2KL divergence loss are λ; λ3 represents the KL divergence soft label loss; λ3 represents the cross-modal contrastive loss weights. For InfoNCE cross-modal contrast loss.

[0040] ① Feature regression loss: Wherein, g: It is a learnable linear projection layer that maps high-dimensional teacher features to a low-dimensional student space. This allows the student output to numerically approximate the teacher's dimensionality-reduced features. B It is the mini-batch size.

[0041] ②KL divergence soft label loss: in, p S and p T These are the softmax output distributions of students and teachers for scene categories / attributes, using temperature. T Scaling. This item aligns the behavior of both at the classification level.

[0042] ③InfoNCE cross-modal contrast loss: in, It is cosine similarity; =0.07, which is the temperature parameter; The student's visual token and the teacher's text embedding in the same scene constitute a positive sample pair. Other scenes within the same batch... This creates negative sample pairs. This step pulls the student's visual tokens into the textual semantic embedding space of the same scene, achieving cross-modal alignment.

[0043] S14. After training, the student-side semantic projection head network is deployed as a lightweight semantic encoder on each agent to extract low-dimensional, fixed-length first semantic features in real time.

[0044] S2. Semantic Communication: Each agent broadcasts its first semantic feature to other agents via a wireless channel; the receiving agent performs adaptive degradation processing on the received semantic feature according to the channel quality to obtain the second semantic feature and generates a communication mask that characterizes the communication quality.

[0045] The purpose of this step is to achieve extremely low-bandwidth semantic communication between agents and adaptively degrade based on channel quality. In step one, each agent was already able to output a fixed-length semantic token. In multi-agent collaborative scenarios, agents need to share perceptual information to make collaborative decisions. Traditional methods directly transmit complete observations or hidden states, which requires large bandwidth and is not interpretable. This step designs a hierarchical semantic communication protocol: agents only transmit semantic tokens and adaptively degrade based on channel quality.

[0046] In this step, the sending agent chooses to broadcast either full semantic features or coarse-grained semantic features based on the channel quality.

[0047] In this step, adaptive degradation processing is performed based on channel quality, specifically including: S21. Calculate the packet loss probability of the current channel based on the distance between the sender and receiver agents.

[0048] This invention employs a distance-dependent lossy channel model. For any two intelligent agents... i (Sender) and j The packet loss probability between (receivers) is: in: The Euclidean distance between the two agents; This is the distance attenuation coefficient; The base packet loss rate (the lowest packet loss rate when the distance is zero). This is the upper limit for packet loss rate to prevent complete channel interruption.

[0049] This model reflects the physical fact that the farther away the agent is, the higher the packet loss rate of the wireless channel.

[0050] S22. Sample a random number, and based on the random number, the packet loss probability, and a preset coarse-grained switching threshold, determine one of the following three transmission modes: If the random number is less than or equal to the packet loss probability, it is determined to be a packet loss mode; If the random number is greater than the packet loss probability and the packet loss probability is less than the coarse-grained switching threshold, then the decision is to use full transmission mode. If the random number is greater than the packet loss probability, and the packet loss probability is greater than or equal to the coarse-grained switching threshold, then the decision is to switch to coarse-grained transmission mode.

[0051] In full-volume transmission mode, the complete first semantic feature is transmitted and channel noise is superimposed; in coarse-grained transmission mode, only a preset portion of the first semantic feature is transmitted, the remaining dimensions are set to zero, and channel noise is superimposed; in packet loss mode, the receiver obtains a vector of all zeros.

[0052] Specifically, for a single action from an intelligent agent j To intelligent agents i For transmission, the system first samples a random number. u ~Uniform(0,1), and based on this random number, the packet loss probability, and the preset coarse-grained switching threshold y=0.3, determine one of the following three transmission modes: ① Packet loss mode: u ≤ p drop , the receiver obtains a zero vector , communication mask = 0.0.

[0053] ② Coarse-grained transmission mode: If u > p drop and p drop ≥y, the channel quality is poor but not completely disconnected. At this time, only the first d c = 16 dimensions (coarse-grained part) of the semantic token are transmitted, and the latter d s − d c = 48 dimensions are set to zero, and then Gaussian noise is superimposed. Communication mask m = 0.5.

[0054] ③ Full-scale transmission mode: If u > p drop and p drop <y, the channel quality is good, and the complete 64-dimensional semantic token is transmitted, and Gaussian noise is superimposed. Communication mask m = 1.0.

[0055] Among them, , ; y = 0.3 is the coarse-grained switching threshold.

[0056] For a system containing N agents (in this embodiment N = 3), each agent broadcasts its first semantic feature to all other agents at each step. The receiver i collects signals from the other N −1 agents to form a received token matrix R i and a communication mask vector m i , which are used as additional inputs to the centralized Critic to provide multi-agent communication state information for the value function. Specifically as follows: Among them, is the received semantic feature sent from agent 1 to agent i ; is the received semantic feature sent from agent N to agent i ; m 1→i is from agent 1 to agenti The communication mask for this link; m N→i To the intelligent agent N To intelligent agents i The communication mask for this link.

[0057] S23. The values ​​of the communication mask are mapped one-to-one with the transmission modes to identify the communication quality of the current channel.

[0058] Specifically, the packet loss mode corresponds to m=0.0, the coarse-grained transmission mode corresponds to m=0.5, and the full transmission mode corresponds to m=1.0.

[0059] S3. Cross-modal fusion: Each agent inputs its own first semantic feature, the received second semantic feature, and its own dynamic state into the cross-modal fusion module. Through a bidirectional cross-attention mechanism, the semantic modality and the dynamic state modality are exchanged and aligned, and the fused joint feature is output.

[0060] The purpose of this step is to deeply integrate semantic features with dynamic states and ensure system robustness in the event of communication failure. After steps S1 and S2, each agent possesses three information sources: one is its own semantic token. (From the Student encoder); secondly, second semantic features received from other agents; and thirdly, its own dynamic state. (From odometry sensor). Semantic and dynamic modalities differ significantly in dimensionality, numerical range, and physical meaning. Simple concatenation cannot capture their interaction. Therefore, this step designs a bidirectional cross-modal attention (BiCMA) mechanism, allowing the two modalities to "ask" each other for information, achieving bidirectional alignment and fusion.

[0061] In this step, the cross-modal fusion module employs a bidirectional cross-modal attention mechanism, specifically including: S31. Modal projection: Projecting the semantic feature set and the dynamic state vector onto a unified feature dimension respectively.

[0062] First, project the two modalities onto a unified model. d model =128-dimensional space.

[0063] ① Semantic projection: Projecting and splitting the semantic feature set composed of the first semantic feature and the second semantic feature into... N s =4 sub-tokens: Where S is the semantic sub-token matrix; zS W is the first semantic feature; sem b is the semantic projection weight matrix; sem For semantic projection bias; LayerNorm is for layer normalization; ②Dynamic Dimensional Upgrading: Upgrading the 6-dimensional dynamic state to... d model : Where D is the dynamic token; x odom W is the dynamic state vector; dyn b is the dynamic projection weight matrix; dyn is the dynamic projection bias term; GELU is the Gaussian error linear unit activation function.

[0064] After projection, the semantic modality is a 4×128 matrix (4 sub-tokens), and the dynamic modality is a 1×128 matrix (1 token).

[0065] S32. Bidirectional Attention Calculation: Perform attention computation for semantically guided dynamics, using the dynamic state vector as the query vector and the set of semantic features as the key and value vectors; Perform dynamic-guided semantic attention computation, using semantic sub-tokens in the semantic feature set as query vectors and dynamic state vectors as key and value vectors.

[0066] Specifically, using multi-head attention ( n heads =4, each head d k =32) Perform cross-attention in two directions.

[0067] Direction 1: Semantic-Guided Dynamics (S→D). This approach uses the dynamic state vector as the query vector and the semantic feature set as the key and value vectors. Intuitively, it allows the dynamic state to "ask" the semantics: What does the current position and velocity mean semantically? Among them, W Q To query the weight matrix; W K W is the key weight matrix; V This is the weight matrix; d k is the attention head dimension; softmax is the attention weight; LN is the layer normalization; D′ is the dynamic feature for semantic enhancement.

[0068] Direction Two: Dynamics-Guided Semantics (D→S). This approach uses semantic sub-tokens from the semantic feature set as query vectors and dynamic state vectors as key and value vectors. Intuitively, the semantic tokens "ask" the dynamics: In the current motion state, which semantic features are most relevant? Here, S′ represents the semantic feature for dynamic enhancement.

[0069] Both directions include residual connections and LayerNorm to prevent gradient vanishing.

[0070] S33. Gated Fusion: The semantic features and dynamic features calculated by bidirectional attention are pooled separately to obtain semantic pooling vectors and dynamic pooling vectors. The semantic pooling vectors and dynamic pooling vectors are weighted and summed by a learnable gating vector to generate joint features. The bias term of the gating vector is initialized to a negative value so that the joint features can automatically and smoothly tend to depend on the dynamic pooling vector in the early stage of training or when the semantic signal fails due to communication degradation or loss.

[0071] Specifically, the output of bidirectional attention is pooled and compressed to a unified dimension: in, For the first k A semantic sub-token; N s S represents the number of semantic sub-tokens; pool This is a semantic pooling vector; For dynamic tokens; d pool This is the dynamic pooling vector.

[0072] Finally, learnable gating is used to mix the information from the two modalities: in, This is the gated weight matrix; This is a gated bias term; This indicates concatenation; σ is the Sigmoid activation function. ⊙ represents the gated vector; h represents element-wise multiplication; fused This is a joint feature.

[0073] Gating bias Initialized to -2.0. This makes σ(-2.0) ≈ 0.12 initially, meaning the gate value is close to zero, and the fused output is almost equal to the pure dynamic feature d. pool This design has two advantages: ① When VLM distillation is insufficient and semantic tokens have not yet learned meaningful information, the system does not rely on semantics but makes decisions using stable dynamic features. As training progresses, the gating value gradually learns an appropriate mixing ratio.

[0074] ② During deployment, if communication is interrupted (semantic tokens are all zero) or VLM output degrades (noise / weak signal), S pool When the amplitude is close to zero, the gating automatically stays low, and the system smoothly returns to pure dynamic control without the need for additional fault detection and switching logic.

[0075] S4. Policy Learning and Optimization: Construct semantically driven intrinsic rewards based on the first semantic features, combine them with extrinsic rewards provided by the environment to form a total reward signal; take the total reward as the optimization objective, adopt a centralized training and decentralized execution paradigm, and use joint features and communication masks to train the multi-agent policy network end-to-end through a heterogeneous agent proximal policy optimization algorithm.

[0076] The purpose of this step is to design better reward signals using semantic information and to perform end-to-end policy optimization for multiple heterogeneous agents. The first three steps have already established a complete data pathway from "perception → communication → fusion". This step addresses how to design better reward signals using semantic information and how to perform end-to-end policy optimization for multiple heterogeneous agents. In complex navigation scenarios, extrinsic task rewards (such as scores for reaching the target) are very sparse. This invention utilizes the semantic understanding capabilities of VLM to construct three intrinsic rewards, enabling agents to obtain meaningful feedback during the exploration phase.

[0077] In this step, the semantically driven intrinsic reward includes at least one of the following: The general formula for intrinsic reward: in, for t The intrinsic reward of each moment; r goal Align rewards with targets; w 1 represents the target alignment reward weight; r novelty Rewards are given for semantic novelty. w 2 represents the semantic novelty reward weight; r info Rewards are given for information gain. w 3 represents the information gain reward weight.

[0078] ①Target alignment reward: obtained by calculating the similarity between the current first semantic feature after mapping and the predefined task target text embedding of the corresponding agent type.

[0079] Objective: To measure how semantically close the currently observed scene is to the task objective.

[0080] Calculation process: Step 1: Prepare the target embedding. Before training begins, use a Sentence Encoder (output...) d e (384-dimensional) Encode the task objective description text for each agent type into a fixed vector. For example, the objective description for a UAV is "to scout a double maze from the air and reveal the fog of war to teammates." These vectors are not updated during training.

[0081] Step 2: Map the Student token. This involves mapping the first semantic feature. Mapped to the sentence embedding space via an alignment head: in, For the mapped semantic embedding; for t The first semantic feature of a given moment; f align To align the head network.

[0082] The structure of the Alignment Head is Linear(64,256)→GELU→Linear(256,384).

[0083] Step 3: Calculate cosine similarity: Among them, e goal Embed the text for the task objective.

[0084] The closer the scene currently perceived by the agent is to the target semantically (e.g., the passage near the exit has been seen). r goal The larger.

[0085] ② Semantic novelty reward: Calculated through a random network distillation model; the random network distillation model contains a target network with fixed parameters and a trainable prediction network. The semantic novelty reward is the error between the prediction output of the prediction network for the current first semantic feature and the output of the target network.

[0086] Objective: To encourage agents to explore semantically "unseen" scene regions and avoid repeatedly lingering in known areas.

[0087] Calculation process: Step 1: Construct two networks with identical structures. (Fixed target network)f fixed Parameters are frozen after random initialization and not updated; a trainable prediction network can be trained. f pred Updated during training. Both have the same structure: Linear(64,128) → ReLU → Linear(128,64).

[0088] Step 2: Perform forward propagation on the current token and calculate the prediction error: The principle is: if an intelligent agent frequently passes through a certain type of scene, f pred The fit on those tokens becomes increasingly accurate, the error decreases, and the reward decreases. However, for tokens in "novel" scenarios, f pred They haven't learned to predict yet, so the error is relatively large, which generates positive incentives for exploration.

[0089] Step 3: Update the prediction network with the collected tokens in batches: Among them, z i Let be the semantic features of the i-th sample; This is the RND loss function.

[0090] Use the Adam optimizer (learning rate) )renew f pred , f fixed It remains unchanged.

[0091] ③Information gain reward: This is obtained by calculating the reduction in information entropy of unexplored areas caused by the reconnaissance agent during the exploration process.

[0092] Objective: To incentivize reconnaissance agents (such as UAVs) to proactively explore unknown areas and uncover more information.

[0093] Calculation process: in, H (fog) represents the information entropy of an unexplored area. When a reconnaissance agent reveals a new area, H A decrease in value results in a positive difference, generating a positive reward. Guarantee that the reward is non-negative.

[0094] This item is only calculated for agents performing reconnaissance missions; for other agents, this item is zero.

[0095] ④ Total Reward Composition: Each step of the intelligent agent i The total reward is the weighted sum of extrinsic and intrinsic rewards: in, β =0.1 is the intrinsic reward coefficient; for t Time-based intelligent agent i Total reward; for t Time-based intelligent agent i External rewards; for t Time-based intelligent agent i The intrinsic reward.

[0096] External rewards Defined by specific tasks (such as positive rewards for reaching the goal, negative penalties for collisions, etc.). β Taking a smaller value ensures that intrinsic rewards play a supporting and guiding role, rather than dominating the strategy learning objective.

[0097] In this step, the optimization process of the heterogeneous agent proximal policy optimization algorithm includes: Each agent uses an independent actor network, whose input is the joint features output from step S3.

[0098] A centralized network of commentators is used, whose inputs are the observation information of all agents, the second semantic features received from other agents, and the global state composed of the corresponding communication masks.

[0099] When updating the policy, each agent proceeds in a preset order. When updating the current agent, the importance sampling weight correction factor is calculated using the new policy of the updated agent.

[0100] Specifically, this invention employs HAPPO (Heterogeneous-Agent Proximal Policy Optimization) for multi-agent policy optimization, following the CTDE (Centralized Training, Decentralized Execution) paradigm.

[0101] ①Actor Network (decentralized, each agent has its own independent system): Each agent i There is an independent Actor network π i The input is the agent's local observation. i In VLM-SC-MARL mode, observations are split into semantic tokens and dynamic states, which are first fused using BiCMA (step S3) to obtain... After passing through an optional GRU sequential memory layer, the action 'a' is finally output through an action output layer (continuous actions are parameterized using a Gaussian distribution, and discrete actions are parameterized using a Categorical distribution). i .

[0102] ②Critic network (centralized, using global information during training): Critic takes as input the concatenated observations (share_obs) from all agents, as well as the tokens and masks transmitted by other agents through the semantic communication channel: Among them, o share For globally shared observations; o1, o2, ..., o N For local observations of each agent; R i To receive the token matrix; m i This is the communication mask vector; V (o share ) is the output of the Critic network.

[0103] Critic uses an MLP to process this concatenated vector and outputs a scalar value estimate. .

[0104] ③ HAPPO sequential update mechanism: The core feature of HAPPO is that each agent updates the strategy in a fixed order. (Update agent) i At that time, using the previously updated agents 1,..., i The new strategy of -1 calculates the importance weight factor. Then, perform gradient updates using a PPO-style truncation target: in, For intelligent agents i Actor loss; For the desired operation; Importance sampling ratio; As an importance weighting correction factor; For GAE advantage estimation; For GAE advantage estimation; =0.2 is the truncation parameter.

[0105] ④ Total loss function: For each agent, the total loss for each round of PPO updates is: in, c v This is the value loss coefficient; To truncate the value function loss; c e The strategy entropy coefficient; For policy entropy (encourages exploration); λ comm For communication sparsity coefficients; L1 regularization for communication sparsity; For intelligent agents i The total loss.

[0106] The total loss function for policy optimization includes a communication sparsity regularization term, which encourages the student-side semantic projection head network to output sparse first semantic features to reduce the effective communication load.

[0107] In this step, the training process also includes a course learning mechanism: multiple training stages with different environmental parameter configurations are preset; during the training process, the success rate of the agent's tasks is monitored, and when the success rate meets the preset conditions, the training environment is automatically switched to the next stage with higher difficulty.

[0108] Specifically, the course manager maintains multiple stages, each defining parameters such as the maximum environment step size L, sensor noise standard deviation, task difficulty coefficient, and whether randomization is used. Stage promotion is based on recent success rate; after promotion, the environment configuration is updated, the experience replay buffer is rebuilt, and the environment is warmed up again. Example 1

[0109] This embodiment takes a typical heterogeneous multi-agent cooperative navigation task as an example to explain in detail the configuration, workflow and effect of the method of the present invention in a practical application scenario.

[0110] (1) Task scenario description In a simulation environment, a 64×64 grid-based maze is constructed. This environment includes both land and water terrain, and three types of heterogeneous intelligent agents are deployed to work collaboratively. UAV (Unmanned Aerial Vehicle): It has a global aerial view and its core mission is to explore and uncover the "fog of war" in the environment, and to plan and indicate accessible navigation sub-targets for ground intelligent agents (UGV / USV).

[0111] UGV (Unmanned Ground Vehicle): It travels in land areas, autonomously avoids obstacles, and eventually finds and reaches the land exit.

[0112] USV (Unmanned Surface Vessel): Navigates in water areas, autonomously avoiding obstacles on the water surface, and ultimately finding and reaching the water surface exit.

[0113] The ultimate mission objective is for UAVs, UGVs, and USVs to work together, with UAVs providing reconnaissance and guidance, to enable UGVs and USVs to reach their respective exits efficiently and safely.

[0114] (2) System observation and action space definition Local observation space: Each agent receives a 101-dimensional local observation vector. The structured splicing method is shown in Table 1 below: Table 1. Local Observation Spatial Composition

[0115] Globally shared observations: Input for centralized Critic networks It is composed of 101-dimensional local observations directly spliced ​​together by three intelligent agents.

[0116] Action Space: The action space of all agents is a 2-dimensional continuous space. However, the meaning is different when mapped to physical control quantities, as shown in Table 2 below: Table 2. Action Space Mapping Table for Intelligent Agents (3) Network structure parameter configuration In this embodiment, the specific parameter configurations for each core module are as follows: ①The Student projection head parameter configuration is shown in Table 3 below: Table 3 Student Projector Head Parameter Configuration Table ②The parameter configuration of the BiCMA fusion module is shown in Table 4 below: Table 4 BiCMA Fusion Module Parameter Configuration Table ③ The Actor network parameters are configured as shown in Table 5 below: Table 5 Actor Network Parameter Configuration Table ④ The Critic network parameter configuration is shown in Table 6 below: Table 6 Critic Network Parameter Configuration Table (4) Reward function design Environmental awards r env The reward is a weighted sum of multiple components. Each agent receives a different combination of reward items: ①UAV Rewards: in, r coverage To cover rewards; r guide To encourage quality rewards;r team Reward the team for its progress; r step Penalty for time step; r terminal To terminate the reward; r jerk Punish Jerk.

[0117] Specifically, the composition of the UAV reward function is shown in Table 7 below: Table 7. Composition of UAV Reward Function Where, Δ d UGV Δ represents the change in distance from the UGV to the target. d USV α represents the change in distance from the USV to the target. team The team progress coefficient; c step Base time penalty coefficient; c esc Escape time coefficient; Let be the norm of the accelerometer; c jerk This is the Jerk penalty coefficient.

[0118] ②UGV / USV Rewards: in, r pot Shaping rewards for A* potential fields; r obs Penalty for obstacles; r terrain Penalty for terrain boundaries; r follow Sub-target follow-up rewards; r collision As a collision penalty.

[0119] Specifically, the composition of the UGV / USV reward function is shown in Table 8 below: Table 8. Composition of UGV / USV Reward Function

[0120] Where Φ(s) is the potential field function; α is the potential field reward coefficient; and γ is the discount factor. d safe This is the safe distance threshold; d coll This is the collision distance threshold; k soft Soft penalty coefficient khard This is a hard penalty coefficient; α follow To follow the reward coefficient; d prev This represents the distance from the previous step to the sub-target. d cur Distance from the current step to the sub-target Total reward after mixing: Where β = 0.1. Intrinsic reward r int It is the sum of the three items in step S4 (target alignment reward, semantic novelty reward, and information gain reward).

[0121] (5) Training hyperparameters The key training hyperparameters used in this embodiment are shown in Table 9 below: Table 9 Key Training Hyperparameters (6) Single-step execution example At a certain moment during the training process t For example, let's demonstrate the complete flow of data within the system. Assume the environment is as follows: The UAV is located at (32,32,10), providing a panoramic view of the entire area; UGV is located at (12,8,0) and faces towards θ =0.5 rad, currently moving in a land area; The USV is located at (50, 45, 0), facing... θ =1.2 rad, currently sailing in the water area.

[0122] ① Environmental observation: The environment returns 101-dimensional observations to each agent. Taking UGV as an example, its observations include its own coordinates, orientation, and the distance to obstacles ahead detected by 36 ray sensors.

[0123] ② Semantic perception: The camera image of the UGV is input into a lightweight semantic encoder deployed on its edge, which extracts 64-dimensional first semantic features in real time. .

[0124] ③ Semantic communication: UGV broadcasts to UAV Calculate the distance between the two points as d ≈ 30 grids, and then determine the packet loss probability based on the channel model. p drop ≈0.95. Because p drop If y=0.3 and no actual packet loss occurs, the UAV enters coarse-grained transmission mode, receiving only the received packets. The first 16 dimensions are superimposed with noise, and the communication mask m is set to 0.5.

[0125] ④ Cross-modal fusion: Within the Actor network of UAV, the BiCMA module receives its own semantic features. Deterioration And its own dynamic state. Because the received UGV feature information is incomplete and the mask value is low, the gating mechanism (initial bias -2.0) automatically reduces the semantic weights, making the fused joint feature h... fused It relies more on the stable dynamic state of the UAV itself to avoid interference from low-quality communication.

[0126] ⑤ Action sampling: The Actor network of UAV is based on h fused Output motion distribution parameters, sample to obtain specific motion [0.28, -0.15], map to physical speed [0.42, -0.225] grids / step and execute.

[0127] ⑥ Reward Calculation: UGV extrinsic rewards: Calculate A* potential field change (+5.505), obstacle distance penalty (-0.75), sub-target following reward (+1.6), and time step penalty (-0.2), totaling... r env = +6.155.

[0128] UGV intrinsic rewards: based on Calculate the target alignment reward (hypothesis +0.45), RND novelty reward (hypothesis +0.72), and information gain is 0. The total is... r int = 0.81.

[0129] UGV Total Rewards: r t = 6.155 + 0.1 × 0.81 = 6.236.

[0130] Implementation Results: Through the above process, the system of this invention enables UAVs to effectively conduct reconnaissance and guidance using semantic understanding. UGVs / USVs can learn efficient and robust navigation strategies in complex heterogeneous cooperative tasks by combining their own state, semantic information from teammates, and environmental feedback. In particular, the hierarchical semantic communication and BiCMA gating fusion mechanism ensure that system decisions are not degraded by noisy information when communication quality is poor over long distances, demonstrating good practicality.

Claims

1. A multi-agent reinforcement learning cooperative control method for semantic communication based on a visual language model, characterized in that: Includes the following steps: S1. Semantic perception: Based on the visual language model VLM, a lightweight semantic encoder is constructed and deployed on the edge of each agent through knowledge distillation, so that each agent can extract the first semantic feature in real time based on its own visual observation; S2. Semantic Communication: Each agent broadcasts its first semantic feature to other agents via a wireless channel; the receiving agent performs adaptive downgrading processing on the received semantic feature according to the channel quality to obtain the second semantic feature and generates a communication mask that characterizes the communication quality. S3. Cross-modal fusion: Each agent inputs its own first semantic feature, the received second semantic feature, and its own dynamic state into the cross-modal fusion module. Through a bidirectional cross-attention mechanism, the semantic modality and the dynamic state modality are exchanged and aligned, and the fused joint features are output. S4. Policy Learning and Optimization: Construct semantically driven intrinsic rewards based on the first semantic features, and combine them with extrinsic rewards provided by the environment to form a total reward signal; With total reward as the optimization objective, a centralized training and decentralized execution paradigm is adopted. By utilizing joint features and communication masks, a heterogeneous agent proximal policy optimization algorithm is used to train the multi-agent policy network end-to-end.

2. The multi-agent reinforcement learning collaborative control method for semantic communication based on visual language model driven according to claim 1, characterized in that: Step S1 specifically includes: S11. Encode the observed images and structured text descriptions of the training scene using a large teacher VLM, extract high-dimensional teacher semantic features, and generate corresponding scene text description embeddings; S12. Construct a student-side semantic projection head network with fewer parameters than the teacher's VLM; S13. Train the student-side semantic projection head network using a joint distillation loss function; the joint distillation loss function includes at least: a feature regression loss to make the output of the student-side semantic projection head network approximate the dimensionality reduction result of the teacher's semantic features, a KL divergence soft label loss to align the classification-level behavior, and a cross-modal contrast loss to align the output of the student-side semantic projection head network with the scene text description embedded in the feature space. S14. After training, the student-side semantic projection head network is deployed as a lightweight semantic encoder on each agent to extract low-dimensional, fixed-length first semantic features in real time.

3. The multi-agent reinforcement learning collaborative control method for semantic communication based on visual language model driven according to claim 2, characterized in that: The student-side semantic projection head network is configured to output full semantic features and coarse-grained semantic features of partial dimensions extracted from the full semantic features. In step S2, the sending agent selects to broadcast either full semantic features or coarse-grained semantic features based on the channel quality.

4. The multi-agent reinforcement learning collaborative control method for semantic communication based on visual language model driven according to claim 1, characterized in that: In step S2, adaptive degradation processing based on channel quality is performed, specifically including: S21. Calculate the packet loss probability of the current channel based on the distance between the sender and receiver agents; S22. Sample a random number, and based on the random number, the packet loss probability, and a preset coarse-grained switching threshold, determine one of the following three transmission modes: If the random number is less than or equal to the packet loss probability, it is determined to be a packet loss mode; If the random number is greater than the packet loss probability and the packet loss probability is less than the coarse-grained switching threshold, then the decision is to use full transmission mode. If the random number is greater than the packet loss probability, and the packet loss probability is greater than or equal to the coarse-grained switching threshold, then the decision is to switch to coarse-grained transmission mode. Furthermore, in full-volume transmission mode, the complete first semantic feature is transmitted and channel noise is superimposed; in coarse-grained transmission mode, only a preset portion of the first semantic feature is transmitted, the remaining dimensions are set to zero, and channel noise is superimposed; in packet loss mode, the receiver obtains a vector of all zeros. S23. The values ​​of the communication mask are mapped one-to-one with the transmission modes to identify the communication quality of the current channel.

5. The multi-agent reinforcement learning collaborative control method for semantic communication based on visual language model driven according to claim 1, characterized in that: In step S3, the cross-modal fusion module employs a bidirectional cross-modal attention mechanism, specifically including: S31. Modal projection: Projecting the semantic feature set and the dynamic state vector onto a unified feature dimension respectively; S32. Bidirectional Attention Calculation: Perform attention computation for semantically guided dynamics, using the dynamic state vector as the query vector and the set of semantic features as the key and value vectors; Perform attention computation guided by dynamics, using semantic sub-tokens in the semantic feature set as query vectors and dynamic state vectors as key and value vectors; S33. Gated Fusion: The semantic features and dynamic features calculated by bidirectional attention are pooled separately to obtain semantic pooling vectors and dynamic pooling vectors; the semantic pooling vectors and dynamic pooling vectors are weighted and summed by a learnable gating vector to generate joint features; the bias term of the gating vector is initialized to a negative value so that the joint features can automatically and smoothly tend to depend on the dynamic pooling vector in the early stage of training or when the semantic signal fails due to communication degradation or loss.

6. The multi-agent reinforcement learning cooperative control method for semantic communication based on visual language model driven according to claim 1, characterized in that: In step S4, the semantically driven intrinsic reward includes at least one of the following: Target alignment reward: obtained by calculating the similarity between the current first semantic feature after mapping and the predefined task target text embedding corresponding to the agent type; Semantic novelty reward: calculated through a random network distillation model; the random network distillation model includes a target network with fixed parameters and a trainable prediction network, and the semantic novelty reward is the error between the prediction output of the prediction network for the current first semantic feature and the output of the target network; Information gain reward: obtained by calculating the reduction in information entropy of unexplored areas caused by the reconnaissance agent during the exploration process.

7. The multi-agent reinforcement learning cooperative control method for semantic communication based on visual language model driven according to claim 2, characterized in that: In step S4, the optimization process of the heterogeneous agent proximal policy optimization algorithm includes: Each agent uses an independent actor network, whose input is the joint features output from step S3; A centralized network of commentators is used, whose inputs are the observation information of all agents, the second semantic features received from other agents, and the global state formed by concatenating the corresponding communication masks; When updating the policy, each agent proceeds in a preset order. When updating the current agent, the importance sampling weight correction factor is calculated using the new policy of the updated agent.

8. The multi-agent reinforcement learning cooperative control method for semantic communication based on visual language model driven according to claim 7, characterized in that: The total loss function for policy optimization in step S4 includes a communication sparsity regularization term to encourage the student-side semantic projection head network to output sparse first semantic features in order to reduce the effective communication load.

9. The multi-agent reinforcement learning cooperative control method for semantic communication based on visual language model driven according to claim 1, characterized in that: The training process in step S4 also includes a course learning mechanism: multiple training stages with different environmental parameter configurations are preset. During training, the success rate of the agent's tasks is monitored. When the success rate meets the preset conditions, the training environment is automatically switched to the next stage with higher difficulty.

Images