Distributed multi-mode ISAC system design method

Abstract

The invention discloses a distributed multi-modal ISAC system design method, and aims to solve the problem that the representation capability of existing single-modal sensing is limited in a complex environment, and a multi-modal cooperative sensing architecture is constructed by integrating heterogeneous sensors such as radio frequency, vision and laser radar and by using a large AI model, semantic communication and a multi-agent system technology. The architecture is based on an interactive mode (I-MAC), and supports direct semantic interaction and collaborative reasoning between edge agents. According to the invention, through the advantage complementation of multiple modes, the communication overhead is effectively reduced, the sensing precision is improved by about 80%, and the robustness and adaptability of the system in a dynamic environment are obviously enhanced.

Description

Technical Field

[0001] This invention relates to the field of multimodal sensing technology, and in particular to a design method for a distributed multimodal ISAC system. Background Technology

[0002] Integrated Communication and Sensing (ISAC), as an emerging paradigm, unifies wireless sensing and communication functions through shared hardware and spectrum framework. This technology utilizes radio frequency (RF) signals to perform both tasks simultaneously, effectively improving resource utilization, reducing hardware redundancy, and significantly enhancing spectrum efficiency, providing a highly promising solution for the development of future intelligent wireless systems. However, traditional ISAC schemes still primarily rely on RF signals as the sole means of sensing, and the limitations of this single-mode approach are becoming increasingly apparent.

[0003] Radio frequency (RF) technology has limitations in complex environments. It can only provide a single-modal perspective, acquiring limited information such as distance, velocity, and coarse spatial features, making it difficult to capture fine semantic or contextual details in complex environments. Furthermore, RF sensing technologies often struggle to adapt to rapidly changing or highly dynamic environments, and their performance can significantly degrade due to multipath propagation, signal interference, and obstruction. These limitations have prompted a paradigm shift from single-modal to multimodal sensing. Different types of sensors operate based on different principles and collect sensory data with varying characteristics. Typical examples of such sensing technologies include radar, lidar, red-green-blue depth cameras, and GPS, collectively known as multimodal sensing. By integrating the complementary advantages of different sensing modalities, multimodal sensing shows great potential in improving sensing accuracy, enhancing robustness, and generalization capabilities. For example, multimodal sensing can accelerate beam selection processes, such as lidar-assisted beam prediction, camera-GPS joint beam prediction, and camera-assisted beam prediction, whereas traditional methods require an exhaustive search of all candidate beam combinations. In addition, multimodal ISAC systems perform better in adapting to dynamic scenes. For example, visual data can effectively compensate for the attenuation of radio frequency signals under adverse conditions such as multipath fading or obstacles.

[0004] Based on the above analysis, the core challenges faced by multimodal ISAC technology can be summarized as follows:

[0005] (1) Heterogeneous Multimodal Fusion: Multimodal ISAC essentially requires the integration of data streams from different sensing modalities, such as radio frequency signals, visual sensors, and lidar. These modalities differ significantly in terms of data dimension, spatiotemporal resolution, coverage, and semantic abstraction. For example, aligning two-dimensional visual images with one-dimensional radar waveforms is particularly challenging due to differences in data structures, coordinate systems, and information content. Furthermore, inconsistencies in sensing frequencies and perception delays further complicate synchronous fusion.

[0006] (2) High communication overhead: In distributed ISAC systems, real-time sensing and decision-making require frequent and large-scale data exchange between sensors, edge nodes, and the central server. This leads to a surge in bandwidth consumption and energy consumption, especially in wireless networks or resource-constrained scenarios. The problem is particularly prominent in low-dynamic scenarios (such as static nighttime monitoring)—the system continues to transmit redundant or minimally informative data (such as continuously repeating video frames), wasting transmission resources and failing to improve situational awareness.

[0007] (3) Context-aware architecture design: Designing a system architecture that can adapt to different operating environments and task requirements remains a fundamental challenge. While rigid deployment is simple and efficient in terms of control and resource management, it often fails to meet the diverse needs of ISAC applications. Taking centralized architecture as an example, although it can rely on powerful computing infrastructure, it is prone to latency and dependency issues in scenarios with high timeliness requirements or in environments with weak infrastructure. While purely distributed systems have the advantage of autonomy, they often face shortcomings such as insufficient coordination and weak global situational awareness.

[0008] Therefore, it is necessary to develop a distributed multimodal ISAC system design method to solve the above problems. Summary of the Invention

[0009] The purpose of this invention is to design a distributed multimodal ISAC system to solve the above problems.

[0010] The present invention achieves the above objectives through the following technical solutions:

[0011] A design method for a distributed multimodal ISAC system includes the following steps:

[0012] S1, Multimodal local perception and feature embedding;

[0013] S2, Context building based on memory enhancement;

[0014] S3, Intent-oriented semantic encoding and interactive transmission;

[0015] S4. Neighbor information fusion based on attention mechanism;

[0016] S5, Distributed Collaborative Decision Making and Execution.

[0017] Specifically, S1 includes:

[0018] S11. The i-th edge agent collects environmental data through locally mounted sensors; let the visual image data collected at time t be... Radio frequency signal data is ;

[0019] S12. Utilizing the embedding layer of a large AI model, data from different modalities are mapped to the same high-dimensional feature space, and discrete raw data are projected onto the same continuous high-dimensional feature subspace, transforming them into a unified token sequence. Furthermore, learnable positional encoding vectors are superimposed on the feature sequence. The resulting local initial token sequence The calculation is as follows:

[0020] ;

[0021] in Represents the concatenation operation of vectors or matrices; This represents a linear projection embedding function for visual image data; This represents the raw visual image data acquired at time t; This represents a linear projection embedding function for radio frequency signal data; This represents the raw radio frequency signal data collected at time t.

[0022] Specifically, S2 includes:

[0023] S21. The agent maintains a memory state. It is used to store historical interaction information; at time t, a gated loop unit is used as the update function; combined with the current sensing input. Update memory:

[0024] ;

[0025] in This represents the memory state of the i-th agent after the update at time t; This represents the update function of the gated loop unit; This represents the memory state at the previous time step t-1;

[0026] S22. The agent, based on the current observation state, retrieves information from its local knowledge base. The system retrieves prior knowledge relevant to the current scene. Then, using prompting engineering techniques, it combines and structures the current perceptual features, updated historical memories, and retrieved domain knowledge to form a reasoning context containing complete spatiotemporal logic and rule constraints. :

[0027] ;

[0028] in This indicates a prompt for a project build operation or function.

[0029] Specifically, S3 specifically includes:

[0030] S31. When agent i needs to interact with neighboring agent j, the semantic encoding process is initiated; using a semantic encoder constructed from a deep neural network, the semantic encoding is derived from the reasoning context. The key intent features are mapped to a sequence of semantic symbols in the complex domain. :

[0031] ;

[0032] in This represents a semantic encoder function; This represents the network parameters of the semantic encoder; Let K represent the complex feature space, where K is the number of subcarriers. The length of the semantic symbol;

[0033] S32. Semantic symbols are transmitted to neighbor node j via a wireless channel; considering the unavoidable physical constraints in actual communication environments, the signal will experience multipath fading during transmission. Additive white Gaussian noise Receive signal Represented as:

[0034] ;

[0035] in This represents the physical signal received by neighbor node j; Represents the channel coding function;

[0036] S33. After receiving the signal, neighboring agent j uses a semantic decoder to recover the semantic features sent by agent i. This process involves end-to-end training to effectively tolerate channel noise and ensure the accurate delivery of semantic intent.

[0037] ;

[0038] in Semantic decoder function Represents the channel decoder function; This represents the network parameters of the semantic decoder.

[0039] Specifically, S4 includes:

[0040] S41. In order to evaluate the value of neighbor information to the current task, agent i will use its own local context. As a query, the received semantic set of all neighbors As key and value:

[0041] ;

[0042] in The query vector represents agent i; Represents the key vector of neighbor j; Represents the value vector of neighbor j; , , These represent the learnable linear projective weight matrices for the query, key, and value, respectively. Let i represent the set of neighboring nodes of agent i;

[0043] S42. Calculate the correlation score between the local state and the semantics of the neighbors, and normalize it using the Softmax function to obtain the weighted fused cooperative features. :

[0044] ;

[0045] This formula automatically prioritizes the most important neighbor information based on the current task requirements; where... This represents the normalized exponential function; Represents the transpose of the key vector; This indicates the dimension of the attention head, used to scale the dot product result to prevent gradient vanishing.

[0046] Specifically, S5 includes:

[0047] S51. In this process, residual connections are introduced to prevent gradient vanishing, and layer normalization is used to ensure training stability, ultimately generating high-level decision features. :

[0048] ;

[0049] in Presentation layer normalization operation; This represents a feedforward neural network that includes a non-linear activation function;

[0050] S52, Utilizing Multi-Agent Policy Networks based on Generate specific physical action distributions and perform the action:

[0051] ;

[0052] in This represents the specific physical action taken by agent i at time t; Represents the weight parameters of the policy network;

[0053] S53. After the action is performed, calculate the team collaboration reward based on environmental feedback. The reward is used for backpropagation to update network parameters, prompting the agent to learn the optimal strategy that maximizes long-term collective benefits.

[0054] ;

[0055] in Representation Strategy The expected cumulative reward objective function is as follows; T represents the mathematical expectation operation; T represents the time step range for prediction or optimization. Indicates the time discount factor; The reward function representing environmental feedback; This represents the environmental state at time t+k; This represents the joint action of all agents except agent i.

[0056] The beneficial effects of this invention are:

[0057] This invention is based on a distributed sensing architecture with multi-agent interaction, deeply integrating large AI models, semantic communication, and multi-agent system technologies to achieve deep multimodal data collaboration in a decentralized environment. Specifically, firstly, distributed edge agents are used for local perception and context construction. Semantic communication technology is used to extract and transmit high-level semantic features oriented towards interactive intentions, effectively avoiding redundant interaction of raw data and significantly reducing the communication overhead of the edge network. Secondly, the masked multi-head self-attention mechanism of large AI models is innovatively applied to the edge side, enabling agents to automatically perform weighted fusion and dependency capture of semantic information from neighboring nodes. This solves the problem of difficult information weight allocation in distributed collaboration, significantly improving perception accuracy (approximately 80%) and system robustness even without a central node. Finally, a distributed decision-making mechanism based on multi-agent collaboration is constructed, supporting edge nodes to perform autonomous reasoning and real-time response, making it particularly suitable for complex dynamic scenarios with high requirements for low latency and high autonomy, such as industrial inspection. Attached Figure Description

[0058] Figure 1 This is a flowchart of the present invention;

[0059] Figure 2 This is a system model diagram of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0062] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0063] In the description of this invention, it should be understood that the terms "upper," "lower," "inner," "outer," "left," "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used to facilitate the description of this invention and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0064] Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0065] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, terms such as "set" and "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0066] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0067] like Figure 1 and 2 As shown, a design method for a distributed multimodal ISAC system includes the following steps:

[0068] S1, Multimodal Local Awareness and Feature Embedding; Step S1 describes the process of mapping heterogeneous multimodal raw data into a unified-dimensional token sequence through an embedding layer and superimposing positional encoding.

[0069] S1 specifically includes:

[0070] S11. The i-th edge agent collects environmental data through locally mounted sensors; let the visual image data collected at time t be... Radio frequency signal data is ;

[0071] S12. To overcome the semantic gap and heterogeneity between visual data (two-dimensional pixel matrix) and radio frequency data (one-dimensional time series), the embedding layer of a large AI model is used to map data from different modalities to the same high-dimensional feature space. This projects the discrete raw data into the same continuous high-dimensional feature subspace, transforming it into a unified token sequence. Furthermore, given the insensitivity of the subsequent Transformer architecture to sequence order, to accurately preserve the key temporal order and spatial structure information implicit in the perceptual data, a learnable positional encoding vector needs to be superimposed on the feature sequence. The resulting local initial token sequence The calculation is as follows:

[0072] ;

[0073] in Represents the concatenation operation of vectors or matrices; This represents a linear projection embedding function for visual image data; This represents the raw visual image data acquired at time t; This represents a linear projection embedding function for radio frequency signal data; This represents the raw radio frequency signal data collected at time t.

[0074] S2. Context construction based on memory enhancement; Step S2 describes the process by which the agent updates the memory module in real time by combining the current environmental observations and retrieves relevant domain knowledge from the knowledge base to construct a complete context for reasoning.

[0075] S2 specifically includes:

[0076] S21. In order for the agent to understand the dynamic evolution of the environment (such as the trajectory of the target and historical interaction intentions), the agent maintains a memory state. This mechanism is used to store historical interaction information. At time t, to effectively capture time dependencies and avoid the gradient vanishing problem, a gated recurrent unit (GRU) is used as the update function. This mechanism can selectively forget irrelevant historical noise and retain key state information, combined with the current perceived input. Update memory:

[0077] ;

[0078] in This represents the memory state of the i-th agent after the update at time t; This represents the update function of the gated loop unit; It represents the memory state of the previous time step t-1, and the adaptive fusion of historical information is achieved through a gating mechanism;

[0079] S22. Purely data-driven reasoning often lacks an understanding of the rules specific to the scenario. To compensate for this deficiency, the agent, based on the current observation state, draws from its local knowledge base... The system retrieves prior knowledge relevant to the current scenario (such as communication protocols and traffic rules). Then, using prompting engineering techniques, it splices and structurally integrates the current perceived features, updated historical memories, and retrieved domain knowledge to form a reasoning context containing complete spatiotemporal logic and rule constraints. :

[0080] ;

[0081] in This indicates a prompt for engineering construction operations or functions; the context includes not only "what was seen" (perception), but also "what happened in the past" (memory) and "what rules should be followed" (knowledge), providing panoramic information support for subsequent accurate decision-making.

[0082] S3. Intent-Oriented Semantic Encoding and Interactive Transmission; Step S3 describes the process of extracting core intent-oriented features from the context using a semantic encoder, mapping them to semantic symbols, transmitting them through a physical channel, and then performing semantic decoding and recovery at the receiving end. Step S3 specifically includes:

[0083] S31. When agent i needs to interact with neighboring agent j, the semantic encoding process is initiated; using a semantic encoder constructed from a deep neural network, the semantic encoding is derived from the reasoning context. The key intent features are mapped to a sequence of semantic symbols in the complex domain. :

[0084] ;

[0085] in This represents a semantic encoder function; This represents the network parameters of the semantic encoder; Let K represent the complex feature space, where K is the number of subcarriers. The length of the semantic symbol;

[0086] S32. Semantic symbols are transmitted to neighbor node j via a wireless channel; considering the unavoidable physical constraints in actual communication environments, the signal will experience multipath fading during transmission. Additive white Gaussian noise Receive signal Represented as:

[0087] ;

[0088] in This represents the physical signal received by neighbor node j; Represents the channel coding function;

[0089] S33. After receiving the signal, neighboring agent j uses a semantic decoder to recover the semantic features sent by agent i. This process involves end-to-end training to effectively tolerate channel noise and ensure the accurate delivery of semantic intent.

[0090] ;

[0091] in Semantic decoder function Represents the channel decoder function; This represents the network parameters of the semantic decoder.

[0092] S4. Neighbor information fusion based on attention mechanism; Step S4 describes the process of dependency capture and weighted fusion of cross-node neighbor semantic information using a masked multi-head self-attention mechanism; Step S4 specifically includes:

[0093] S41. In order to evaluate the value of neighbor information to the current task, agent i will use its own local context. As a query, the received semantic set of all neighbors As key and value:

[0094] ;

[0095] in The query vector represents agent i; Represents the key vector of neighbor j; Represents the value vector of neighbor j; , , These represent the learnable linear projective weight matrices for the query, key, and value, respectively. Let i represent the set of neighboring nodes of agent i;

[0096] S42. Calculate the correlation score between the local state and the semantics of the neighbors, and normalize it using the Softmax function to obtain the weighted fused cooperative features. :

[0097] ;

[0098] This formula automatically prioritizes the most important neighbor information based on the current task requirements (e.g., paying more attention to neighbors with good visibility when there is obstruction); where This represents the normalized exponential function; Represents the transpose of the key vector; This indicates the dimension of the attention head, used to scale the dot product result to prevent gradient vanishing.

[0099] S5, Distributed Collaborative Decision Making and Execution; Step S5 describes the process of performing a deep transformation on the fused features through a feedforward network and residual normalization to generate a high-level feature representation for the final collaborative decision making.

[0100] Specifically,

[0101] Step 5 specifically includes:

[0102] S51. In this process, residual connections are introduced to prevent gradient vanishing, and layer normalization is used to ensure training stability, ultimately generating high-level decision features. :

[0103] ;

[0104] in Presentation layer normalization operation; This represents a feedforward neural network that includes a non-linear activation function;

[0105] S52, Utilizing Multi-Agent Policy Networks based on Generate specific physical action distributions (such as beamforming vectors, collaborative sensing commands), and execute actions:

[0106] ;

[0107] in This represents the specific physical action taken by agent i at time t; Represents the weight parameters of the policy network;

[0108] S53. To improve the collaboration efficiency of multi-agent teams, the system establishes a feedback mechanism based on reinforcement learning. After an action is executed, the team collaboration reward is calculated based on environmental feedback (such as the improvement in perception accuracy and communication rate). The reward is used for backpropagation to update network parameters, prompting the agent to learn the optimal strategy that maximizes long-term collective benefits.

[0109] ;

[0110] in Representation Strategy The expected cumulative reward objective function is as follows; T represents the mathematical expectation operation; T represents the time step range for prediction or optimization. Indicates the time discount factor; The reward function representing environmental feedback; This represents the environmental state at time t+k; This represents the joint action of all agents except agent i.

[0111] This application adopts a decentralized distributed architecture, utilizing agents distributed at the edge for multimodal perception and collaborative reasoning. By deeply integrating large AI models, semantic communication and multi-agent system technologies, it sequentially executes five core steps: multimodal local perception, memory-based context construction, intent-oriented semantic interaction, attention-based information fusion and distributed collaborative decision-making, thereby achieving efficient collaborative perception without a central node.

[0112] In the multimodal local perception step of this application, the edge agent collects heterogeneous sensing data such as images and radio frequency signals; then, using the embedding layer technology of the large AI model, the raw data of different modalities are mapped into a token sequence of a unified dimension, and position encoding is superimposed to generate a local initial perception state containing spatiotemporal information.

[0113] In the memory-based context construction step of this application, the agent maintains a memory module that is updated over time to store historical interaction experiences; at the same time, it combines real-time observation data to retrieve relevant domain knowledge from the local knowledge base, thereby constructing a complete reasoning context that includes historical memory and prior knowledge.

[0114] In the semantic interaction step oriented towards intent, this application employs semantic communication technology, using a semantic encoder to extract core semantic features oriented towards collaborative intent from the reasoning context; these features are mapped into compact semantic symbols and sent to neighboring agents via a wireless channel; the receiving end recovers the semantic information of the neighboring agent through a semantic decoder after receiving the signal.

[0115] In the attention-based information fusion step, this application utilizes the masked multi-head self-attention mechanism of a large AI model as the fusion engine; the agent uses its own local context as the query vector and all received neighbor semantic information as the key and value vectors, automatically calculates the cross-correlation weights and performs weighted aggregation, thereby achieving deep feature fusion across nodes.

[0116] In the distributed collaborative decision-making step, this application inputs the fused features into a feedforward neural network and a layer normalization module for deep transformation to extract high-level decision features; then, a multi-agent policy network is used to generate specific physical actions or collaborative instructions based on these features and control the hardware devices to execute them.

[0117] This application deeply integrates multimodal information such as radio frequency signals, visual images, and LiDAR point clouds, fully leveraging the complementary advantages of different sensors in spatial resolution, anti-interference capabilities, and semantic representation to achieve more accurate and robust environmental perception. Compared with traditional single-modal ISAC systems, this multimodal fusion strategy significantly improves perception accuracy (experiments show an improvement of approximately 80%) and the system's adaptability in dynamic environments, making it particularly suitable for complex scenarios such as multipath fading, occlusion, or severe weather. Furthermore, to further improve the system's collaborative efficiency and generalization capabilities across different application scenarios, this method introduces three key technologies: large AI models, semantic communication, and multi-agent systems, constructing an I-MAC (interaction-based) architectural paradigm, thereby significantly reducing communication overhead and enhancing decision intelligence.

[0118] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A design method for a distributed multimodal ISAC system, characterized in that, Including the following steps: S1, Multimodal local perception and feature embedding; S2, Context building based on memory enhancement; S3, Intent-oriented semantic encoding and interactive transmission; S4. Neighbor information fusion based on attention mechanism; S5, Distributed Collaborative Decision Making and Execution.

2. The design method for a distributed multimodal ISAC system according to claim 1, characterized in that, S1 specifically includes: S11. The i-th edge agent collects environmental data through locally mounted sensors; let the visual image data collected at time t be... Radio frequency signal data is ; S12. Utilizing the embedding layer of a large AI model, data from different modalities are mapped to the same high-dimensional feature space, and discrete raw data are projected onto the same continuous high-dimensional feature subspace, transforming them into a unified token sequence. Furthermore, learnable positional encoding vectors are superimposed on the feature sequence. The resulting local initial token sequence The calculation is as follows: ； in Represents the concatenation operation of vectors or matrices; This represents a linear projection embedding function for visual image data; This represents the raw visual image data acquired at time t; This represents a linear projection embedding function for radio frequency signal data; This represents the raw radio frequency signal data collected at time t.

3. The design method for a distributed multimodal ISAC system according to claim 2, characterized in that, S2 specifically includes: S21. The agent maintains a memory state. It is used to store historical interaction information; at time t, a gated loop unit is used as the update function; combined with the current sensing input. Update memory: ； in This represents the memory state of the i-th agent after the update at time t; This represents the update function of the gated loop unit; This represents the memory state at the previous time step t-1; S22. The agent, based on the current observation state, retrieves information from its local knowledge base. The system retrieves prior knowledge relevant to the current scene. Then, using prompting engineering techniques, it combines and structures the current perceptual features, updated historical memories, and retrieved domain knowledge to form a reasoning context containing complete spatiotemporal logic and rule constraints. : ； in This indicates a prompt for a project build operation or function.

4. The design method for a distributed multimodal ISAC system according to claim 3, characterized in that, S3 specifically includes: S31. When agent i needs to interact with neighboring agent j, the semantic encoding process is initiated; using a semantic encoder constructed from a deep neural network, the semantic encoding is derived from the reasoning context. The key intent features are mapped to a sequence of semantic symbols in the complex domain. : ； in This represents a semantic encoder function; This represents the network parameters of the semantic encoder; Let K represent the complex feature space, where K is the number of subcarriers. The length of the semantic symbol; S32. Semantic symbols are transmitted to neighbor node j via a wireless channel; considering the unavoidable physical constraints in actual communication environments, the signal will experience multipath fading during transmission. Additive white Gaussian noise Receive signal Represented as: ； in This represents the physical signal received by neighbor node j; Represents the channel coding function; S33. After receiving the signal, neighboring agent j uses a semantic decoder to recover the semantic features sent by agent i. This process involves end-to-end training to effectively tolerate channel noise and ensure the accurate delivery of semantic intent. ； in Represents semantic decoder function Represents the channel decoder function; This represents the network parameters of the semantic decoder.

5. The design method for a distributed multimodal ISAC system according to claim 4, characterized in that, S4 specifically includes: S41. In order to evaluate the value of neighbor information to the current task, agent i will use its own local context. As a query, the received semantic set of all neighbors As key and value: ； in The query vector represents agent i; Represents the key vector of neighbor j; Represents the value vector of neighbor j; , , These represent the learnable linear projective weight matrices for the query, key, and value, respectively. Let i represent the set of neighboring nodes of agent i; S42. Calculate the correlation score between the local state and the semantics of the neighbors, and normalize it using the Softmax function to obtain the weighted fused cooperative features. : ； This formula automatically prioritizes the most important neighbor information based on the current task requirements; where... This represents the normalized exponential function; This represents the transpose of the key vector; This indicates the dimension of the attention head, used to scale the dot product result to prevent gradient vanishing.

6. The design method for a distributed multimodal ISAC system according to claim 5, characterized in that, S5 specifically includes: S51. In this process, residual connections are introduced to prevent gradient vanishing, and layer normalization is used to ensure training stability, ultimately generating high-level decision features. : ； in Presentation layer normalization operation; This represents a feedforward neural network that includes a non-linear activation function; S52, Utilizing Multi-Agent Policy Networks based on Generate specific physical action distributions and perform the action: ； in This represents the specific physical action taken by agent i at time t; Represents the weight parameters of the policy network; S53. After the action is performed, calculate the team collaboration reward based on environmental feedback. The reward is used for backpropagation to update network parameters, prompting the agent to learn the optimal strategy that maximizes long-term collective benefits. ； in Representation Strategy The expected cumulative reward objective function is as follows; T represents the mathematical expectation operation; T represents the time step range for prediction or optimization. Indicates the time discount factor; The reward function representing environmental feedback; This represents the environmental state at time t+k; This represents the joint action of all agents except agent i.

Images