A time-delay-aware multi-modal wireless infrastructure model system and collaborative inference method

Abstract

The application discloses a kind of multi-modal wireless infrastructure model systems and collaborative inference method based on delay perception, the system includes: multi-flow input and Tokenization module is used to receive multi-modal data and convert multi-modal data into Token sequence;Multi-encoder network is used to extract self-modal feature from Token sequence, and generate cross-modal feature from self-modal feature;Dynamic feature gate network is used to obtain the weight of each modal feature by score layer calculation;Delay perception routing switch is used to perform double-track shunt control according to the physical layer QoS delay index of current task;Microscopic collaborative network is used to generate multi-source local feedback data by multi-modal agent, and reconcile cross-modal cognitive conflict using thought chain inference engine.The application meets the feature explainability requirement in wireless deep learning network, solves the intrinsic conflict between LLM inference delay and physical layer hard real-time requirement, and improves the model robustness.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a delay-aware multimodal wireless basic model system and a collaborative reasoning method. Background Technology

[0002] As sixth-generation mobile communication technology (6G) evolves towards AI-native and integrated sensing and communication (ISAC) networks, the network infrastructure needs to simultaneously process heterogeneous and highly asymmetric multimodal data. This data includes high-dimensional and abstract channel state information (CSI), 3D LiDAR (Light Detection and Ranging) point clouds, and visual RGB images.

[0003] Existing wireless deep learning networks exhibit three major bottlenecks when processing multi-source heterogeneous data: First, traditional multimodal networks are mostly monolithic black-box structures, making it impossible for operators to quantitatively audit the contribution of each network depth and modal feature to the final control decision, thus lacking design interpretability. Second, if existing technologies blindly introduce heavyweight large language models (LLMs) for advanced perceptual reasoning, the inherent latency (typically in the hundreds of milliseconds) caused by token generation and deep Transformer computation will be significantly higher than the millisecond-level hard real-time QoS (Quality of Request) required for tasks such as physical layer milli-beamforming and active precoding. The inherent conflict between service (quality of service) requirements and the inherent conflict between them is irreconcilable. Furthermore, in joint training, because the physical characteristics of wireless channels (such as CSI features) are inherently weaker than visual signals in terms of spatial connectivity, their physical properties such as multipath fading are easily "masked" or "washed away" by the strong visual features in the latent space, resulting in poor robustness of the model in visually obstructed scenarios such as severe weather. In other words, modal features with lower information density are easily masked by modal features with higher information density, resulting in insufficient robustness of the model.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a latency-aware multimodal wireless basic model system and a collaborative inference method, aiming to solve the problems of lack of interpretability, conflict between LLM inference latency and physical layer hard real-time requirements, and insufficient robustness in existing wireless deep learning networks.

[0006] To achieve the above-mentioned objectives, the present invention provides a time-delay-aware multimodal wireless fundamental model system, the time-delay-aware multimodal wireless fundamental model system comprising: The multi-stream input and tokenization module is used to receive multimodal data and convert the multimodal data into a token sequence through data segmentation and location embedding operators; A multi-encoder network is used to extract self-modal features from the token sequence using a self-attention mechanism, and to generate cross-modal features from the self-modal features using an asymmetric cross-modal injection mechanism. A dynamic feature gating network is used to calculate the weights of the self-modal features and the weights of the cross-modal features through a scoring layer; The latency-aware routing switch is used to perform dual-track traffic splitting control based on the physical layer QoS latency index of the current task. When the current task is a hard real-time communication task, the fast inference path is activated, and when the current task is a latency-tolerant awareness task, the cognitive inference path is activated. A micro-collaborative network is used in the cognitive reasoning path to generate multi-source local feedback data for the perception task through a multimodal agent based on the self-modal features, the cross-modal features, the weights of the self-modal features and the weights of the cross-modal features. It also uses the thought chain reasoning engine of a large language model to reconcile cross-modal cognitive conflicts and output the processing results and semantic explanation text.

[0007] Optionally, the multimodal data includes channel state data, point cloud data, and visual data; the token sequence includes a channel token sequence, a point cloud token sequence, and a visual token sequence; and the multi-stream input and tokenization module includes: A multi-stream input unit is used to receive channel state data, point cloud data, and visual data; The tokenization unit is used to convert the channel state data, the point cloud data, and the visual data into a unified format channel token sequence, point cloud token sequence, and visual token sequence through data segmentation and location embedding operators, respectively.

[0008] Optionally, the self-modal features include channel features, point cloud features, and visual features; the cross-modal features include channel cross-modal features, point cloud cross-modal features, and visual cross-modal features; and the multi-encoder network includes: A CSI encoder is used to extract channel features from the channel token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the CSI encoder. In the deep Transformer module of the CSI encoder, the channel features are used as query vectors, and the concatenation features of the point cloud features and the visual features are used as key-value pairs to generate channel cross-modal features using an asymmetric cross-modal injection mechanism. A point cloud encoder is used to extract point cloud features from the point cloud token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the point cloud encoder. In the deep Transformer module of the point cloud encoder, the point cloud features are used as query vectors, and the concatenation features of the channel features and the visual features are used as key-value pairs to generate point cloud cross-modal features using an asymmetric cross-modal injection mechanism. A visual encoder is used to extract visual features from the visual token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the visual encoder. In the deep Transformer module of the visual encoder, the visual features are used as query vectors, and the concatenation features of the channel features and the point cloud features are used as key-value pairs to generate visual cross-modal features using an asymmetric cross-modal injection mechanism. The channel features, point cloud features, visual features, channel cross-modal features, point cloud cross-modal features, and visual cross-modal features constitute a multimodal initial joint feature vector.

[0009] Optionally, the time-delay-aware multimodal wireless fundamental model system further includes: A fast inference head is used in the fast inference path to linearly combine the channel features and the channel cross-modal features according to the weights of the channel features and the weights of the channel cross-modal features to obtain channel combination features, and to perform fast inference on the communication task according to the channel combination features to obtain a fast inference result for the communication task.

[0010] Optionally, the large language model includes a central LLM orchestrator, and the micro-cooperative network includes: An equal-weighted multimodal latent space is used to map the initial joint feature vector of the multimodal modes to the equal-weighted multimodal latent space constructed based on contrastive learning constraints, so as to obtain the implicit feature representations aligned to each modality. The Top-k purification module is used to calculate the similarity between the prompt words of the perception task and the implicit representation of the features in the equal multimodal latent space through the Top-k feature selection operator, and sort the implicit representations of the features in descending order of similarity, and take the top k implicit representations of the features as the core spatiotemporal feature implicit representations. A centralized data warehouse is used to receive and store the implicit representation of the core spatiotemporal features, the initial joint feature vector of the multimodal and the upper-layer integrated computing action instructions corresponding to the perception task, and to aggregate the initial joint feature vector of the multimodal and the upper-layer integrated computing action instructions in the current time slot to obtain the current global task state. A multimodal intelligent agent is used to read and parse the current global task state, obtain the target orientation and region of interest for local feature purification, call the local dedicated toolchain based on the implicit representation of the core spatiotemporal features, and use the local dedicated toolchain to perform local feature purification on the token sequence based on the target orientation and the region of interest, generate multi-source local feedback data for the perception task, and asynchronously write the multi-source local feedback data back to the centralized data warehouse. The central LLM orchestrator is used to reconcile the cross-modal cognitive conflict when it is determined that a cross-modal cognitive conflict has occurred, based on the weights of the initial joint feature vector of the multimodal system and the multi-source local feedback data. It outputs the processing results and semantic explanation text, and issues control update instructions to the multimodal agent to form a dynamic control closed loop.

[0011] Optionally, the local dedicated toolchain includes a channel local dedicated toolchain, a point cloud local dedicated toolchain, and a vision local dedicated toolchain; the multi-source local feedback data includes channel local feedback data, point cloud local feedback data, and vision local feedback data; and the multimodal intelligent agent includes: The channel agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the channel mode, it calls the channel-specific toolchain. Based on the target orientation and the region of interest, it uses Fast Fourier Transform and state filtering to purify the channel fine feature matrix from the channel token sequence. It also calculates the channel self-test probability score based on the signal-to-noise ratio of the received signal. The channel fine feature matrix and the channel self-test probability score are used as channel local feedback data, and the channel local feedback data is asynchronously written back to the centralized data warehouse. A point cloud agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the point cloud modality, the agent calls the point cloud local dedicated toolchain. Based on the target orientation and the region of interest, the agent uses a density-based topology segmentation algorithm to extract the fine feature matrix of the point cloud from the point cloud token sequence. The agent calculates the point cloud self-verification probability score based on the effective dynamic target point cloud density and noise ratio. The agent uses the fine feature matrix of the point cloud and the point cloud self-verification probability score as the point cloud local feedback data and asynchronously writes the point cloud local feedback data back to the centralized data warehouse. A visual intelligent agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the visual modality, the agent invokes the visual local dedicated toolchain. Based on the target orientation and the region of interest, the agent uses a convolutional feature map edge extraction algorithm to extract a visual fine feature matrix from the visual token sequence and calculates a visual self-verification probability score based on image contrast. The visual fine feature matrix and the visual self-verification probability score are used as visual local feedback data, and the visual local feedback data is asynchronously written back to the centralized data warehouse.

[0012] Optionally, the central LLM orchestrator includes: A data acquisition unit is used to acquire the multi-source local feedback data written back by the multimodal agent from the centralized data warehouse; The conflict determination unit is used to determine that a cross-modal cognitive conflict has occurred when, based on multi-source local feedback data and with the weights of the multimodal initial joint feature vector as the physical prior, a nonlinear deviation in the motion trajectory prediction of different modalities is detected and the channel self-verification probability score, the point cloud self-verification probability score, or the visual self-verification probability score is lower than the compliance threshold. The thought chain reasoning engine is used to reconcile the cross-modal cognitive conflict, output the processing results and semantic explanation text, and simultaneously issue control update instructions to the multimodal agent. The sampling rate or filter parameters of the local dedicated toolchain are dynamically adjusted through the control update instructions to form a dynamic control closed loop.

[0013] Optionally, the centralized data warehouse is deployed on a cloud or edge server as a shared blackboard.

[0014] Optionally, the scoring layer includes a learnable, fully connected, lightweight deep neural network.

[0015] To achieve the above-mentioned objectives, this invention also provides a collaborative reasoning method for a time-delay-aware multimodal wireless fundamental model system, wherein the collaborative reasoning method for a time-delay-aware multimodal wireless fundamental model system includes: The multi-stream input and tokenization module receives multimodal data and converts the multimodal data into a token sequence through data segmentation and position embedding operators; The multi-encoder network uses a self-attention mechanism to extract self-modal features from the token sequence and uses an asymmetric cross-modal injection mechanism to generate cross-modal features from the self-modal features; The dynamic feature gating network calculates the weights of the self-modal features and the weights of the cross-modal features through a scoring layer; The latency-aware routing switch performs dual-track traffic splitting control based on the physical layer QoS latency index of the current task. When the current task is a hard real-time communication task, the fast inference path is activated, and when the current task is a latency-tolerant awareness task, the cognitive inference path is activated. The micro-collaborative network, in the cognitive reasoning path, generates multi-source local feedback data for the perception task through a multimodal agent based on the self-modal features, the cross-modal features, the weights of the self-modal features and the weights of the cross-modal features. It also uses the thought chain reasoning engine of the large language model to reconcile cross-modal cognitive conflicts and outputs the processing results and semantic explanation text.

[0016] In this invention, a multi-stream input and tokenization module receives multimodal data and converts it into a token sequence using data segmentation and location embedding operators. A multi-encoder network extracts self-modal features from the token sequence using a self-attention mechanism and generates cross-modal features from the self-modal features using an asymmetric cross-modal injection mechanism. A dynamic feature gating network calculates the weights of the self-modal features and the cross-modal features through a scoring layer. A latency-aware routing switch performs dual-track traffic control based on the physical layer QoS latency index of the current task. When the current task is a hard real-time communication task, a fast inference path is activated; when the current task is a latency-tolerant perception task, a cognitive inference path is activated. A micro-collaborative network generates multi-source local feedback data for the perception task through a multimodal agent in the cognitive inference path, based on the self-modal features, the cross-modal features, the weights of the self-modal features, and the weights of the cross-modal features. It then uses the thought chain inference engine of a large language model to reconcile cross-modal cognitive conflicts and outputs the processing results and semantic explanation text. This invention calculates scalar weights through a dynamic feature-gated network, providing quantitative audit metrics for hierarchical and modal contributions, thus meeting the feature interpretability requirements in wireless deep learning networks. It resolves the intrinsic conflict between LLM inference latency and the physical layer's hard real-time requirements through a latency-aware routing switch, and improves model robustness by reconciling cross-modal cognitive conflicts through a thought chain inference engine. Attached Figure Description

[0017] Figure 1 This is a structural diagram of a preferred embodiment of the delay-aware multimodal wireless basic model system of the present invention; Figure 2 This is a flowchart of the micro-collaborative network of the present invention; Figure 3 This is a flowchart of a preferred embodiment of the collaborative reasoning method for a time-delay-aware multimodal wireless basic model system according to the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] As sixth-generation mobile communication technology (6G) evolves towards AI-native and communication-aware integrated (ISAC) networks, network infrastructure needs to simultaneously process heterogeneous and highly asymmetric multimodal data. This data includes high-dimensional and abstract channel state information (CSI), 3D LiDAR point clouds, and visual RGB images.

[0020] Existing wireless deep learning networks exhibit three major bottlenecks when processing multi-source heterogeneous data: First, traditional multimodal networks are mostly integrated black-box structures, making it impossible for operators to quantitatively audit the contribution of each network depth and modal feature to the final control decision, thus lacking design interpretability. Second, if existing technologies blindly introduce heavy large language models (LLM) for advanced perceptual inference, the inherent latency (typically in the hundreds of milliseconds) caused by token generation and deep Transformer computation will create an irreconcilable intrinsic conflict with the millisecond-level hard real-time QoS requirements of tasks such as physical layer milli-beamforming and active precoding. Third, in joint training, because the physical features of wireless channels (such as CSI features) are inherently weaker than visual signals in terms of spatial connectivity, their physical properties such as multipath fading are easily "masked" or "washed away" by strong visual features in the latent space, resulting in poor robustness of the model in visually obstructed scenarios such as severe weather. In other words, modal features with lower information density are easily masked by modal features with higher information density, leading to insufficient model robustness.

[0021] To address the aforementioned technical problems, this invention provides a latency-aware multimodal wireless basic model system. A multi-stream input and tokenization module receives multimodal data and converts it into a token sequence through data segmentation and location embedding operators. A multi-encoder network extracts self-modal features from the token sequence using a self-attention mechanism and generates cross-modal features from the self-modal features using an asymmetric cross-modal injection mechanism. A dynamic feature gating network calculates the weights of the self-modal features and the cross-modal features through a scoring layer. A latency-aware routing switch is used... The invention employs a dual-track traffic splitting control based on the physical layer QoS latency metrics of the current task. When the current task is a hard real-time communication task, a fast inference path is activated; when the current task is a latency-tolerant perception task, a cognitive inference path is activated. A micro-collaborative network is used in the cognitive inference path to generate multi-source local feedback data for the perception task through a multimodal agent, based on the self-modal features, the cross-modal features, the weights of the self-modal features, and the weights of the cross-modal features. Furthermore, it utilizes a large language model's thought chain inference engine to reconcile cross-modal cognitive conflicts and outputs the processing results and semantically interpreted text. This invention calculates scalar weights through a dynamic feature gating network, providing quantitative audit metrics for hierarchical and modal contributions, meeting the feature interpretability requirements in wireless deep learning networks. It resolves the intrinsic conflict between LLM inference latency and the physical layer's hard real-time requirements through a latency-aware routing switch, and improves model robustness by reconciling cross-modal cognitive conflicts through the thought chain inference engine.

[0022] The application content will be further explained below with reference to the accompanying drawings and through the description of the embodiments.

[0023] A preferred embodiment of the time-delay-aware multimodal wireless fundamental model system of the present invention is as follows: Figure 1 As shown, it specifically includes: The multi-stream input and tokenization module is used to receive multimodal data and convert the multimodal data into a token sequence through data segmentation and location embedding operators; A multi-encoder network is used to extract self-modal features from the token sequence using a self-attention mechanism, and to generate cross-modal features from the self-modal features using an asymmetric cross-modal injection mechanism. A dynamic feature gating network is used to calculate the weights of the self-modal features and the weights of the cross-modal features through a scoring layer; The latency-aware routing switch is used to perform dual-track traffic splitting control based on the physical layer QoS latency index of the current task. When the current task is a hard real-time communication task, the fast inference path is activated, and when the current task is a latency-tolerant awareness task, the cognitive inference path is activated. A micro-collaborative network is used in the cognitive reasoning path to generate multi-source local feedback data for the perception task through a multimodal agent based on the self-modal features, the cross-modal features, the weights of the self-modal features and the weights of the cross-modal features. It also uses the thought chain reasoning engine of a large language model to reconcile cross-modal cognitive conflicts and output the processing results and semantic explanation text.

[0024] Specifically, this invention constructs a two-layer technical solution consisting of macroscopic "multi-stream coding and delay-based routing" and microscopic "equal latent space and agent blackboard collaboration": At the macroscopic level, a multi-encoder network is designed and asymmetric cross-modal injection is performed, with a dynamic feature gating network assigning context-adaptive weights to features of different depths; through a delay-aware routing switch, CSI features are directly intercepted to perform real-time beamforming to bypass LLM delays. At the microscopic level, an equal latent space and Top-k mechanism based on contrastive learning are introduced, working in conjunction with local expert agents (channel, vision, point cloud, etc.) to iteratively write back feedback under a shared blackboard protocol, and a central LLM resolves multimodal cognitive conflicts such as "visual occlusion and electromagnetic direct vision" through a Chain of Thought (CoT). This invention provides a 6G edge base station or network element system that balances physical hard control with all-weather robust perception. It calculates scalar weights through a dynamic feature-gated network, providing quantitative audit indicators of hierarchical and modal contributions, thus meeting the feature interpretability requirements in wireless deep learning networks. It resolves the intrinsic conflict between LLM inference latency and the physical layer's hard real-time requirements through a latency-aware routing switch, and improves model robustness by reconciling cross-modal cognitive conflicts through a thought chain inference engine.

[0025] In one implementation of this embodiment, the multimodal data includes channel state data, point cloud data, and visual data; the token sequence includes a channel token sequence, a point cloud token sequence, and a visual token sequence; and the multi-stream input and tokenization module includes: A multi-stream input unit is used to receive channel state data, point cloud data, and visual data; The tokenization unit is used to convert the channel state data, the point cloud data, and the visual data into a unified format channel token sequence, point cloud token sequence, and visual token sequence through data segmentation and location embedding operators, respectively.

[0026] Specifically, the system first receives high-dimensional heterogeneous Channel State Information (CSI) streams (i.e., channel state data), 3D point cloud streams (i.e., point cloud data), and visual RGB image streams (i.e., visual data) through a multi-stream input and tokenization module. This module converts these raw physical data into a unified format token sequence through built-in data patching and positional embedding operators.

[0027] In one implementation of this embodiment, the self-modal features include channel features, point cloud features, and visual features; the cross-modal features include channel cross-modal features, point cloud cross-modal features, and visual cross-modal features; and the multi-encoder network includes: A CSI encoder is used to extract channel features from the channel token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the CSI encoder. In the deep Transformer module of the CSI encoder, the channel features are used as query vectors, and the concatenation features of the point cloud features and the visual features are used as key-value pairs to generate channel cross-modal features using an asymmetric cross-modal injection mechanism. A point cloud encoder is used to extract point cloud features from the point cloud token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the point cloud encoder. In the deep Transformer module of the point cloud encoder, the point cloud features are used as query vectors, and the concatenation features of the channel features and the visual features are used as key-value pairs to generate point cloud cross-modal features using an asymmetric cross-modal injection mechanism. A visual encoder is used to extract visual features from the visual token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the visual encoder. In the deep Transformer module of the visual encoder, the visual features are used as query vectors, and the concatenation features of the channel features and the point cloud features are used as key-value pairs to generate visual cross-modal features using an asymmetric cross-modal injection mechanism. The channel features, point cloud features, visual features, channel cross-modal features, point cloud cross-modal features, and visual cross-modal features constitute a multimodal initial joint feature vector.

[0028] Specifically, the token sequence is then delivered to a multi-encoder network. For example... Figure 1 As shown, the multi-encoder network comprises three parallel dedicated channels: a CSI encoder (i.e., ... Figure 1 Wireless encoders, visual encoders, and point cloud encoders (i.e., wireless encoders) Figure 1 (The perceptual encoder in the image).

[0029] Shallow and mid-level network stages: Each encoder uses a pure Transformer block to perform a self-attention mechanism, independently extracting self-modal features of the wireless channel, spatial vision, and 3D geometry in its dedicated channel, effectively achieving initial noise isolation.

[0030] In the deep network stage: the system introduces an asymmetric cross-modal injection mechanism. Utilizing a cross-attention structure, the CSI features (i.e., channel features) extracted from the CSI encoder are used as the query, while the visual context (i.e., visual features) extracted from the visual encoder and the spatial geometric features (i.e., point cloud features) extracted from the point cloud encoder are actively used as the key and value, respectively, and injected into the query of the CSI encoder. The channel features are constrained by the external spatial physical boundary. Simultaneously, the CSI features are injected into the point cloud encoder and the visual encoder in reverse, thereby enabling the spatial perception modality to naturally possess the prior knowledge of electromagnetic physical layer propagation characteristics.

[0031] It should be noted that CSI encoders, visual encoders, and point cloud encoders include different network depths of shallow, medium, and deep layers, respectively. However, since the feature extraction mechanisms of shallow and medium layers are consistent, for ease of description, the shallow network (narrowly defined) and medium layer network are collectively referred to as shallow network (broadly defined) in the preceding text. That is, the self-modal features mentioned above include shallow self-modal features extracted by shallow networks and medium self-modal features extracted by medium networks. The cross-modal features mentioned above are deep cross-modal features extracted by deep networks. The channel features mentioned above include shallow channel features extracted by shallow networks and medium channel features extracted by medium networks. The same applies to point cloud features and visual features.

[0032] In one implementation of this embodiment, a dynamic feature gating network is connected to a multi-encoder network. The dynamic feature gating network extracts high-dimensional embedding vectors (i.e., shallow self-modal features, mid-modal features, and deep cross-modal features) from the outputs of the three encoders at different network depths (Low, Mid, and High, i.e., deep). Subsequently, the dynamic feature gating network calls a scoring layer to calculate the adaptive scalar weights of the high-dimensional embedding vectors for each modality and level in real time, providing a quantitative audit metric for the contribution of each level and modality. This scoring layer is composed of a learnable, fully connected, lightweight deep neural network (MLP, Multilayer Perceptron). Network operators can audit these scalar weights to intuitively understand which network level or modality contributes most to the decision in the current time slot, achieving explicit quantitative auditing and solving the problem of lack of feature interpretability in existing wireless deep learning networks.

[0033] In one implementation of this embodiment, the scoring layer includes a learnable MLP fully connected lightweight deep neural network.

[0034] In one implementation of this embodiment, a latency-aware routing switch is used to perform dual-track traffic splitting control based on the physical layer QoS latency index of the current task: When the task is a hard real-time communication task, the fast inference path is activated, directly calling the low-level, mid-level, or high-level physical features (channel features) output by the CSI encoder, combining them with the scalar weights output by the dynamic feature gating network, and performing physical layer tasks such as instantaneous beamforming or channel estimation, bypassing the large language model; when the task is a latency-tolerant awareness task, the cognitive inference path is activated, routing all features of all modalities and all levels, as well as the scalar weights output by the dynamic feature gating network, to the large language model (central LLM orchestrator), combining the equal latent space aligned by contrastive learning with the shared blackboard protocol to perform multi-agent closed-loop collaborative inference and output semantic interpretation. This invention overcomes the intrinsic conflict between the low latency requirements of the wireless physical layer and the high latency inference of the large model.

[0035] In one implementation of this embodiment, the time-delay-aware multimodal wireless basic model system further includes: A fast inference head is used in the fast inference path to linearly combine the channel features and the channel cross-modal features according to the weights of the channel features and the weights of the channel cross-modal features to obtain channel combination features, and to perform fast inference on the communication task according to the channel combination features to obtain a fast inference result for the communication task.

[0036] Specifically, after the dynamic feature-gated network calculates the scalar weights, the latency-aware routing switch obtains the physical layer QoS latency index of the current integrated computing task and performs dual-track traffic splitting: Hard real-time communication task track (fast inference path): If the current task is determined to be instantaneous beamforming or channel estimation that must be completed within milliseconds, the latency-aware routing switch immediately intercepts the low-level, mid-level, or high-level physical features output by the CSI encoder, linearly combines them using the scalar weights output by the dynamic feature gating network, and directly submits them to the fast inference head to output communication control parameters. This path completely bypasses the downstream heavy-duty large language model, avoiding the computational latency caused by token generation.

[0037] Latency-tolerant perceptual task track (cognitive reasoning path): If the current task is determined to be a latency-tolerant spatial perception task (such as semantic map construction or high-precision digital twin), the latency-tolerant routing switch activates the cognitive reasoning path, packages all modal features / embedding vectors at all levels, as well as the scalar weights output by the dynamic feature gating network, and routes them to the subsequent micro-cooperative network (i.e., the cognitive reasoning head).

[0038] In one implementation of this embodiment, the large language model includes a central LLM orchestrator, and the micro-cooperative network includes: An equal-weighted multimodal latent space is used to map the initial joint feature vector of the multimodal modes to the equal-weighted multimodal latent space constructed based on contrastive learning constraints, so as to obtain the implicit feature representations aligned to each modality. The Top-k purification module is used to calculate the similarity between the prompt words of the perception task and the implicit representation of the features in the equal multimodal latent space through the Top-k feature selection operator, and sort the implicit representations of the features in descending order of similarity, and take the top k implicit representations of the features as the core spatiotemporal feature implicit representations. A centralized data warehouse is used to receive and store the implicit representation of the core spatiotemporal features, the initial joint feature vector of the multimodal and the upper-layer integrated computing action instructions corresponding to the perception task, and to aggregate the initial joint feature vector of the multimodal and the upper-layer integrated computing action instructions in the current time slot to obtain the current global task state. A multimodal intelligent agent is used to read and parse the current global task state, obtain the target orientation and region of interest for local feature purification, call the local dedicated toolchain based on the implicit representation of the core spatiotemporal features, and use the local dedicated toolchain to perform local feature purification on the token sequence based on the target orientation and the region of interest, generate multi-source local feedback data for the perception task, and asynchronously write the multi-source local feedback data back to the centralized data warehouse. The central LLM orchestrator is used to reconcile the cross-modal cognitive conflict when it is determined that a cross-modal cognitive conflict has occurred, based on the weights of the initial joint feature vector of the multimodal system and the multi-source local feedback data. It outputs the processing results and semantic explanation text, and issues control update instructions to the multimodal agent to form a dynamic control closed loop.

[0039] In one implementation of this embodiment, the centralized data warehouse is deployed on a cloud or edge server as a shared blackboard.

[0040] Specifically, once the task enters the cognitive reasoning path, the system first maps multi-layer features to an equal-weighted multimodal latent space constructed by contrastive learning constraints. Utilizing a method where CSI is the primary modality and other modalities are aligned equally with the primary modality through contrastive language-image pre-training (CLIP), the system ensures that electromagnetic features such as multipath fading in wireless channels share equivalent aligned representations with visual and point cloud representations at the semantic layer, preventing physical characteristics from being washed away by visual signals. That is, the equal-weighted multimodal latent space is constructed based on contrastive learning objectives, enabling features corresponding to the CSI stream, visual RGB image stream, and 3D point cloud stream to share equivalent aligned representations within the same semantic space. Next, the Top-k feature selection operator calculates the cosine similarity between the current task prompt token and the implicit representations of features in the equal-weighted multimodal latent space, dynamically retrieving the top k most core spatiotemporal feature implicit representations, discarding static background redundancy, and significantly streamlining the information window delivered to the large language model (i.e., filtering out environmental background redundancy and compressing the context window of the central LLM orchestrator).

[0041] like Figure 2As shown, the simplified and refined core spatiotemporal features are implicitly represented and delivered to a centralized data warehouse, which serves as a shared blackboard. This centralized data warehouse (which can be deployed in the cloud or on an edge server) maintains and stores the current global task state in real time. The current global task state refers to the total control tensor formed by the aggregation of the multimodal initial joint feature vector and the upper-layer integrated computing action instruction token (such as "locking onto the guided target") within the current time slot. It should be noted that the multi-stream input and the sequence of modal tokens transformed by the tokenization module, after deep cross-modal injection and feature mapping by the multi-encoder network, constitute the multimodal initial joint feature vector. The centralized data warehouse receives and stores upper-layer integrated computing action instruction tokens issued from the upper-layer network control plane or business application layer in real time; these upper-layer integrated computing action instructions are related to the current task.

[0042] At this point, the multimodal expert agent cluster (i.e., multimodal agents) and the centralized data warehouse are connected bidirectionally, the publish-subscribe protocol is activated, the current global task status is read concurrently, and the core spatiotemporal feature implicit representations retrieved by the Top-k feature selection operator are combined. Each agent calls its local dedicated toolchain to perform local feature purification, and generates local feedback data containing modal fine feature matrix and modal self-verification probability score (i.e. modal confidence weight) which is asynchronously written back to the shared blackboard (i.e., the centralized data warehouse).

[0043] Subsequently, the central LLM orchestrator, serving as the core brain, is bidirectionally connected to the centralized data warehouse to pull multi-source local feedback data written back by various expert agents from the centralized data warehouse. Using scalar weights as the physical prior, the CoT inference engine of the thinking chain is activated to execute multi-level conflict arbitration logic. When a nonlinear deviation in the motion trajectory prediction of different modalities is detected and the self-verification probability score of a certain modality is lower than the compliance threshold, a cross-modal cognitive conflict is determined to have occurred. The conflict is reconciled through the thinking chain inference engine, and the final integrated perception decision result and readable semantic explanation text are output. At the same time, control update instructions are sent to the multimodal expert agent cluster for reverse balancing, dynamically adjusting the sampling rate or filter parameters of the local dedicated toolchain of each expert agent to form a dynamic control closed loop.

[0044] In one implementation of this embodiment, the local dedicated toolchain includes a channel local dedicated toolchain, a point cloud local dedicated toolchain, and a vision local dedicated toolchain; the multi-source local feedback data includes channel local feedback data, point cloud local feedback data, and vision local feedback data; and the multimodal intelligent agent includes: The channel agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the channel mode, it calls the channel-specific toolchain. Based on the target orientation and the region of interest, it uses Fast Fourier Transform and state filtering to purify the channel fine feature matrix from the channel token sequence. It also calculates the channel self-test probability score based on the signal-to-noise ratio of the received signal. The channel fine feature matrix and the channel self-test probability score are used as channel local feedback data, and the channel local feedback data is asynchronously written back to the centralized data warehouse. A point cloud agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the point cloud modality, the agent calls the point cloud local dedicated toolchain. Based on the target orientation and the region of interest, the agent uses a density-based topology segmentation algorithm to extract the fine feature matrix of the point cloud from the point cloud token sequence. The agent calculates the point cloud self-verification probability score based on the effective dynamic target point cloud density and noise ratio. The agent uses the fine feature matrix of the point cloud and the point cloud self-verification probability score as the point cloud local feedback data and asynchronously writes the point cloud local feedback data back to the centralized data warehouse. A visual intelligent agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the visual modality, the agent invokes the visual local dedicated toolchain. Based on the target orientation and the region of interest, the agent uses a convolutional feature map edge extraction algorithm to extract a visual fine feature matrix from the visual token sequence and calculates a visual self-verification probability score based on image contrast. The visual fine feature matrix and the visual self-verification probability score are used as visual local feedback data, and the visual local feedback data is asynchronously written back to the centralized data warehouse.

[0045] Specifically, the multimodal expert agent cluster activates the publish-subscribe protocol, concurrently reads the current global task state, and combines the implicit representation of core spatiotemporal features retrieved by the Top-k feature selection operator. Each agent then calls its local dedicated toolchain to perform local feature purification, generating local feedback data that is asynchronously written back to the shared blackboard. This includes: Channel agent (i.e.) Figure 2 The wireless intelligent agent in the system calls its local dedicated toolchain, which includes phase cleaning and Doppler extraction operators, and uses fast Fourier transform and state filtering to filter out random multipath interference from the channel token sequence, extract a low-dimensional channel fine feature matrix that reflects the multipath power delay spectrum, and calculates the channel self-verification probability score based on the signal-to-noise ratio of the received signal (referring to the received wireless physical layer signal, which is the physical entity that extracts the channel state information stream).

[0046] The visual agent invokes its local dedicated toolchain, which includes target bounding box detection and contour segmentation operators, to perform convolutional feature map edge extraction on the input visual token sequence, filter out ambient lighting redundancy, extract a visual fine feature matrix containing the motion edges of spatial targets, and calculate a visual self-test probability score based on image contrast. The visual agent analyzes the grayscale dynamic range of the pixels in the current time slot RGB image through the local dedicated toolchain to obtain the image contrast in real time.

[0047] Point Cloud Agent: It calls its local dedicated toolchain containing spatial clustering isolation operators, uses a density-based topology segmentation algorithm to cluster the point cloud token sequence, filters out static reflection noise from the ground or buildings, extracts the fine feature matrix of the point cloud containing the three-dimensional geometric contours of dynamic targets, and calculates the point cloud self-verification probability score based on the effective dynamic target point cloud density and noise ratio after clustering and isolation.

[0048] The three agents combine their generated modal fine feature matrices with modal self-verification probability scores to form local feedback data, which is then sent back to the centralized data warehouse.

[0049] In one implementation of this embodiment, the central LLM orchestrator includes: A data acquisition unit is used to acquire the multi-source local feedback data written back by the multimodal agent from the centralized data warehouse; The conflict determination unit is used to determine that a cross-modal cognitive conflict has occurred when, based on multi-source local feedback data and with the weights of the multimodal initial joint feature vector as the physical prior, a nonlinear deviation in the motion trajectory prediction of different modalities is detected and the channel self-verification probability score, the point cloud self-verification probability score, or the visual self-verification probability score is lower than the compliance threshold. The thought chain reasoning engine is used to reconcile the cross-modal cognitive conflict, output the processing results and semantic explanation text, and simultaneously issue control update instructions to the multimodal agent. The sampling rate or filter parameters of the local dedicated toolchain are dynamically adjusted through the control update instructions to form a dynamic control closed loop.

[0050] Specifically, the central LLM orchestrator, acting as the core brain, is bidirectionally connected to the centralized data warehouse, pulling multi-source local feedback data from it. Using scalar weights as its physical prior, the central LLM orchestrator activates its internal thought chain reasoning engine to execute multi-level conflict arbitration logic. When the system detects that different modalities (such as vision and channel) produce nonlinear deviations in the prediction of the motion trajectory of the same target, and the self-validation probability score of one of the modalities is lower than the set compliance threshold (e.g., in severe weather or strong backlight scenarios, the visual self-validation probability score is lower), the system will detect nonlinear deviations in the prediction of the motion trajectory of the same target. The channel self-validation probability score When a cross-modal cognitive conflict occurs, the central LLM orchestrator determines that the system has encountered a conflict and reconciles the conflict through the MindChain CoT inference engine, outputting the final integrated perception decision result and a readable semantic explanation text. The specific steps for reconciling the conflict through the MindChain CoT inference engine are as follows: Level 1 (Environmental Diagnosis): By comparing multiple self-test probability scores, an explicit conclusion is drawn from the CoT inference chain: the visual self-test probability score of the current time slot is at an extremely low level, indicating that there is visual interference in the external environment, and the confidence drift of the visual fine feature matrix is ​​generated. Level 2 (Prior Alignment and Harmonization): Retrieve the scalar weight prior from the output of the dynamic feature gating network, and find that the wireless channel mode weights are in a high position and the channel self-validation probability score is high. Extremely high and point cloud self-validation probability score When the threshold is exceeded, the CoT inference chain automatically performs logical reconciliation: the wireless channel has non-line-of-sight electromagnetic penetration and is not affected by visual obstruction. It is determined that at this time, the channel fine feature matrix and point cloud fine feature matrix should be fully trusted. Level 3 (Fusion Decision Output): Fusion of synchronized and aligned channel and point cloud features generates highly reliable integrated perception decision results and outputs readable semantic explanation text (e.g., output: Visual modality is blocked, the system has dynamically and comprehensively relied on CSI channel features and point cloud features to lock the guidance path).

[0051] Finally, while outputting results externally, the central LLM orchestrator sends control update instructions to the multimodal expert agent cluster for reverse balancing. The multimodal expert agent cluster asynchronously receives these instructions and executes closed-loop control adjustments: the instructions instruct the visual agent to reduce the sampling rate of the target bounding box detection operator in its local dedicated toolchain and increase the filter smoothing time constant to suppress jitter caused by visual drift; simultaneously, the instructions instruct the channel agent to dynamically increase the matrix resolution of the phase cleaning operator and command the coordinating gateway to increase the proportion of time-frequency resource blocks fed back by the physical layer CSI in the next time slot. This process achieves self-healing from failure in multi-round long-chain agent tasks and a highly reliable, all-weather flow control dynamic closed loop.

[0052] Furthermore, based on the aforementioned latency-aware multimodal wireless fundamental model system, this invention also provides a collaborative reasoning method for a latency-aware multimodal wireless fundamental model system. A preferred embodiment of the collaborative reasoning method for a latency-aware multimodal wireless fundamental model system is as follows: Figure 3 As shown, it specifically includes: The multi-stream input and tokenization module receives multimodal data and converts the multimodal data into a token sequence through data segmentation and position embedding operators; The multi-encoder network uses a self-attention mechanism to extract self-modal features from the token sequence and uses an asymmetric cross-modal injection mechanism to generate cross-modal features from the self-modal features; The dynamic feature gating network calculates the weights of the self-modal features and the weights of the cross-modal features through a scoring layer; The latency-aware routing switch performs dual-track traffic splitting control based on the physical layer QoS latency index of the current task. When the current task is a hard real-time communication task, the fast inference path is activated, and when the current task is a latency-tolerant awareness task, the cognitive inference path is activated. The micro-collaborative network, in the cognitive reasoning path, generates multi-source local feedback data for the perception task through a multimodal agent based on the self-modal features, the cross-modal features, the weights of the self-modal features and the weights of the cross-modal features. It also uses the thought chain reasoning engine of the large language model to reconcile cross-modal cognitive conflicts and outputs the processing results and semantic explanation text.

[0053] Specifically, a collaborative reasoning method for a time-delay-aware multimodal wireless fundamental model system includes the following steps: Step S101: Receive heterogeneous multimodal wireless data streams, divide them into data blocks, and perform tokenization transformation to obtain a token sequence; Step S102: Input the token sequence into the multi-encoder network, extract independent modal features using the self-attention mechanism in the shallow / medium layers, and perform asymmetric cross-modal injection constraints using the cross-attention mechanism in the deep layers; Step S103: Utilize a dynamic feature-gated network to evaluate the feature / embedding vector of each modality at each network depth, and calculate the adaptive context scalar weights; Step S104: Detect the QoS latency requirement of the current task based on the latency-aware routing switch. If the current task is determined to be a physical layer hard real-time task, the physical features output by the CSI encoder are directly diverted to the fast inference path for fast processing with zero LLM latency. If the current task is determined to be a complex spatial awareness task, then proceed to step S105. Step S105: Map the multi-layer features to the equal-weighted multimodal latent space, and refine the features using the Top-k feature selection operator, then publish them to the shared blackboard collaborative network; Step S106: Drive multiple modality-specific agents to iteratively update the shared blackboard state, and use the central LLM orchestrator to reconcile cross-modal conflicts using the thought chain inference engine, outputting the final processing result and readable semantic explanation text.

[0054] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0055] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0056] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multimodal wireless fundamental model system based on time delay awareness, characterized in that, The time-delay-aware multimodal wireless basic model system includes: The multi-stream input and tokenization module is used to receive multimodal data and convert the multimodal data into a token sequence through data segmentation and location embedding operators; A multi-encoder network is used to extract self-modal features from the token sequence using a self-attention mechanism, and to generate cross-modal features from the self-modal features using an asymmetric cross-modal injection mechanism. A dynamic feature gating network is used to calculate the weights of the self-modal features and the weights of the cross-modal features through a scoring layer; The latency-aware routing switch is used to perform dual-track traffic splitting control based on the physical layer QoS latency index of the current task. When the current task is a hard real-time communication task, the fast inference path is activated, and when the current task is a latency-tolerant awareness task, the cognitive inference path is activated. A micro-collaborative network is used in the cognitive reasoning path to generate multi-source local feedback data for the perception task through a multimodal agent based on the self-modal features, the cross-modal features, the weights of the self-modal features and the weights of the cross-modal features. It also uses the thought chain reasoning engine of a large language model to reconcile cross-modal cognitive conflicts and output the processing results and semantic explanation text.

2. The multimodal wireless basic model system based on time delay awareness according to claim 1, characterized in that, The multimodal data includes channel state data, point cloud data, and visual data; the token sequence includes a channel token sequence, a point cloud token sequence, and a visual token sequence; and the multi-stream input and tokenization module includes: A multi-stream input unit is used to receive channel state data, point cloud data, and visual data; The tokenization unit is used to convert the channel state data, the point cloud data, and the visual data into a unified format channel token sequence, point cloud token sequence, and visual token sequence through data segmentation and location embedding operators, respectively.

3. The multimodal wireless basic model system based on time delay awareness according to claim 2, characterized in that, The self-modal features include channel features, point cloud features, and visual features; the cross-modal features include channel cross-modal features, point cloud cross-modal features, and visual cross-modal features; the multi-encoder network includes: A CSI encoder is used to extract channel features from the channel token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the CSI encoder. In the deep Transformer module of the CSI encoder, the channel features are used as query vectors, and the concatenation features of the point cloud features and the visual features are used as key-value pairs to generate channel cross-modal features using an asymmetric cross-modal injection mechanism. A point cloud encoder is used to extract point cloud features from the point cloud token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the point cloud encoder. In the deep Transformer module of the point cloud encoder, the point cloud features are used as query vectors, and the concatenation features of the channel features and the visual features are used as key-value pairs to generate point cloud cross-modal features using an asymmetric cross-modal injection mechanism. A visual encoder is used to extract visual features from the visual token sequence using a self-attention mechanism in the shallow and middle Transformer modules of the visual encoder. In the deep Transformer module of the visual encoder, the visual features are used as query vectors, and the concatenation features of the channel features and the point cloud features are used as key-value pairs to generate visual cross-modal features using an asymmetric cross-modal injection mechanism. The channel features, point cloud features, visual features, channel cross-modal features, point cloud cross-modal features, and visual cross-modal features constitute a multimodal initial joint feature vector.

4. The multimodal wireless basic model system based on time delay awareness according to claim 3, characterized in that, The time-delay-aware multimodal wireless basic model system also includes: A fast inference head is used in the fast inference path to linearly combine the channel features and the channel cross-modal features according to the weights of the channel features and the weights of the channel cross-modal features to obtain channel combination features, and to perform fast inference on the communication task according to the channel combination features to obtain a fast inference result for the communication task.

5. The multimodal wireless basic model system based on time delay awareness according to claim 4, characterized in that, The large language model includes a central LLM editor, and the micro-cooperative network includes: An equal-weighted multimodal latent space is used to map the initial joint feature vector of the multimodal modes to the equal-weighted multimodal latent space constructed based on contrastive learning constraints, so as to obtain the implicit feature representations aligned to each modality. The Top-k purification module is used to calculate the similarity between the prompt words of the perception task and the implicit representation of the features in the equal multimodal latent space through the Top-k feature selection operator, and sort the implicit representations of the features in descending order of similarity, and take the top k implicit representations of the features as the core spatiotemporal feature implicit representations. A centralized data warehouse is used to receive and store the implicit representation of the core spatiotemporal features, the initial joint feature vector of the multimodal and the upper-layer integrated computing action instructions corresponding to the perception task, and to aggregate the initial joint feature vector of the multimodal and the upper-layer integrated computing action instructions in the current time slot to obtain the current global task state. A multimodal intelligent agent is used to read and parse the current global task state, obtain the target orientation and region of interest for local feature purification, call the local dedicated toolchain based on the implicit representation of the core spatiotemporal features, and use the local dedicated toolchain to perform local feature purification on the token sequence based on the target orientation and the region of interest, generate multi-source local feedback data for the perception task, and asynchronously write the multi-source local feedback data back to the centralized data warehouse. The central LLM orchestrator is used to reconcile the cross-modal cognitive conflict when it is determined that a cross-modal cognitive conflict has occurred, based on the weights of the initial joint feature vector of the multimodal system and the multi-source local feedback data. It outputs the processing results and semantic explanation text, and issues control update instructions to the multimodal agent to form a dynamic control closed loop.

6. The multimodal wireless basic model system based on time delay awareness according to claim 5, characterized in that, The local dedicated toolchain includes a channel local dedicated toolchain, a point cloud local dedicated toolchain, and a vision local dedicated toolchain. The multi-source local feedback data includes channel local feedback data, point cloud local feedback data, and vision local feedback data. The multimodal intelligent agent includes: The channel agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the channel mode, it calls the channel-specific toolchain. Based on the target orientation and the region of interest, it uses Fast Fourier Transform and state filtering to purify the channel fine feature matrix from the channel token sequence. It also calculates the channel self-test probability score based on the signal-to-noise ratio of the received signal. The channel fine feature matrix and the channel self-test probability score are used as channel local feedback data, and the channel local feedback data is asynchronously written back to the centralized data warehouse. A point cloud agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the point cloud modality, the agent calls the point cloud local dedicated toolchain. Based on the target orientation and the region of interest, the agent uses a density-based topology segmentation algorithm to extract the fine feature matrix of the point cloud from the point cloud token sequence. The agent calculates the point cloud self-verification probability score based on the effective dynamic target point cloud density and noise ratio. The agent uses the fine feature matrix of the point cloud and the point cloud self-verification probability score as the point cloud local feedback data and asynchronously writes the point cloud local feedback data back to the centralized data warehouse. A visual intelligent agent is used to read and parse the current global task state to obtain the target orientation and region of interest for local feature purification. Based on the implicit representation of the core spatiotemporal features of the visual modality, the agent invokes the visual local dedicated toolchain. Based on the target orientation and the region of interest, the agent uses a convolutional feature map edge extraction algorithm to extract a visual fine feature matrix from the visual token sequence and calculates a visual self-verification probability score based on image contrast. The visual fine feature matrix and the visual self-verification probability score are used as visual local feedback data, and the visual local feedback data is asynchronously written back to the centralized data warehouse.

7. The multimodal wireless basic model system based on time delay awareness according to claim 6, characterized in that, The central LLM orchestrator includes: A data acquisition unit is used to acquire the multi-source local feedback data written back by the multimodal agent from the centralized data warehouse; The conflict determination unit is used to determine that a cross-modal cognitive conflict has occurred when, based on multi-source local feedback data and with the weights of the multimodal initial joint feature vector as the physical prior, a nonlinear deviation in the motion trajectory prediction of different modalities is detected and the channel self-verification probability score, the point cloud self-verification probability score, or the visual self-verification probability score is lower than the compliance threshold. The thought chain reasoning engine is used to reconcile the cross-modal cognitive conflict, output the processing results and semantic explanation text, and simultaneously issue control update instructions to the multimodal agent. The sampling rate or filter parameters of the local dedicated toolchain are dynamically adjusted through the control update instructions to form a dynamic control closed loop.

8. The multimodal wireless basic model system based on time delay awareness according to claim 5, characterized in that, The centralized data warehouse is deployed in the cloud or on edge servers as a shared blackboard.

9. The multimodal wireless basic model system based on time delay awareness according to claim 1, characterized in that, The scoring layer comprises a learnable, fully connected, lightweight deep neural network.

10. A collaborative reasoning method for a time-delay-aware multimodal wireless fundamental model system, wherein the collaborative reasoning method for a time-delay-aware multimodal wireless fundamental model system is implemented based on the time-delay-aware multimodal wireless fundamental model system according to any one of claims 1-9, characterized in that, The collaborative reasoning method for the time-delay-aware multimodal wireless fundamental model system includes: The multi-stream input and tokenization module receives multimodal data and converts the multimodal data into a token sequence through data segmentation and position embedding operators; The multi-encoder network uses a self-attention mechanism to extract self-modal features from the token sequence and uses an asymmetric cross-modal injection mechanism to generate cross-modal features from the self-modal features. The dynamic feature gating network calculates the weights of the self-modal features and the weights of the cross-modal features through a scoring layer; The latency-aware routing switch performs dual-track traffic splitting control based on the physical layer QoS latency index of the current task. When the current task is a hard real-time communication task, the fast inference path is activated, and when the current task is a latency-tolerant awareness task, the cognitive inference path is activated. The micro-collaborative network, in the cognitive reasoning path, generates multi-source local feedback data for the perception task through a multimodal agent based on the self-modal features, the cross-modal features, the weights of the self-modal features and the weights of the cross-modal features. It also uses the thought chain reasoning engine of the large language model to reconcile cross-modal cognitive conflicts and outputs the processing results and semantic explanation text.

Images