Anomaly causal segmentation method and system fusing multi-modal perception and causal reasoning

Abstract

This invention relates to the fields of computer vision and artificial intelligence, specifically providing an anomaly causal segmentation method and system that integrates multimodal perception and causal reasoning. Based on static and dynamic slots, a hybrid risk warning word is generated. The multimodal key information token set and the hybrid risk warning word are fused to generate a global risk token representing the overall risk state. The global risk token undergoes spatiotemporal dependency modeling and optimization to obtain an optimized global risk token sequence. Based on the optimized global risk token sequence and the hybrid risk warning word, multi-step causal reasoning is performed to generate a high-semantic risk analysis token and obtain a risk representation. Based on the high-semantic risk analysis token and the risk representation, risk region location and segmentation are achieved through recursive interaction to complete the anomaly causal segmentation, thereby improving the accuracy and interpretability of the anomaly causal segmentation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically providing an anomaly causal segmentation method and system that integrates multimodal perception and causal reasoning. Background Technology

[0002] Existing methods face challenges when dealing with complex scenarios involving intelligent agents, such as difficulty in achieving integrated dynamic modeling of multimodal high-dimensional data, lack of risk cause indication and anomaly identification mechanisms, and inflexible response to expert knowledge diversion. These shortcomings significantly limit the interpretability of risk segmentation and the intelligent response capability of the system under complex events.

[0003] With the development of large-scale models, multimodal large language models are being applied to autonomous driving, robot perception, and intelligent question answering scenarios. In agent scenarios, causal reasoning is used to investigate the causes of short-term anomalies in the agent's surroundings. Although traditional multimodal fusion methods have made some progress in feature representation and information complementarity, they still suffer from limitations in generalization, insufficient anomaly detection sensitivity, and low accuracy. Summary of the Invention

[0004] This application provides an anomaly causal segmentation method and system that integrates multimodal perception and causal reasoning to solve the problems of insufficient sensitivity in risk anomaly detection and inaccurate risk causal analysis.

[0005] This application provides an anomaly causal segmentation method that integrates multimodal perception and causal reasoning, the method comprising: S1. Multi-source heterogeneous data acquisition and preprocessing: Acquire multi-source heterogeneous data, perform standardization and modal alignment preprocessing on the multi-source heterogeneous data, and input the preprocessed modal data into the corresponding modal encoder for encoding to generate the original token sequence of each modality. S2. Hybrid Risk Warning Term Design: Generate hybrid risk warning terms based on static slots and dynamic slots; S3. Adaptive Token Simplification: Adaptively filter and simplify the original token sequences of each modality to obtain a multimodal key information token set; S4. Cross-modal fusion processing: The multimodal key information token set and the mixed risk warning words are fused to generate a global risk token representing the overall risk status. S5. Spatiotemporal modality adaptation processing: The spatiotemporal dependency relationship of the global risk token is modeled and optimized to obtain the optimized global risk token sequence. S6. Risk Causal Analysis and Segmentation: Based on the optimized global risk token sequence and the mixed risk warning words, multi-step causal reasoning is performed to generate a high semantic risk analysis token and obtain a risk representation; according to the high semantic risk analysis token and the risk representation, risk area location and segmentation are achieved through recursive interaction to complete abnormal causal segmentation.

[0006] In some embodiments, the step of fusing the multimodal key information token set with the hybrid risk warning words to generate a global risk token representing the overall risk status includes: A deformable attention mechanism is applied to the token group of each modality in the multimodal key information token set to extract and enhance features within the group, thereby obtaining enhanced token groups for each modality. Using each modal enhancement token group as a query and the hybrid risk warning word as a key and value, the association weight between the hybrid risk warning word and each modal enhancement token group is calculated through a deformable cross-attention mechanism to obtain a cross-modal interaction token group that integrates risk warning information. A deformable attention mechanism is applied to the cross-modal interaction token group, focusing on the risk feature with the highest weight, to obtain the updated token groups for each modality; The updated modal token groups are then combined to generate a global risk token that represents the overall risk status.

[0007] In some embodiments, the step of locating and segmenting risk regions through recursive interaction based on the high semantic risk analysis token and the risk representation to complete anomaly causal segmentation includes: The high semantic risk analysis token and the risk representation are aligned in feature dimensions to obtain fusion input features with consistent dimensions; the high semantic risk analysis token contains preset anomaly root cause semantic category labels; The fused input features are input into a hierarchical query-aware decoder, which includes a spatial localization layer, a semantic association layer, and a causal determination layer. The high semantic risk analysis token is used as the query vector. The decoder interacts with the risk representation layer by layer through a multi-level deformable cross-attention mechanism corresponding to the three-layer structure of the decoder. The output includes a root cause region mask, an association region mask, and a corresponding risk mask and probability distribution to complete the abnormal causal segmentation. The specific process of interacting with the risk representation layer by layer through a multi-level deformable cross-attention mechanism corresponding to the three-layer structure of the decoder is as follows: The spatial positioning layer performs spatial feature sampling on the risk representation through a deformable cross-attention mechanism and outputs a preliminary segmentation mask for the risk region. The semantic association layer calculates the semantic similarity between the query vector and the region features corresponding to the preliminary segmentation mask, and filters out candidate regions that match the semantic category label of the anomaly root cause based on the semantic similarity. The causal determination layer calculates the risk propagation gradient between regions based on the attention weights of the candidate regions, and recursively determines the root cause region and related region in the candidate regions until a preset convergence condition is met.

[0008] The present application provides an anomaly causal segmentation method and system that integrates multimodal perception and causal reasoning. The system achieves integrated multimodal spatiotemporal features through pseudo-image transformation and CLIP unified tokens. It also introduces hybrid risk warning word design and adaptive token simplification, cross-membrane linked attention, and spatiotemporal modality adapters to dynamically aggregate high-confidence risk tokens for accurate mining of influencing causes. Through MEA (Multimodal Image Processing) integration of multimodal perception and causal reasoning, the system achieves conditional knowledge diversion through risk expert selection and fine-tuning mechanisms. Downstream query perception decoder, in conjunction with hierarchical causal reasoning, effectively outputs the main cause of the anomaly and the risk mask. This improves the model's expressive power and reasoning efficiency while also enhancing the accuracy of anomaly causal segmentation. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of the abnormal causal segmentation method that integrates multimodal perception and causal reasoning provided by the present invention; Figure 2 This is a schematic diagram illustrating the architecture principle of the multimodal expert fusion intelligent risk perception system provided by the present invention; Figure 3 This is a schematic diagram illustrating the principle and process of the adaptive clustering token simplification provided by the present invention; Figure 4 This is a schematic diagram of the cross-membrane linkage attention principle provided by the present invention; Figure 5 This is a schematic diagram of the risk expert selection and fine-tuning mechanism provided by the present invention; Figure 6 This is a schematic diagram of the hierarchical risk decoding and adaptive masking process of the query-aware decoder provided by the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0012] The following is combined Figures 1 to 6 The illustrated embodiments describe the technical solution of the present invention: This application provides an embodiment of an anomaly causal segmentation method and system that integrates multimodal perception and causal reasoning, referring to... Figure 1 As shown, the abnormal causal segmentation method integrating multimodal perception and causal reasoning provided in this embodiment includes the following steps: S110: Architecture design of a multimodal expert fusion intelligent risk perception system.

[0013] In some embodiments, the implementation of step S110 (architecture design of a multimodal expert fusion intelligent risk perception system) may include: It should be noted that existing methods suffer from several drawbacks. These include the difficulty in integrating multimodal high-dimensional information into dynamic modeling, the lack of risk cause indication and anomaly cause identification mechanisms, the susceptibility of cause signals to be redundantly masked in token representation, and the weak response capability of expert knowledge diversion. Consequently, the interpretability of risks and the intelligent response capability of the system in complex scenarios are significantly limited.

[0014] It should be noted that the system uses CLIP unified tokens to achieve integrated multimodal spatiotemporal features, and introduces hybrid risk warning word design and adaptive clustering token simplification, cross-membrane linked attention, and spatiotemporal modality adapters to dynamically aggregate high-confidence risk tokens and accurately mine the influencing causes. Conditional knowledge diversion is achieved through MEA risk expert selection and fine-tuning mechanisms, and downstream query-aware decoders, in conjunction with hierarchical causal reasoning, effectively output the main causes of anomalies and risk masks. This improves the model's expressive power and reasoning efficiency while achieving accuracy in anomaly causal reasoning performance in complex autonomous scenarios.

[0015] like Figure 2The diagram illustrates the architecture of a multimodal expert fusion intelligent risk perception system. The system first acquires multi-source data related to the operational safety of the intelligent agent in the environment, based on external sensing devices such as cameras, radar, microphones, and weather sensors. This multi-source data includes real-time images, audio, weather data, geographic information, scene rules, and semantic text. All raw modal information is categorized and processed into tokenization. Text and semantic information undergo high-semantic token extraction using the CLIP-Text model. Multi-source sensor signals, such as those from cameras, radar, microphones, and weather sensors, are fused using the CLIP-Image model to achieve cross-modal, high-dimensional integrated representation, thereby improving the semantic compression and modal collaboration capabilities of complex structures.

[0016] Specifically, the overall system framework is as follows: First, multi-source perception and rule data are collected, including: text data (traffic rules or safety regulations), camera (visual targets), millimeter-wave radar (obstacle ranging and speed measurement), lidar (high-precision 3D point cloud), microphone (abnormal audio), and weather station (environmental parameters such as rainfall / visibility).

[0017] It should be noted that by integrating tokens from different sources through a cross-membrane linkage attention mechanism, the associative understanding of multimodal information can be achieved; and by combining time and space dimensions through a spatiotemporal modal adapter, the information can be made more relevant to the task scenario of risk perception.

[0018] Specifically, multimodal token feature fusion and task adaptation: a cross-modal linkage attention mechanism is adopted to integrate text tokens and cross-modal pseudo tokens (image, radar, audio, and meteorological feature embedding) in two dimensions. First, the semantic correlation of key tokens within a single modality is strengthened, as well as the spatial location correlation of radar point cloud tokens and the temporal sequence correlation of audio tokens. Then, the semantic mapping relationship between tokens in multiple modalities is explored, such as the correlation between the rainfall text token of meteorological parameters and the feature token of radar detection accuracy decline.

[0019] Specifically, the multi-source heterogeneous data includes text data, visual data, radar data, audio data, and meteorological data. Differentiated standardization and cross-modal coarse alignment are performed according to the characteristics of each modality of data to obtain standardized multi-source data. It should be noted that token extraction and rule-based structural modeling are performed on text data, feature extraction and spatiotemporal target association are performed on radar point clouds, visual images, and audio signals, and meteorological data is calibrated and updated in real time to provide standardized input for subsequent multimodal fusion inference.

[0020] Features of each modality are extracted from standardized multi-source data. The features of each modality are input into a spatiotemporal correlation scene modeling network. The semantic relationship between the features of each modality in time and space is constructed through the network, and a feature-level scene token that integrates global scene information is generated. The feature-level scene tokens and standardized multi-source data are aligned in dimension through linear mapping, and then the features are stitched together to obtain multimodal data that integrates scene information. The CLIP multimodal model is used to perform unified semantic modeling on multimodal data with fused scene information, and output cross-modal aligned modality-specific vector representations corresponding to text, vision, radar, audio, and meteorology, respectively. The cross-modal aligned modality-specific vector representations are input into the corresponding modality-specific encoders according to modality categories to generate the original token sequence for each modality.

[0021] It should be noted that after the initial token extraction and high-dimensional alignment, the system introduces a deformable attention token simplification network to adaptively purify and align the highly redundant token streams obtained from different modalities; then, the deformable attention network is used to adaptively filter and optimize the tokens, retaining key information.

[0022] Specifically, in risk causality detection: a sparse gated hybrid expert MEA model is introduced, using hybrid risk warnings and scene tokens as dual guiding information. Based on the risk type of the scene token, such as collision risk, intrusion risk, equipment failure risk, and modal contribution, the risk is dynamically distributed to the corresponding domain expert network. For example, vision experts are responsible for target recognition risks, and radar experts are responsible for distance and speed risks. Flexible interaction is achieved through weighted fusion of expert outputs. The weights are dynamically adjusted by the gated network according to the real-time scene complexity. To adapt to the risk causality rules of specific domains, an Adapter-LORA joint fine-tuning strategy is adopted. A domain Adapter (capturing risk causal relationship features) is inserted into the backbone network of the MEA model. This ensures fine-tuning efficiency while avoiding catastrophic forgetting, thereby improving the accuracy of risk causality detection.

[0023] It should be noted that the risk perception layer, based on the fused multimodal features, outputs a preliminary risk factor segmentation mask and its surrounding risk probability distribution through a risk target decoder.

[0024] It should be noted that the causal reasoning layer organizes the above-mentioned mask, probability distribution, original risk warning and other information into a structured description and inputs it into an LLM dedicated to risk reasoning. The LLM is responsible for risk causal detection and generating an in-depth risk analysis report containing root cause, evolution and correlation.

[0025] It should be noted that, in order to enhance the system's ability to dynamically monitor and fuse high-risk factors, a cross-membrane linkage attention mechanism is integrated on the basis of token adaptive purification. This enables dynamic spatial anomaly enhancement, establishes global risk complementarity and heterogeneous causal relationships among multiple modalities, and improves the system's explanatory power of principal risk and accuracy of spatial segmentation in complex scenarios.

[0026] S120: Based on static and dynamic slots, generate hybrid risk words, adaptively filter and simplify the original token sequences of each modality, and obtain a multimodal key information token set.

[0027] In some embodiments, the implementation of step S120 (generating hybrid risk words based on static and dynamic slots, adaptively filtering and simplifying the original token sequences of each modality to obtain a multimodal key information token set) may include: It should be noted that high-dimensional perceptual tokens suffer from significant redundancy, and the primary risk signal is easily masked by information noise, affecting the efficiency of downstream segmentation and primary cause attribution. By utilizing an attention-k-nearest neighbor clustering-adaptive strategy, high-confidence primary cause tokens are dynamically aggregated, effectively compressing the number of tokens within and outside the modality and increasing the effective information density. This enhances the representational ability of key risk tokens, making anomaly segmentation and primary cause explanation more focused and efficient.

[0028] It should be noted that static and dynamic prompts are organically combined, and hybrid risk words are generated based on static and dynamic slots, serving as guiding signals for subsequent token processing, attention mechanisms, and expert routing.

[0029] Specifically, hybrid risk warning words are generated based on static feature slots and dynamic task slots; For example, the inputs are: static slots: fixed instructions such as risk detection type, output requirements, and standard sensing area; dynamic slots: real-time environmental status (foggy / complex street), sensor status, expert activation suggestions, and historical memory recall information. Output: A sequence of hybrid risk warning words (including [state][cls]: natural description of the environment / state, [sensor]: list of multimodal signal sources, [anomaly-cls]: structured fields such as key anomaly types / events); the hybrid risk warning word vector can be directly input into the unified warning vector of the backbone model to guide the subsequent token processing and reasoning direction.

[0030] For example, in a static slot: [anomaly-cls]: emergency stop, unmarked obstacle; [mask]: output high-risk segmentation mask and anomaly category ahead; Dynamic slot: [state][cls]: foggy weather, complex interaction scenarios, [pos]: timestamp + spatial coordinate positioning, [expert-hint]: currently active visual-radar expert or algorithm mechanism.

[0031] It should be noted that, under the premise that the detection instruction structure is constant, if a rare environment or scene changes suddenly, the dynamic slot will automatically fill the visibility of <10m. It is recommended to pay attention to distant moving targets, and the model expert branch and mask output will be adjusted accordingly.

[0032] It should be noted that in the multimodal perception task of intelligent agents, the expression dimensions and spatial distribution of different modalities are significantly different. The token simplification mechanism using adaptive clustering can achieve comprehensive optimization of the token embedding space of each modality, including dimensionality reduction and alignment, computational efficiency, and representation density.

[0033] like Figure 3 The diagram illustrates the principle and process of adaptive clustering for token simplification: the left side shows the original token pool generated by the multimodal input after encoding by each modality encoder, followed by the multimodal token input pool after dimensionality reduction and initial simplification; the middle section of the process demonstrates the core simplification mechanism: the Q branch acts as a dynamic command center, guiding the token grouping process through red guide arrows, thus abandoning static grouping rules; First, dynamic guiding signals (such as those generated by the query branch in deformable attention) are produced. Figure 3 (Indicated by red arrows in the diagram) The tokens are grouped in real time according to the input content; then, the K-nearest neighbor algorithm is used to intelligently aggregate tokens based on semantic similarity in the embedding space. The clusters circled in the diagram represent the aggregation groups formed based on their proximity to the main token (or cluster center). Finally, as shown on the right side of the diagram, the weighted aggregation of each cluster is processed to output a simplified set of multimodal key information tokens.

[0034] It should be noted that each Guide stage not only adjusts the merging scale of each group of tokens, but can also adaptively adjust it based on conditions such as anomaly detection density and environmental complexity.

[0035] The specific process is as follows: 1) Dynamic Token Grouping Mechanism for Multimodal Input: The Q-branch (Query branch) serves as the dynamic command center, guiding token grouping across modalities via Guide arrows, rather than static rules or fixed thresholds. This dynamic Q-branch command mechanism allows for real-time adjustments to the grouping strategy based on changes in input content, achieving dynamic adaptive grouping decisions. 2) Enhanced Aggregation Based on K-Nearest Neighbors and Embedded Space Similarity: Deep integration of the K-nearest neighbor algorithm with embedded space similarity calculation enables intelligent token aggregation. During aggregation, not only spatial distance is considered, but also embedded space semantic similarity is combined for more accurate token aggregation, thereby improving the accuracy and rationality of token aggregation. 3) Dual Adaptive Adjustment Mechanism in the Guide Stage: Each Guide stage simultaneously implements adaptive adjustment in two dimensions: Dynamic adjustment of token grouping size: Automatically adapting the number of tokens in each group based on quantified parameters of the input content's feature complexity; Dynamic adaptation of anomaly detection density: Dynamically adjusting the anomaly detection density threshold based on quantified parameters of the input data's risk level or distribution characteristics.

[0036] The adaptive adjustment mechanism for group size works as follows: It should be noted that the adaptive adjustment mechanism for group size is the first adjustment mechanism in the Guide phase. It dynamically adjusts the size of the token group guided by each Query anchor point based on the anomaly detection density of the local area. The anomaly detection density of the area where each Query anchor point is located is evaluated in real time. The calculation process of anomaly detection density comprehensively considers multiple factors such as the strength of abnormal signals, the frequency of abnormal events, and historical anomaly patterns in the spatial neighborhood around the anchor point.

[0037] For the The specific formula for calculating the anomaly detection density for each anchor point is as follows: ; in, For anomaly detection density, For the first anchor points The set of tokens within the neighborhood; For the first The strength of the abnormal signal for each token. For the first The frequency of historical anomalies for each token For the first The degree of matching of the historical abnormal patterns of each token. , , These are the corresponding weighting coefficients.

[0038] It should be noted that this density calculation process achieves a quantitative assessment of the anomaly risk in local areas, providing a reliable input signal for subsequent group size adjustment. Based on the anomaly detection density, the system dynamically determines the group size through an adaptive mapping function.

[0039] Group size The calculation formula is: ; in, For group size, Based on the group size, To adjust the amplitude coefficient, In response to the steepness coefficient, Density threshold It is the hyperbolic tangent function.

[0040] when hour, The function outputs a negative value, making The system uses a smaller group size; when hour, The function outputs a positive value, making The system uses a large group size.

[0041] It should be noted that the adaptive adjustment process enables the system to automatically adjust the information retention strategy according to the density of local anomalies, achieving precise modeling of high-risk areas and efficient processing of low-risk areas.

[0042] It should be noted that the adaptive adjustment of group size is reflected not only in the group size of individual anchor points, but also in the coordination of the global grouping strategy. By introducing a group size balancing mechanism, the system ensures a reasonable balance in the distribution of group sizes across different anchor points, avoiding computational load imbalances caused by extreme differences in group sizes. This balancing mechanism is implemented through constraint functions.

[0043] The formula for calculating load balancing is: ; in, For load balancing, For group size, For the balance coefficient, The average size for all groups. To the maximum group size, This is the minimum group size.

[0044] It should be noted that the adaptive adjustment of detection sensitivity is the second adjustment mechanism in the Guide stage. It dynamically adjusts the system's response strength to abnormal signals according to the complexity of the environment, so that the system uses higher sensitivity in simple environments to capture subtle anomalies, and lower sensitivity in complex environments to reduce false alarm interference.

[0045] It should be noted that the environmental complexity assessment process comprehensively considers multiple dimensions such as the visual complexity of the scene, the density of moving targets, lighting conditions, and weather conditions.

[0046] The formula for calculating environment complexity is as follows: ; in, Due to environmental complexity, For the first Complexity features in several dimensions For the first The weight coefficients corresponding to the complexity features in each dimension The Sigmoid activation function is used for normalization; The number of dimensions.

[0047] It should be noted that, based on environmental complexity, the system dynamically adjusts the detection sensitivity parameters through an adaptive function. The adjustment of detection sensitivity is mainly reflected in the spatial decay coefficient during similarity calculation. Balance coefficient in attention calculation Two key parameters: Spatial attenuation coefficient. Controlling the influence of spatial distance on similarity, a larger The value makes the system pay more attention to spatially proximate tokens, smaller ones. This value allows the system to consider tokens from more distant locations.

[0048] The formula for adjusting the environmental complexity is as follows: ; in, This is an adjustment coefficient for environmental complexity. Based on the attenuation coefficient, To attenuate the adjustment amplitude, For environmental complexity.

[0049] It should be noted that, Approaching 0 As the system grows larger, it focuses more on local spatial information; when the environment is highly complex... Approaching 1, By reducing the amount of information, the system can utilize a wider range of spatial information to cope with complex scenarios.

[0050] It should be noted that the balance coefficient in attention calculation Controlling the contribution of weighted similarity to attention scores, a larger The value makes the system rely more on weighted similarity information, and a smaller value... The value makes the system more reliant on standard dot product attention.

[0051] The formula for calculating the balance coefficient of environmental complexity is: ; in, This is a balance coefficient for environmental complexity. Based on the basic balance coefficient, To balance the adjustment range, For environmental complexity.

[0052] It should be noted that when At higher levels, As the number of similarities increases, the system relies more heavily on weighted similarity that includes important weights and spatial information to improve its adaptability to complex scenarios; when At lower levels, The system relies more on standard attention mechanisms to maintain computational efficiency.

[0053] It should be noted that the adaptive adjustment of detection sensitivity is also reflected in the dynamic adjustment of importance weights in similarity calculation. The system adjusts the importance weights based on environmental complexity. .

[0054] The method for calculating the importance weights adjusted for environmental complexity is as follows: ; in, Adjust the importance weights according to environmental complexity. As the standard importance weight, To enhance importance weight, Due to environmental complexity, The minimum complexity threshold, The minimum complexity threshold, This is the adjustment coefficient.

[0055] It should be noted that this piecewise function allows the system to use standard weights in low-complexity environments and enhanced weights in high-complexity environments to increase attention to key information.

[0056] It should be noted that the group size and detection sensitivity are improved through collaborative optimization to enhance overall performance. Firstly, the anomaly detection density... With environmental complexity There is an inherent correlation between them; high-complexity environments are often accompanied by higher anomaly detection density. Therefore, the two adjustment mechanisms are correlated in terms of input signals. Secondly, adjusting the group size affects the granularity of token aggregation, while adjusting the detection sensitivity affects the token selection strategy. Both together determine the final token simplification effect. Finally, the two mechanisms share contextual information. Information exchange enables mutual reference and coordination in the regulatory process.

[0057] It should be noted that the collaborative work is accomplished through a joint optimization function, and the system determines the group size. With detection sensitivity parameters , At the same time, we not only consider their respective independent optimization objectives, but also the overall effect after their collaboration.

[0058] The joint optimization objective function is defined as: ; in, To jointly optimize the objective function, To reconstruct the loss, It is a sparsity regularization term. For sensitivity regularization, For group size, To detect sensitivity parameters; It should be noted that, Reconstruction loss measures the degree to which the simplified token retains the original information. As a sparse regularization term, using a smaller group size can improve efficiency; This is a sensitivity regularization term that balances detection sensitivity and false alarm rate. By jointly optimizing this objective function, the system can find the optimal combination of group size and detection sensitivity, achieving synergistic optimization of performance and efficiency.

[0059] It should be noted that the collaborative operation of the dual adaptive adjustment mechanism is also reflected in the dynamic feedback mechanism. The system monitors the anomaly detection performance after token simplification and dynamically adjusts the learning rate and update strategy of the adjustment parameters. When detection performance declines, the system enhances the response strength of group size adjustment and increases attention to high-risk areas; when the false alarm rate increases, the system reduces detection sensitivity and reduces the response to noise signals.

[0060] It should be noted that all adjustment parameters and mapping functions of the dual adaptive adjustment mechanism in the Guide phase are optimized through gradient backpropagation. During the training phase, the system simultaneously optimizes token simplification and anomaly detection performance using a multi-task loss function, enabling the adaptive adjustment mechanism to learn the optimal parameter mapping relationship. The loss function is designed to consider multiple objectives such as reconstruction error, detection accuracy, and computational efficiency, achieving multi-objective optimization through weighted summation. ; in, To reconstruct the loss, To detect the loss, For efficiency loss, , , These are the corresponding weighting coefficients.

[0061] It should be noted that through adaptive control in the Guide phase, the system can achieve high-density representation of high-risk areas and efficient processing of low-risk areas, providing high-quality, streamlined token input for subsequent high-level modules such as multimodal fusion, expert routing, and causal reasoning.

[0062] Specifically, based on the deformable attention-guided multimodal input token dynamic grouping mechanism, the original token sequences of each modality are adaptively grouped based on the semantic importance weight of each token to obtain several token subsets within each modality; The K-nearest neighbor algorithm is deeply integrated with the embedding space similarity calculation. Feature aggregation is performed on several token subsets in each modality within the embedding space. During the aggregation process, the spatial distance and semantic similarity of the token embedding vectors are taken into account to form several token clusters. A dual adaptive adjustment mechanism is implemented for each token cluster to simultaneously achieve dynamic adjustment of token group size and dynamic adaptation of anomaly detection density; Based on the results of the dual adaptive adjustment mechanism, for the first The nearest neighbor groups corresponding to each anchor point are weighted and summed using normalized similarity weights to generate a simplified token group corresponding to each anchor point. The specific calculation formula is as follows: Specifically, for the first The nearest neighbor groups corresponding to each anchor point are weighted and summed using normalized similarity weights to generate a simplified token group corresponding to each anchor point. The specific calculation formula is as follows: ; in, Indicates the first Each anchor point corresponds to a simplified token group; For the first The set of nearest neighbor K-grouped indexes corresponding to each query vector; For the first Value branch characteristics corresponding to each Token; For the first The anchor point and the first Gating weights for each token; The set of simplified token groups corresponding to all anchor points constitutes the multimodal key information token set.

[0063] It should be noted that, unlike typical KNN coarse indexing, in this application, Q serves as the guiding center, and tokens with high similarity and semantic proximity in K actively cluster around Q, forming a dynamic clustering structure. The affiliation of each group of K-Tokens not only reflects local spatial or semantic consistency, but also strengthens the dense representation of key scenario tokens such as anomaly areas, small targets, and boundaries, reducing the spatial dilution of key information.

[0064] It should be noted that, based on the aggregation of all Q groups, the system ultimately outputs K concise tokens with high representation density, containing the most core risk and environmental information. The information is highly condensed, providing the most representative token input for subsequent high-level modules such as multimodal fusion, interactive attention, expert routing, and causal chain reasoning.

[0065] It should be noted that during the token aggregation process, the amount of information carried by different tokens and their importance to downstream tasks vary. By dynamically adjusting the weights, the system can enhance high-value information and suppress low-value information, thereby maximizing the retention of key task-related information while compressing information.

[0066] It should be noted that by combining semantic similarity with importance score, the system obtains an initial weight allocation. This initial weight allocation normalizes the product of semantic similarity and importance, so that tokens that are both semantically similar to the Query anchor and have high importance receive a larger initial weight.

[0067] The system adjusts the initial weights using a spatial distance decay function: ; in, For the Token The anchor point and the first The initial weight adjustment coefficient for each anchor point The spatial attenuation coefficient, For the Token The anchor point and the first Spatial distance between anchor points For the Token The anchor point and the first Preliminary weight allocation for each anchor point; It is an exponential function.

[0068] It should be noted that this design increases the weight of tokens that are spatially adjacent, while decreasing the weight of tokens that are spatially distant, thereby strengthening the local spatial pattern.

[0069] The system calculates the matching degree between the current token and historical anomaly patterns, and uses this matching degree as a weight adjustment factor. Tokens with high historical matching degrees are assigned higher initial weights to leverage historical knowledge to guide the current weight allocation. This mechanism is implemented through weighted fusion. ; in, For Token number The degree of matching between each anchor point and historical anomaly patterns. Historical information fusion coefficient; For the Token The anchor point and the first Weighted fusion of anchor points For the Token The anchor point and the first Preliminary weight allocation for each anchor point.

[0070] It should be noted that by fusing multi-source information, the initial weight allocation not only considers the feature information at the current moment, but also incorporates historical experience knowledge, providing a high-quality starting point for subsequent weight updates.

[0071] S130: Cross-modal fusion processing: The multimodal key information token set and mixed risk warning words are fused to generate a global risk token that represents the overall risk status.

[0072] In some embodiments, the implementation of step S130 (cross-modal fusion processing: fusing the multimodal key information token set with mixed risk warning words to generate a global risk token representing the overall risk status) may include: like Figure 4 This is a schematic diagram of the cross-membrane linkage attention principle. The input of this module is the token-group of each modality (visual / radar / semantic, etc.) after adaptive clustering. Multiple group tokens + risk guidance, and mixed risk warning words are introduced. First, self-attention is calculated within the group. Then, deformable attention is calculated across modalities. Deformable attention is performed again on the updated token-group to focus on the most critical risk token.

[0073] Specifically, a deformable attention mechanism is applied to the token group of each modality in the multimodal key information token set to extract and enhance features within the group, resulting in enhanced token groups for each modality. Using each modal augmented token group as the query and the mixed risk warning words as the key and value, the association weight between the mixed risk warning words and each modal augmented token group is calculated through a deformable cross-attention mechanism to obtain a cross-modal interactive token group that integrates risk warning information; A deformable attention mechanism is applied to the cross-modal interaction token group, focusing on the risk feature with the highest weight, to obtain the updated token groups for each modality; The updated modal token groups are concatenated to generate a global risk token that represents the overall risk status. The specific calculation formula for the deformable cross-attention mechanism is as follows: ; in, Indicates the first Feature vector of each query token; Indicates the first The set of sampling point indexes corresponding to each query token (usually a fixed-size set, such as...) (corresponding to a 3×3 regular grid). Indicates the first Initial reference coordinates for each sampling point; Indicates the first The learnable offset of each sampling point; Indicated in the Value feature map Above, in the coordinate system, bilinear interpolation is used. The feature vector obtained by sampling at point; This is a cross-attention calculation function with deformable offset, and its output is a query. The weighted sum of the Value features at a series of corresponding deformed sampling points; Attention weights; Attention weight The calculation formula is: ; in, The feature vector representing the mixed risk warning is used to dynamically adjust the attention weights and enhance the focus on risk-related features; For feature dimensions; This represents the coordinates on the key feature map K through bilinear interpolation. The feature vector obtained from sampling is used as the key vector for matching with the query when calculating attention weights; This represents the coordinates on the key feature map K through bilinear interpolation. The feature vector obtained by sampling at point, where It is a summation index that iterates through all sample points; is the transpose function, used to calculate the inner product between two vectors; It is an exponential function.

[0074] It should be noted that the core logic of the formula is: to perform a weighted summation of the neighboring Token and Value features after the offset, according to the dynamic attention weights, and output the first... Attention aggregation features of each query token. Hybrid risk alerts directly participate in guiding deformable attention, focusing on the dominant risk factors of different groups of tokens; this attention weight is dynamically adjusted by the feature information of hybrid risk alert words.

[0075] It should be noted that a global risk token is formed by fusing the tokens through a globally deformable aggregation module, thereby strengthening the dominant risk factors and suppressing irrelevant noise. Intramodal self-attention can strengthen the spatial dependencies within each modality; intermodal deformable cross-attention, guided by risk warning words, establishes risk associations between cross-modal tokens, achieving information complementarity.

[0076] S140: Spatiotemporal modality adaptation processing: The spatiotemporal dependency relationship of the global risk token is modeled and optimized to obtain the optimized global risk token sequence.

[0077] In some embodiments, the implementation of step S140 (spatiotemporal modality adaptation processing: modeling and optimizing the spatiotemporal dependency relationship of the global risk token to obtain an optimized global risk token sequence) may include: It should be noted that in the multimodal scenario of intelligent agents, risks and abnormal events not only have spatial distribution, but also rely more on temporal dynamic evolution; adaptive learning is carried out by using a spatiotemporal modality adapter.

[0078] First, the modal and hierarchical risk token-groups output from the previous stage are fed into the self-attention and cross-attention modules at each time step to generate different time slices ( The spatial risk token representation sequence is then used. Next, temporal positional encoding is employed to sequentially associate risk token groups across different time slices, forming a dynamic spatiotemporal information flow. STMA aggregates spatial tokens across time using a sliding window (or recursive gating) method, encoding their temporal changes and relationships, thereby achieving spatiotemporal tracking and fusion of anomalous signals. It should be noted that, to achieve efficient adaptive modeling of spatiotemporal features, the core of the STMA module adopts a standard depth-wise 3D convolution (DWConv3D) structure to dynamically fuse token information across the spatiotemporal dimensions.

[0079] Specifically, global risk tokens are arranged in chronological order to generate an initial global risk token sequence, and the feature quantity of the initial global risk token sequence is denoted as... ; Building a spatiotemporal modal adapter The spatiotemporal modality adapter fuses depthwise separable 3D convolution and deformable temporal attention mechanisms, and its specific calculation formula is as follows: ; in, Represents the features after dimensionality reduction Perform depthwise separable 3D convolution operations to extract spatiotemporal features; This is the downsampling weight matrix, used to reduce the feature dimension and thus reduce computational cost; This is an upsampling weight matrix used to recover the feature dimensions to match the original input. Consistency is required to enable residual joins; As a deformable temporal attention mechanism, for The output features are used for temporal dependency modeling, and the attention range on time is dynamically adjusted through learnable offsets to achieve more flexible temporal feature enhancement. It serves as a spatiotemporal modal adapter, fusing the original features with the enhanced spatiotemporal features through residual connections.

[0080] The initial global risk token sequence is input into the spatiotemporal modality adapter. Spatiotemporal dependency modeling and feature optimization are performed using depthwise separable 3D convolution and deformable temporal attention to obtain the optimized global risk token sequence.

[0081] It should be noted that the spatial-temporal global risk token should be entered first. After linear mapping, multi-scale spatiotemporal features are extracted using DWConv3D. After activation and dimensionality increase / decrease operations, the features are summed with the original token residuals and finally reshaped back to the original spatial dimensions to output the optimized global risk token sequence.

[0082] In this process, a deformable temporal attention mechanism is introduced, enabling high-confidence spatial risk features to be dynamically aggregated, memorized, and filtered over time. The specific process is as follows: Will arrive Space risk token sequence at time As input, the spatiotemporal modal adapter Temporal information is fused through a deformable temporal attention mechanism to output the current frame. Main risk token The calculation formula is as follows: ; in, Indicates the current frame A dynamic neighborhood set in the time domain Indicates the first Learnable temporal offset for each temporal sampling point; Indicates at time Spatial risk token characteristics (which may be a globally pooled characteristic or a characteristic specific to a location); Indicates the current frame With the The normalized attention weights among the time-series sampling points satisfy the following conditions: ; This parameter represents the center position and width of the deformable attention window.

[0083] It should be noted that after the spatiotemporal aggregation is completed, the risk token, which is the result of summarizing and interacting with the risk information from multiple time slices, will be sent to the risk inference module.

[0084] S150: Risk Causal Analysis and Segmentation: Based on the optimized global risk token sequence and mixed risk warning words, multi-step causal reasoning is performed to generate a high semantic risk analysis token and obtain a risk representation; based on the high semantic risk analysis token and risk representation, risk area location and segmentation are achieved through recursive interaction to complete the abnormal causal segmentation.

[0085] In some embodiments, step S150 (risk causal analysis and segmentation: based on the optimized global risk token sequence and mixed risk warning words, perform multi-step causal reasoning to generate a high semantic risk analysis token and obtain a risk representation; based on the high semantic risk analysis token and risk representation, achieve risk area location and segmentation through recursive interaction to complete abnormal causal segmentation) includes: It should be noted that, firstly, the risk warning token is parsed and integrated, and then used as a scenario guide to implement content-aware sparse routing for all expert large model branches. The risk token automatically activates the optimal or most matching expert head based on the content and structure, while the remaining experts remain frozen or activated infrequently, thereby greatly reducing the computational burden and improving inference efficiency.

[0086] It should be noted that in the multimodal perception system for complex scenarios of intelligent agents, a sparse expert selection and efficient fine-tuning method based on the MEA framework is adopted to realize soft expert routing driven by hybrid risk prompts and large language model optimization.

[0087] Specifically, based on the optimized global risk token sequence and hybrid risk warning words with causal guidance, causal graph reasoning and structured semantic analysis are performed through a large language model to mine the causal relationships between risk factors and generate structured risk representations. Then, this representation is used as a query input perceptual decoder to comprehensively analyze the structured risk representations under causal constraints. Under the guidance of multi-layer deformable cross-attention, the spatial localization, semantic segmentation, and causal contribution quantification of risk factors are achieved. Finally, the causal attribution mask and the corresponding causal contribution probability distribution are output to complete the abnormal causal segmentation.

[0088] Specifically, such as Figure 5 The diagram shows the MEA risk expert selection and fine-tuning mechanism: The integrated risk warning token is first distributed to the expert branch through gating routing. Each path is aggregated in a weighted or concatenated manner after parallel flow of FFN and Adapter-LORA to obtain the optimal risk analysis feature flow.

[0089] In theory, the MEA routing strategy is determined by the gating vector function. Decision, given the inclusion A collection of experts and input The system output is calculated as follows: ; in, For the gated vector function, the input... Mapped to Dimensional routing score; For the normalized route weight vector, by... Output Normalization yields the desired result, which satisfies the condition. ; For the first The output of each expert network is formed by parallel processing and aggregation of FFN and Adapter-LORA dual streams; it is the optimal risk analysis feature stream ultimately output by the MEA mechanism.

[0090] It should be noted that the routing supports efficient strategies such as top-k and gating masks, so that in actual inference, very few main branches (or even a single expert path) are sufficient to carry out the main risk modeling tasks.

[0091] It should be noted that within each activated expert branch, an innovative LORA fine-tuning structure is adopted, with the adapter and the native FFN (feedforward network) running side by side. The LORA-Adapter covers the backbone model with low-rank bottleneck parameters, requiring only a small number of parameters to achieve efficient adaptation to new risks, incremental categories, or scenario switching.

[0092] Using a hybrid risk warning word and an optimized global risk token sequence as joint input, a sparse gated hybrid expert model (MoE) is used for dynamic feature space mapping and modality adaptation, and a new approach is introduced. This represents the optimal risk analysis feature flow that is ultimately output by the MEA mechanism.

[0093] It should be noted that the routing supports efficient strategies such as top-k and gating masks, so that in actual inference, very few main branches (or even a single expert path) are sufficient to carry out the main risk modeling tasks.

[0094] It should be noted that within each activated expert branch, an innovative LORA fine-tuning structure is adopted, with the adapter and the native FFN (feedforward network) running side by side. The LORA-Adapter covers the backbone model with low-rank bottleneck parameters, requiring only a small number of parameters to achieve efficient adaptation to new risks, incremental categories, or scenario switching.

[0095] Using a hybrid risk warning word and an optimized global risk token sequence as joint input, a sparse gated hybrid expert model (MoE) is used for dynamic feature space mapping and modality adaptation, and a new approach is introduced. The structure undergoes feature enhancement processing, and risk semantics are extracted layer by layer by combining a multi-step causal reasoning mechanism to generate a high semantic risk analysis token, and risk representation is obtained based on the high semantic risk analysis token. The structure, specifically the expression, is as follows: ; in, It is the input feature, that is, the individual token in the optimized global risk token sequence. 3D feature vector; For the dimension reduction projection matrix, the features are transformed from... Compression of 3D space into a lower dimension dimension; The upscaling projection matrix transforms features from low dimension to high dimension. Dimension restored to its original dimension And satisfy , To ensure low-rank constraints, the Adapter is a lightweight module with very few parameters. Represents the set of real numbers. Representing all Action and The set of real matrices in the column.

[0096] Specifically, the high semantic risk analysis token and risk representation are aligned in terms of feature dimensions to obtain fusion input features with consistent dimensions. The high semantic risk analysis token contains preset semantic category labels for the root causes of anomalies. It should be noted that the preset semantic category labels for abnormal root causes are set as [bearing wear, broken gear teeth, lubricating oil failure, rotor imbalance] in industrial equipment fault diagnosis; and as [emergency braking of the vehicle in front, pedestrian suddenly crossing the road, foreign objects scattered on the road, traffic signal recognition error] in autonomous driving risk perception scenarios. These labels are based on knowledge bases or common analyses.

[0097] The fused input features are fed into a hierarchical query-aware decoder, which includes a spatial localization layer, a semantic association layer, and a causal determination layer. A high semantic risk analysis token is used as the query vector. The decoder interacts with the risk representation layer by layer through a multi-level deformable cross-attention mechanism corresponding to the three-layer structure of the decoder. The output includes a root cause region mask, an association region mask, and a corresponding risk mask and probability distribution to complete the abnormal causal segmentation. The specific process of interacting with the risk representation layer by layer through a multi-level deformable cross-attention mechanism corresponding to the three-layer structure of the decoder is as follows: The spatial positioning layer uses a deformable cross-attention mechanism to sample spatial features of the risk representation and outputs a preliminary segmentation mask for the risk region. The semantic association layer calculates the semantic similarity between the query vector and the corresponding region features of the preliminary segmentation mask, and selects candidate regions that match the semantic category label of the anomaly root cause based on the semantic similarity. The causal determination layer calculates the risk propagation gradient between regions based on the attention weights of candidate regions, and recursively determines the root cause region and related region in the candidate regions until the preset convergence condition is met.

[0098] The preset convergence conditions are convergence condition 1 and convergence condition 2. The iteration will terminate if either convergence condition is met. Specifically, convergence condition 1: convergence based on regional feature similarity. When the cosine similarity of the candidate region feature vectors output by two adjacent iterations is greater than or equal to the similarity threshold of 95%, when the similarity is less than 95%, the semantic matching degree of the candidate region is insufficient, and root cause region misjudgment is likely to occur; when the similarity reaches 95%, the semantic features of the candidate region have tended to be stable.

[0099] Convergence condition 2: Convergence based on risk propagation gradient. When the L2 norm of the risk propagation gradient calculated in two adjacent iterations is less than the gradient threshold, it is considered converged. Based on experience, the gradient threshold is 1e-3. When the gradient L2 norm is less than 1e-3, it indicates that the risk transmission relationship between regions has become stable. Continuing to iterate will not significantly improve the segmentation effect and will increase the computation time.

[0100] For example, the specific process is as follows: The first step is to construct query vectors: integrate the high semantic risk analysis token generated by LLM with the learnable positional encoding to form an initial query vector set Q, where each query vector corresponds to a risk factor to be detected; The second step is hierarchical decoding iteration: Construct an L-layer query-aware decoder, with each layer performing: deformable cross-attention: enabling the query vector to adaptively sample key regions of the risk representation, as shown in the formula: Query self-interaction: Modeling the dependencies between risk factors through multi-head self-attention; Asymptotic mask generation: Predicting coarse-to-fine masks based on the current query vector. Query self-interaction: Modeling the dependencies between risk factors through multi-head self-attention; Asymptotic mask generation: Predicting coarse-to-fine masks based on the current query vector. , which represents the probability that each risk factor is an abnormal cause.

[0101] like Figure 6 The diagram shows the hierarchical risk decoding and adaptive masking process of the query-aware decoder: First, the token sequence merged from the original resolution (from the multimodal backbone and the token simplification module) and the high semantic risk analysis token generated by the Large Language Model (LLM) are input into the query-aware decoder. Finally, the risk mask and risk probability distribution are output.

[0102] For example, in the first stage, the original size input is processed through deformable cross attention. Interact with queries and features to extract a preliminary risk perception representation: ; in, It is a sequence of risk analysis query tokens generated by the Large Language Model (LLM). For the number of queries, For feature dimensions; The first layer of merge-Token visual features, representing the original dimensions, are used as the key and value in the attention mechanism, respectively. The length of the visual feature sequence; This is the offset parameter in deformable attention, used to dynamically adjust the feature sampling position; The initial risk perception features output will be used to generate specific risk probabilities and mask responses. It represents the set of real numbers.

[0103] It should be noted that after obtaining the initial risk probability and mask response, the decoder output will be cyclically sampled and then subjected to deformable cross-attention inference with the risk token at their respective scales. By extracting features at different resolution scales, the system can capture multi-granular spatial patterns and achieve segmentation optimization from coarse to fine.

[0104] For the layer( 2) The features are first downsampled to reduce their resolution: ; Next, through deformable cross attention Interacting with the LLM query yields the risk perception features at the current scale: ; in, The total number of layers is used to obtain the segmentation mask at this scale after each operation. With risk score ; For the first layer Deformable offset parameters; features The data is then fed into two independent prediction heads, which generate segmentation masks for the current scale. With risk score ; ; in, For the first Segmentation mask prediction of layer output; For the first The risk score output by the layer (usually after) Normalized probability distribution.

[0105] It should be noted that the multi-layer recursive cross-attention mechanism ensures that the model can dynamically integrate high semantic risk knowledge of the global scene with local multimodal perception details. For example, at high resolution scales, it can accurately match and locate high-confidence anomalous regions to achieve spatial-level risk mask output; at medium and low resolution scales, it can utilize global and regional semantic consistency to optimize the probability distribution of risk category attribution and source tracing; different levels complement each other, embedding the upper-level mask and probability into the lower-level fusion path to accelerate convergence and improve segmentation / probability consistency.

[0106] This hierarchical complementarity mechanism is achieved by embedding upper-layer features into the lower-layer computation path: , ; in, For the first Layered risk perception feature representation; Upsampling operations, such as bilinear interpolation, are used to... The spatial resolution is increased to match the current layer; It should be noted that the upsampling operation can ensure the effective fusion of multi-scale information, so that the global semantic information of the low-resolution layer can guide the local detail segmentation of the high-resolution layer.

[0107] Finally, the segmentation masks and risk probability distributions at all levels are aggregated using attention: ; in, This represents the risk probability distribution. For spatial partitioning mask, Indicates the first After layer operations, the segmentation mask and risk score at this scale are obtained. Total number of floors; For the first Layer aggregation weights; For the concatenation operation, it indicates the first... Risk score of layer With segmentation mask They are concatenated into a single feature vector.

[0108] The formula is calculated using a learnable attention mechanism:

[0109] in, It is a learnable aggregation matrix used to model the relationship between features between layers; Total number of floors; For the first Layer aggregation weights; This represents the initial characteristics of risk perception. For the first Layered risk perception feature representation; This represents the summation of positive numerical similarity scores across all layers to achieve normalization. It is the set of real numbers.

[0110] It should be noted that high-probability risk factors are screened based on probability thresholds, and their corresponding masks are then weighted and fused to generate the final anomaly causal segmentation map. This completes the mapping from risk characterization to causal segmentation.

[0111] Specifically, the probability thresholds are adaptively determined based on the statistical characteristics of the risk probability distribution. The probability thresholds include the root cause determination threshold and the association determination threshold. Based on experience, the root cause determination threshold is set to 0.8, and the association determination threshold is set to 0.6.

[0112] Based on the risk probability distribution of each region output by the causal determination layer, an adaptive threshold algorithm is used to determine the probability threshold, which includes the root cause determination threshold and the association determination threshold. Based on the root cause determination threshold, risk factors with probability values ​​higher than the root cause determination threshold are selected from the risk probability distribution as root cause risk factors. The risk masks corresponding to each root cause risk factor are weighted and fused according to their risk probability values ​​to generate root cause region masks. Based on the association determination threshold, risk factors with probability values ​​higher than the association determination threshold and not belonging to root cause risk factors are selected from the risk probability distribution as associated risk factors. The risk masks corresponding to each associated risk factor are then weighted and fused according to their risk probability values ​​to generate an associated region mask. The root cause region mask and the associated region mask are superimposed to generate the final abnormal causal segmentation map, thus completing the abnormal causal segmentation.

[0113] It should be noted that, guided by hybrid risk warnings and scenario tokens, a sparse-gated hybrid expert (MEA) model enables dynamic expert triage and flexible interaction for risk events, coupled with efficient fine-tuning strategies such as domain-specific Adapter-LORA. Finally, through a hierarchical query-aware decoder, risk tokens generated by the original and large language models are integrated to recursively achieve adaptive generation of multi-resolution spatial risk masks and probability distributions. This effectively enables the agent to locate, interpret, and quantify high-confidence risks globally and locally, quantifying the risk level of each spatial location or semantic entity and providing direct basis for decision-making.

[0114] Based on the same inventive concept, embodiments of this application also provide an anomaly causal segmentation system that integrates multimodal perception and causal reasoning, including: The data acquisition module is used to acquire multi-source heterogeneous data, preprocess the multi-source heterogeneous data, and obtain standardized multimodal input data and the original token sequence corresponding to each modality. The hybrid risk warning word generation module is used to process static task templates and real-time dynamic context information to generate hybrid risk warning words; The adaptive token simplification module is used to adaptively filter the original token sequences corresponding to each modality to obtain a set of multimodal key information tokens. The cross-modal fusion processing module takes a set of multimodal key information tokens and mixed risk warning words as input, and performs feature interaction and weighting under the guidance of mixed risk warning words through a cross-modal linkage attention mechanism to obtain updated token-groups for each modality and aggregates the global risk token. The spatiotemporal modal adapter module is used to dynamically model the global risk token in the spatiotemporal dimension, and obtain an optimized global risk token sequence. The Risk Causal Analysis and Interpretation module is used to perform deep semantic analysis and causal reasoning on the optimized global risk token based on the Large Language Model (LLM) and combined with mixed risk warning words. Finally, it outputs the root cause region mask, the associated region mask and the corresponding risk probability distribution to complete the abnormal causal segmentation.

[0115] It should be noted that this system adopts a multi-level, multi-modal backbone model integration for the entire perception-decision chain. At the perception end, CLIP-Text is used to encode text such as rules and regulations, while CLIP-Image is used to uniformly encode point cloud and non-visual information such as speech, time series, and meteorological visualization data. All modal raw data undergoes rigorous frame synchronization, spatiotemporal interpolation, and multi-sensor spatial calibration to ensure high consistency across modal information. Text and semantic labels are parsed and uniformly encoded through an adaptive hierarchical risk warning engineering module. Upstream features, through semantic understanding and expert routing mechanisms centered on large models such as LLaMA2, are linked with hybrid risk warning words, token adaptive clustering simplification, and spatiotemporal modality adaptation to collaboratively promote unified alignment of multi-source information, aggregation of risk principals, and higher-order inference. All branches support modular plug-and-play functionality, and the core adaptive structure is highly decoupled from the main intervention training weights, laying the foundation for end-to-end fine-tuning and efficient multi-task adaptation of the system.

[0116] Specifically, the training process adopts a phased, progressive strategy. In the initial stage, all multimodal backbone encoders and alignment layer weights are frozen, and only mid-to-high-level parameters such as risk indication, token simplification, spatiotemporal modality adapter, MEA expert selection, and query decoder are trained to ensure the model's basic transfer adaptability across multimodal scenarios and task data. After the aforementioned core modules converge, some backbone models (such as some mid-to-high-level visual / point cloud components, CLIP-Adapter, and large model adapter) are gradually unfrozen. Deep joint alignment of the multimodal feature system and downstream causal inference links is implemented, and the freezing and unfreezing ratio is dynamically adjusted based on validation set performance to achieve a fine balance between feature inheritance and scenario adaptation. In the final stage, end-to-end fine-tuning of local structures such as risk mask segmentation and causal link inference is focused to improve decision accuracy and interpretability robustness in extreme multi-agent environments. The entire process employs synchronous multi-card processing, mixed precision, and dynamic loss weighting. Input sample and label augmentation covers mosaic, cutmix, random occlusion, and simulation generation, providing assurance for modeling complex real-world scenarios and challenging extreme samples.

[0117] For example, the model optimizer uses AdamW, with an initial learning rate of 2e-4 during the backbone freezing phase and 1e-4 during the end-to-end joint fine-tuning phase, β1=0.9, β2=0.999, and weight-decay=0.01. The total number of training batches is set to 500 rounds on the public dataset, 300 rounds each on the large model synthetic data and simulation dataset, and the termination round is dynamically adjusted through an early-stopping mechanism based on the validation set risk interpretation and mask IoU score. The batch size per card is set to 16 to fully align multimodal inputs and improve computational and convergence efficiency. The multi-task loss function includes the cross-entropy of the spatial segmentation mask, the KL divergence loss of the global risk probability distribution, and the sparse gating loss of the expert route, which are automatically weighted according to the weight of each task. The learning rate scheduling adopts a strategy combining Cosine Annealing and cyclic warm-up. The entire process is carried out in a distributed mixed-precision training on NVIDIA-A100-80GB×4 servers, and the deep learning platform is PyTorch-2.0, to ensure the efficiency and reproducibility of the system training process.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An abnormal causal segmentation method integrating multimodal perception and causal reasoning, characterized in that, include: S1. Multi-source heterogeneous data acquisition and preprocessing: Acquire multi-source heterogeneous data, perform standardization and modal alignment preprocessing on the multi-source heterogeneous data, and input the preprocessed modal data into the corresponding modal encoder for encoding to generate the original token sequence of each modality. S2. Hybrid Risk Warning Term Design: Generate hybrid risk warning terms based on static slots and dynamic slots; S3. Adaptive Token Simplification: Adaptively filter and simplify the original token sequences of each modality to obtain a multimodal key information token set; S4. Cross-modal fusion processing: The multimodal key information token set and the mixed risk warning words are fused to generate a global risk token representing the overall risk status. S5. Spatiotemporal modality adaptation processing: The spatiotemporal dependency relationship of the global risk token is modeled and optimized to obtain the optimized global risk token sequence. S6. Risk Causal Analysis and Segmentation: Based on the optimized global risk token sequence and the mixed risk warning words, perform multi-step causal reasoning to generate a high semantic risk analysis token and obtain a risk representation; Based on the high semantic risk analysis token and the risk representation, risk area location and segmentation are achieved through recursive interaction to complete the abnormal causal segmentation.

2. The abnormal causal segmentation method integrating multimodal perception and causal reasoning according to claim 1, characterized in that, The preprocessing of the multi-source heterogeneous data, including standardization and modal alignment, followed by inputting the preprocessed modal data into the corresponding modal encoder for encoding to generate the original token sequence for each modality, includes: The multi-source heterogeneous data includes text data, visual data, radar data, audio data, and meteorological data. Differentiated standardization and cross-modal coarse alignment are performed according to the characteristics of each modality of data to obtain standardized multi-source data. Features of each modality are extracted from the standardized multi-source data. The features of each modality are input into a spatiotemporal correlation scene modeling network. The semantic correlation between the features of each modality in the time and space dimensions is constructed through the network to generate a feature-level scene token that integrates global scene information. The feature-level scene token and the standardized multi-source data are aligned in dimension through linear mapping, and then the features are spliced ​​to obtain multimodal data that integrates scene information. The CLIP multimodal model is used to perform unified semantic modeling on the multimodal data of the fused scene information, and output modality-specific vector representations that correspond to text, vision, radar, audio, and meteorology respectively. The modality-specific vector representations aligned across modalities are input into the corresponding modality-specific encoders according to modality categories to generate the original token sequences for each modality.

3. The abnormal causal segmentation method integrating multimodal perception and causal reasoning according to claim 1, characterized in that, The adaptive filtering and simplification of the original token sequences for each modality yields a multimodal key information token set, including: Based on a deformable attention-guided multimodal input token dynamic grouping mechanism, the original token sequences of each modality are adaptively grouped to obtain several token subsets within each modality; The K-nearest neighbor algorithm is deeply integrated with the embedding space similarity calculation. Within the embedding space, features are aggregated for several token subsets in each modality. During the aggregation process, the spatial distance and semantic similarity of the token embedding vectors are taken into account to form several token clusters. A dual adaptive adjustment mechanism is implemented for each of the token clusters to simultaneously achieve dynamic adjustment of the token group size and dynamic adaptation of the anomaly detection density; Based on the results of the aforementioned dual adaptive adjustment mechanism, for the first... The nearest neighbor groups corresponding to each anchor point are weighted and summed using normalized similarity weights to generate a simplified token group corresponding to each anchor point. The specific calculation formula is as follows: ； in, Indicates the first Each anchor point corresponds to a simplified token group; For the first The nearest neighbors of each query vector Grouped index set; For the first Value branch characteristics corresponding to each Token; For the first The anchor point and the first The gating weight of each token; the set of the simplified token groups corresponding to all anchor points constitutes the multimodal key information token set.

4. The abnormal causal segmentation method integrating multimodal perception and causal reasoning according to claim 1, characterized in that, The process of fusing the multimodal key information token set with the hybrid risk warning words to generate a global risk token representing the overall risk status includes: A deformable attention mechanism is applied to the token group of each modality in the multimodal key information token set to extract and enhance features within the group, thereby obtaining enhanced token groups for each modality. Using each modal enhancement token group as a query and the hybrid risk warning word as a key and value, the association weight between the hybrid risk warning word and each modal enhancement token group is calculated through a deformable cross-attention mechanism to obtain a cross-modal interaction token group that integrates risk warning information. A deformable attention mechanism is applied to the cross-modal interaction token group, focusing on the risk feature with the highest weight, to obtain the updated token groups for each modality; The updated modal token groups are then combined to generate a global risk token that represents the overall risk status.

5. The abnormal causal segmentation method integrating multimodal perception and causal reasoning according to claim 4, characterized in that, The specific calculation formula for the deformable cross-attention mechanism is as follows: ； in, Indicates the first Feature vector of each query token; Indicates the first A set of sampling point indexes corresponding to each query token; Indicates the first Initial reference coordinates for each sampling point; Indicates the first The learnable offset of each sampling point; Indicated in the Value feature map Above, in the coordinate system, bilinear interpolation is used. The feature vector obtained by sampling at point; This is a cross-attention calculation function with deformable offset, and its output is a query. The weighted sum of the Value features at a series of corresponding deformed sampling points; Attention weights; The formula for calculating the attention weight is: ； in, The feature vector representing the mixed risk warning; For feature dimensions; Indicated in the Key feature map Above, in the coordinate system, bilinear interpolation is used. The feature vector obtained by sampling at point; Indicated in the Key feature map Above, in the coordinate system, bilinear interpolation is used. The feature vector obtained by sampling at point, where It is a summation index that iterates through all sample points; It is the transpose function; It is an exponential function.

6. The abnormal causal segmentation method integrating multimodal perception and causal reasoning according to claim 1, characterized in that, The process of modeling and optimizing the spatiotemporal dependency relationship of the global risk token to obtain an optimized global risk token sequence includes: The global risk tokens are arranged chronologically according to the initial global risk token sequence to generate an initial global risk token sequence. The feature values ​​of the initial global risk token sequence are denoted as follows: ; Building a spatiotemporal modal adapter The spatiotemporal modality adapter fuses depthwise separable 3D convolution and deformable temporal attention mechanisms, and its specific calculation formula is as follows: ； in, Represents the features after dimensionality reduction Perform depthwise separable 3D convolution operations; This is the downsampling weight matrix; For upsampling weight matrix; As a deformable temporal attention mechanism, for The output features are used for temporal dependency modeling; For spatiotemporal modal adapters; The deformable temporal attention mechanism is used for... The output features are subjected to temporal dependency weighting and feature enhancement. The initial global risk token sequence is input into the spatiotemporal modality adapter to perform spatiotemporal dependency modeling and feature optimization, resulting in an optimized global risk token sequence.

7. The abnormal causal segmentation method integrating multimodal perception and causal reasoning according to claim 1, characterized in that, The process of performing multi-step causal reasoning based on the optimized global risk token sequence and the hybrid risk warning words to generate a high-semantic risk analysis token and obtain a risk representation includes: Using the hybrid risk warning words and the optimized global risk token sequence as joint inputs, a sparse gated hybrid expert model (MoE) is used for dynamic feature space mapping and modality adaptation, and a new approach is introduced. The structure undergoes feature enhancement processing, and risk semantics are extracted layer by layer by combining a multi-step causal reasoning mechanism to generate a high semantic risk analysis token, and risk representation is obtained based on the high semantic risk analysis token. The The structure, specifically the expression, is as follows: ； in, For individual tokens in the optimized global risk token sequence 3D feature vector; For the dimension reduction projection matrix, the features are transformed from... Compression of 3D space into a lower dimension dimension; To increase the dimension of the projection matrix, the features are transformed from low dimension. Dimension restored to its original dimension And satisfy ; It represents the set of real numbers.

8. The abnormal causal segmentation method integrating multimodal perception and causal reasoning according to claim 1, characterized in that, The step of locating and segmenting risk regions through recursive interaction based on the high semantic risk analysis token and the risk representation to complete the anomaly causal segmentation includes: The high semantic risk analysis token and the risk representation are aligned in feature dimensions to obtain fusion input features with consistent dimensions; the high semantic risk analysis token contains preset anomaly root cause semantic category labels; The fused input features are input into a hierarchical query-aware decoder, which includes a spatial localization layer, a semantic association layer, and a causal determination layer. The high semantic risk analysis token is used as the query vector. The decoder interacts with the risk representation layer by layer through a multi-level deformable cross-attention mechanism corresponding to the three-layer structure of the decoder. The output includes a root cause region mask, an association region mask, and a corresponding risk mask and probability distribution to complete the abnormal causal segmentation. The specific process of interacting with the risk representation layer by layer through a multi-level deformable cross-attention mechanism corresponding to the three-layer structure of the decoder is as follows: The spatial positioning layer performs spatial feature sampling on the risk representation through a deformable cross-attention mechanism and outputs a preliminary segmentation mask for the risk region. The semantic association layer calculates the semantic similarity between the query vector and the region features corresponding to the preliminary segmentation mask, and filters out candidate regions that match the semantic category label of the anomaly root cause based on the semantic similarity. The causal determination layer calculates the risk propagation gradient between regions based on the attention weights of the candidate regions, and recursively determines the root cause region and related region in the candidate regions until a preset convergence condition is met.

9. The abnormal causal segmentation method integrating multimodal perception and causal reasoning according to claim 8, characterized in that, The output includes a root cause region mask, an associated region mask, and corresponding risk masks and probability distributions. The process of completing the anomaly causal segmentation specifically includes: Based on the risk probability distribution of each region output by the causal determination layer, an adaptive threshold algorithm is used to determine the probability threshold, which includes the root cause determination threshold and the association determination threshold. Based on the root cause determination threshold, risk factors with a probability value higher than the root cause determination threshold are selected from the risk probability distribution as root cause risk factors, and the risk masks corresponding to each root cause risk factor are weighted and fused according to their risk probability values ​​to generate a root cause region mask. Based on the association determination threshold, risk factors with a probability value higher than the association determination threshold and not belonging to the root cause risk factors are selected from the risk probability distribution as associated risk factors, and the risk masks corresponding to each associated risk factor are weighted and fused according to their risk probability values ​​to generate an associated region mask. The root cause region mask and the associated region mask are superimposed to generate the final abnormal causal segmentation map, thereby completing the abnormal causal segmentation.

10. An abnormal causal segmentation system integrating multimodal perception and causal reasoning, characterized in that, include: The data acquisition module is used to acquire multi-source heterogeneous data, preprocess the multi-source heterogeneous data, and obtain standardized multimodal input data and the original token sequence corresponding to each modality. The hybrid risk warning word generation module is used to process static task templates and real-time dynamic context information to generate hybrid risk warning words; The adaptive token simplification module is used to adaptively filter the original token sequences corresponding to each modality to obtain a multimodal key information token set. The cross-modal fusion processing module is used to take the multimodal key information token set and the mixed risk warning words as input, and through the cross-modal linkage attention mechanism, under the guidance of the mixed risk warning words, perform fusion processing on the multimodal key information token set to obtain the updated token-group of each modality, and aggregate the global risk token. The spatiotemporal modal adapter module is used to dynamically model the global risk token in the spatiotemporal dimension to obtain an optimized global risk token sequence. The risk causal analysis module is used to perform deep semantic analysis and causal reasoning on the optimized global risk token sequence based on the Large Language Model (LLM) and the mixed risk warning words. Finally, it outputs the root cause region mask, the associated region mask and the corresponding risk probability distribution to complete the abnormal causal segmentation.

Images