Crisis early warning system and method based on multi-modal data fusion

Abstract

The application discloses a crisis early warning system and method based on multi-modal data fusion, and belongs to the technical field of crisis early warning. The method specifically comprises the following steps: collecting multi-modal data of a target individual, including physiological signal data, behavior data and semantic data; comparing the multi-modal data with preset group baseline data; detecting conflict signals among different modes; determining dominant mode characteristics through counterfactual simulation; constructing a psychological state evolution model based on the dominant mode characteristics; reasoning a potential psychological crisis trigger point; when the psychological crisis trigger point is identified, generating early warning information associated with an actual life situation, and outputting corresponding intervention suggestions. The application can overcome the limitations of existing single mode, static indicators and correlation judgment, improve the identification accuracy and timeliness of psychological crisis, and enhance the accuracy of early warning results.

Description

Technical Field

[0001] This invention belongs to the field of crisis early warning technology, specifically a crisis early warning system and method based on multimodal data fusion. Background Technology

[0002] Existing technologies for detecting psychological crises include: detection methods based on single physiological signals, which quantify psychological stress using physiological parameters such as heart rate, electroencephalography (EEG), and skin conductance; emotion analysis methods based on text and voice, which determine psychological tendencies through social media text, voice tone, and word usage habits; and monitoring methods based on behavioral patterns, which infer psychological states by analyzing users' daily behavioral data, such as movement trajectories, sleep cycles, and communication frequency.

[0003] In recent years, multimodal learning and data fusion technologies have brought new possibilities to psychological crisis early warning. By integrating heterogeneous data from multiple sources such as physiological, behavioral, semantic, speech, and image data, it is possible to understand an individual's psychological state more comprehensively from different dimensions. However, the following problems still exist: how to resolve conflicts in multimodal data, model causal chains, perform temporal dynamic reasoning, and how to combine early warning information with real-life situations. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a crisis early warning system and method based on multimodal data fusion. This system can achieve cross-modal data alignment and conflict resolution, and by combining temporal reasoning and causal modeling, it outputs early warning information that matches real-life scenarios, thereby improving the accuracy and practicality of psychological crisis prediction.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Crisis early warning methods based on multimodal data fusion include:

[0007] Collect multimodal data of the target individual, including physiological signal data, behavioral data, and semantic data;

[0008] The multimodal data is compared with preset group baseline data to detect conflict signals between different modes, and the dominant mode characteristics are determined through counterfactual simulation.

[0009] Based on the dominant modality features, a psychological state evolution model is constructed to infer potential psychological crisis triggers.

[0010] When a psychological crisis trigger is identified, early warning information related to the actual life situation is generated, and corresponding intervention suggestions are output.

[0011] Specifically, the multimodal data is compared with preset group baseline data to detect conflict signals between different modalities, and the dominant modality characteristics are determined through counterfactual simulation, including:

[0012] The multimodal data of the target individuals are mapped to a unified time axis, and a multimodal control matrix is ​​established based on the preset population baseline data.

[0013] In the multimodal comparison matrix, the difference signals between different modes are identified. When the degree of difference exceeds a threshold, the corresponding modal signal is marked as a conflict signal.

[0014] For the conflict signal, a corresponding counterfactual sample is generated, which is used to simulate the psychological state inference result when the modality signal is missing;

[0015] By comparing the inference results of the counterfactual samples with the actual multimodal inference results, the dominant mode is determined based on the magnitude of the deviation, and the features of the dominant mode are output.

[0016] Specifically, the inference results of the counterfactual samples are compared with the actual multimodal inference results. The dominant mode is determined based on the magnitude of the deviation, and the features of the dominant mode are output, including:

[0017] The difference between the inference results of the counterfactual samples and the actual multimodal inference results is measured to obtain the deviation value corresponding to each mode;

[0018] A deviation sequence is constructed based on the distribution of the deviation values, and stable deviation intervals and abnormal deviation peaks are identified in the deviation sequence.

[0019] Using the stable deviation range as a reference, the mode corresponding to the abnormal deviation peak is marked as the candidate dominant mode;

[0020] Cross-comparison is performed among candidate dominant modes. If there is an alternation of bias between modes, the mode that maintains the best performance within a preset time window is selected as the final dominant mode, and the corresponding features are output.

[0021] Specifically, the construction of a psychological state evolution model based on dominant modality features to infer potential psychological crisis triggers includes:

[0022] The dominant modal features are serialized in chronological order to establish a temporal chain of mental states;

[0023] A multi-scale segmentation window is introduced on the time series chain to extract the first fluctuation feature and the second evolution feature respectively;

[0024] The first fluctuation feature and the second evolution feature are used to perform interactive reasoning to generate a candidate set of psychological state inflection points;

[0025] The candidate set is reorganized into causal chains to screen out turning points with a tendency to trigger continuously, which can be used as potential psychological crisis triggers.

[0026] Specifically, the step of serializing the dominant modality features in chronological order to establish a temporal chain of mental states includes:

[0027] The dominant modal features are initially sorted according to the acquisition timestamp, and redundant features with overlapping or missing segments are removed.

[0028] Temporal hierarchical labels are attached to the sorted dominant modality feature sequence to divide the features into micro-instantaneous segments, daily cycle segments, and cross-cycle segments.

[0029] Based on the time layer labels, the connection relationship between different segments is reconstructed to generate a time recursive chain;

[0030] Transition nodes across segments are identified in the time recursion chain, and these transition nodes are used as key anchor points for the mental state time sequence chain, thus constructing the mental state time sequence chain.

[0031] Specifically, the first fluctuation feature and the second evolutionary feature are used for interactive reasoning to generate a candidate set of psychological state inflection points, including:

[0032] The first fluctuation feature and the second evolution feature are projected onto a unified time correlation plane to establish a mapping relationship between them.

[0033] On the time-related plane, identify the coupled segments of the first wave feature and the second evolution feature, and record the start and end nodes of their interaction;

[0034] The coupled segments are arranged sequentially to generate interaction chains, and the node with the highest frequency of occurrence is marked in each interaction chain;

[0035] All the most frequent nodes are aggregated into a candidate set, which is then used as a candidate set for psychological turning points.

[0036] Specifically, the candidate set is reorganized into causal chains to screen out turning points with a tendency to trigger continuously, as potential trigger points for psychological crises, including:

[0037] Arrange the nodes in the candidate set of psychological state turning points according to time order and causal dependency to generate an initial causal chain, and mark the causal strength between nodes in the chain;

[0038] In the initial causal chain, identify the break points where the causal strength is below a threshold, and introduce virtual transition nodes or alternative nodes to complete the causal break points;

[0039] The completed causal chain is traversed multiple times to check the stability of the causal path within each consecutive time window, and the causal sub-chains that repeat in multiple windows are recorded.

[0040] The core nodes in the repeated causal subchains are extracted as candidate key nodes, and cross-comparison is performed between different subchains to remove nodes that do not have cross-chain coherence.

[0041] The selected set of key nodes is marked as potential trigger points for psychological crises.

[0042] Specifically, when a psychological crisis trigger point is identified, generating early warning information related to the actual life situation includes:

[0043] The psychological crisis trigger points are associated with the target individual's environmental data, which includes time, geographical location, and social scene information.

[0044] Based on the environmental data, a situation mapping table is constructed, and psychological crisis trigger points are projected to three situational dimensions: daily life, work interaction, and social relationships.

[0045] The situational fragment coupled with the psychological crisis trigger point is identified in the situational mapping table, and an event chain is generated for the situational fragment. The event chain consists of triggering conditions, behaviors, and extended scenarios.

[0046] The event chain is transformed into an early warning information template, and a set of corresponding intervention instructions is embedded in the early warning information template to form early warning information that is related to real-life situations.

[0047] A crisis early warning system based on multimodal data fusion is used to implement the aforementioned crisis early warning method based on multimodal data fusion, including: a data acquisition module, a modal feature determination module, an inference module, and an early warning module;

[0048] The data acquisition module is used to collect multimodal data of the target individual, including physiological signal data, behavioral data, and semantic data;

[0049] The modality feature determination module is used to compare the multimodal data with preset group baseline data, detect conflict signals between different modalities, and determine the dominant modality feature through counterfactual simulation.

[0050] The reasoning module constructs a psychological state evolution model based on the dominant modality features to infer potential psychological crisis triggers.

[0051] The early warning module is used to generate early warning information related to the actual life situation and output corresponding intervention suggestions when a psychological crisis trigger point is identified.

[0052] Specifically, the reasoning module includes: a feature extraction unit, a reasoning unit, and a recombination and filtering unit;

[0053] The feature extraction unit extracts a first fluctuation feature and a second evolution feature based on the dominant mode feature;

[0054] The reasoning unit is used to perform interactive reasoning between the first fluctuation feature and the second evolution feature to generate a candidate set of psychological state inflection points.

[0055] The reorganization and screening unit is used to reorganize the causal chain of the candidate set and screen out turning points with continuous triggering tendency as potential psychological crisis trigger points.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] This invention proposes a crisis early warning system and method based on multimodal data fusion. By combining time series modeling, cross-modal conflict resolution, and causal chain reasoning, it establishes a dynamic psychological state evolution model oriented towards individuals. After identifying potential crisis trigger points, it generates multi-level early warning information associated with real-life scenarios, realizing a complete closed loop from data acquisition and state reasoning to scenario-based output. This method can overcome the limitations of existing single-modality, static indicator, and correlation judgment methods, improve the accuracy and timeliness of psychological crisis identification, enhance the precision of early warning results, and improve the pertinence and application value of early warning information by combining it with individual real-life situations. Attached Figure Description

[0058] Figure 1 A flowchart of the crisis early warning method based on multimodal data fusion provided by the present invention;

[0059] Figure 2 The flowchart for generating psychological crisis trigger points provided by this invention;

[0060] Figure 3 A flowchart for generating a candidate set of psychological state inflection points provided by the present invention;

[0061] Figure 4 This is a diagram illustrating the architecture of a crisis early warning system based on multimodal data fusion, as provided by the present invention. Detailed Implementation

[0062] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0065] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0066] Example 1

[0067] Please see Figures 1-3 The present invention provides an embodiment of a crisis early warning method based on multimodal data fusion, comprising the following specific steps:

[0068] Step S1: Collect multimodal data of the target individual, including physiological signal data, behavioral data, and semantic data.

[0069] In this embodiment, multimodal data of the target individual is acquired through a multi-source acquisition device. Physiological signal data is recorded in real time by wearable sensors, including indicators such as heart rate variability, skin conductance, and body movement changes. Behavioral data is monitored through mobile terminals and environmental sensing devices, including the individual's travel trajectory, frequency of social interaction, and sleep and activity cycles. Semantic data is extracted by natural language parsing of voice and text inputs to reveal emotional tendencies and cognitive expressions. It should be noted that during the acquisition process, data from different modalities are all timestamped and are time-series aligned and outlier removed through a preprocessing module, thus forming a structured multimodal data stream. The principle of this step is to utilize the complementary responses of different modalities to psychological states: physiological signals reveal the body's stress level, behavioral patterns characterize daily habit deviations, and semantic information reflects internal cognition and emotional changes.

[0070] Step S2: Compare the multimodal data with the preset population baseline data, detect conflict signals between different modes, and determine the dominant mode characteristics through counterfactual simulation.

[0071] The specific steps of step S2 are as follows:

[0072] Step S201: Map the multimodal data of the target individuals to a unified time axis, and establish a multimodal control matrix based on the preset population baseline data.

[0073] In this embodiment, to avoid alignment deviations caused by differences in sampling intervals between different modalities, interpolation and segmented aggregation are used to smooth downsample high-frequency data and appropriately expand low-frequency data to achieve temporal consistency. Subsequently, a preset population baseline data is introduced as a reference. This baseline data is constructed by collecting the average physiological characteristics, typical behavioral patterns, and common semantic expressions of similar groups over a long period of time. A matrix correspondence is established between the aligned target individual data and the population baseline, so that the individual's performance in each modality within the same time period is compared with the population reference value, thereby obtaining a multimodal comparison matrix.

[0074] Step S202: Identify the difference signals between different modes in the multimodal comparison matrix. When the degree of difference exceeds a threshold, mark the corresponding mode signal as a conflict signal.

[0075] In this embodiment, the differences between adjacent time segments are smoothed and a difference sequence is generated in the time dimension. Then, a threshold judgment rule is set for the difference sequence. When the difference between a certain mode and other modes in the same time segment continues to exceed the threshold, the mode signal is identified as inconsistent with the overall performance and is marked as a conflict signal. By identifying abnormal offsets through cross-modal comparison, single-mode distortion caused by sensor fluctuations, environmental noise or individual short-term behavioral abnormalities can be eliminated, and finally a set of marked conflict signals is obtained.

[0076] Step S203: For the conflict signal, generate a corresponding counterfactual sample, which is used to simulate the psychological state inference result when the modality signal is missing.

[0077] In this embodiment, after identifying the conflict signal, the data corresponding to that modality is temporarily removed from the multimodal comparison matrix, and an alternative inference path is constructed based on the data of the remaining modalities to generate counterfactual samples. Specifically, firstly, the non-conflicting modal data is subjected to feature expansion and temporal filling to make up for the information gaps in the time chain of the removed modality; secondly, the cross-modal correlation mapping model is used to derive possible psychological state estimates at the missing modality positions and generate a contrast output that is different from that under complete data; finally, the counterfactual sample is recorded as the psychological state inference result of the simulated individual under the condition of lacking the input of the modality signal.

[0078] Step S204: Compare the inference results of the counterfactual sample with the actual multimodal inference results, determine the dominant mode based on the magnitude of the deviation, and output the dominant mode features.

[0079] The specific steps of step S204 are as follows:

[0080] Step S2041: The difference between the inference result of the counterfactual sample and the actual multimodal inference result is measured to obtain the deviation value corresponding to each modality.

[0081] In this embodiment, a time sliding window approach is used to accumulate and smooth the difference values, thereby obtaining a continuous difference curve. Subsequently, the difference curves of each modality under counterfactual and complete conditions are normalized, and its characteristic indicators at different time periods are extracted, ultimately forming a sequence of deviation values ​​corresponding to each modality. By simulating the difference between the result of removing a specific modality and the complete inference result, the independent contribution and sensitivity of the modality in the overall psychological state inference can be quantified, providing measurable reference data for subsequent screening of dominant modalities.

[0082] Step S2042: Construct a deviation sequence according to the distribution of the deviation values, and identify stable deviation intervals and abnormal deviation peaks in the deviation sequence.

[0083] In this embodiment, the modal deviation values ​​obtained in step S2041 are arranged in chronological order to form a continuous deviation sequence, reflecting the relative stability of the mode in different time segments. Subsequently, the deviation sequence is divided into intervals and statistically analyzed. The mean and fluctuation amplitude of the deviation values ​​within a local range are calculated using a sliding window, and stable deviation intervals that remain within a small range over a long period are identified. At the same time, local extreme points that exceed a set threshold and show a sharp increase are marked and defined as abnormal deviation peaks. Through this method, the distribution of stable intervals and abnormal peaks are obtained simultaneously in a deviation sequence, providing clear dynamic difference characteristics for subsequent determination of candidate dominant modes. By comparing the deviation change patterns over continuous time periods, the stable performance of the mode under normal conditions and the extreme shifts of sudden abnormalities can be distinguished.

[0084] Step S2043: Using the stable deviation range as a reference, mark the mode corresponding to the abnormal deviation peak as the candidate dominant mode.

[0085] In this embodiment, after obtaining the deviation sequence of each mode, the identified stable deviation interval is first used as a reference baseline to represent the typical fluctuation range of the mode under normal conditions. Subsequently, the deviation points located outside the reference baseline and marked as abnormal peaks are aggregated and analyzed to determine whether they have a persistent or concentrated distribution. When a mode continuously exhibits obvious abnormal peaks in multiple time windows, and the difference between these peaks and the reference interval remains consistent, the mode is marked as a candidate dominant mode. By comparing the relative relationship between the stable interval and the abnormal peaks, occasional noise fluctuations and representative key shifts can be effectively distinguished, and modes with stronger influence in psychological state inference can be identified.

[0086] Step S2044: Further cross-comparison is performed among the candidate dominant modes. If there is an alternation of bias between modes, the mode that maintains the best performance within the preset time window is selected as the final dominant mode, and the corresponding features are output.

[0087] In this embodiment, after obtaining multiple candidate dominant modes, the deviation sequences of these modes are first cross-compared on the same time axis to identify whether different modes alternately show dominance in adjacent time windows. Specifically, the distribution patterns of the mean deviation and abnormal peak values ​​of each candidate mode are statistically analyzed within a preset time window. If a mode only performs well in a local window but tends to weaken in other windows, it is determined that its stability is insufficient. Furthermore, when the phenomenon of alternating deviations is found, the mode that maintains dominance in most sub-windows is selected as the final dominant mode by comparing the comprehensive performance of each mode in the entire preset window range, and its corresponding feature vector is extracted. Through cross-comparison and windowed statistics, erroneous selection caused by short-term fluctuations can be avoided, ensuring that the output final dominant mode has consistency and representativeness across time periods.

[0088] In this embodiment,

[0089] Step S3: Construct a psychological state evolution model based on the dominant modality features and infer potential psychological crisis triggers.

[0090] The specific steps of step S3 are as follows:

[0091] Step S301: Sequentialize the dominant modality features in chronological order to establish a temporal chain of mental states.

[0092] like Figure 2 As shown, the specific steps of step S301 are as follows:

[0093] Step S3011: The dominant modal features are initially sorted according to the acquisition timestamp, and redundant features with overlapping or missing segments are removed.

[0094] In this embodiment, the dominant modal features are arranged sequentially according to the timestamps attached to the acquisition end to ensure that all data points are within a unified time series framework. Specifically, for features with overlapping timestamps, the most representative data is retained and redundant parts are removed by comparing the completeness of the features, the reliability of the sampling source, or the signal strength. For segments with gaps or missing timestamps, interpolation or marking of gap positions is performed using the continuity information of the preceding and following time periods to avoid breakage of the time series. Through sorting and redundancy cleanup, the time misalignment caused by equipment synchronization errors or data loss during the original acquisition process can be eliminated, thereby obtaining a continuous and conflict-free time-series feature sequence.

[0095] Step S3012: Add time-level labels to the sorted dominant modality feature sequence to divide the features into micro-instantaneous segments, daily cycle segments, and cross-cycle segments.

[0096] In this embodiment, after the dominant modality feature sequence is sorted by time, time-layer labels are assigned based on the time span and repetition pattern of the features. Specifically, firstly, features with extremely short durations and related to immediate reactions are identified and labeled as micro-instantaneous segments; secondly, features exhibiting periodic fluctuations in diurnal patterns or daily activities are labeled as daily cycle segments; finally, features covering a longer period, spanning multiple cycles, and reflecting long-term psychological trends are labeled as cross-cycle segments.

[0097] Step S3013: Reconstruct the connection relationship between different segments based on the time layer labels to generate a time recursive chain.

[0098] In this embodiment, after obtaining the dominant modal feature sequence with time-layered labels, the adjacent boundary points between micro-instantaneous segments, daily periodic segments, and cross-period segments are first identified according to the label type. Then, connection rules between segments are established at these boundary points. For example, instantaneous segments are used as trigger nodes to connect to periodic segments within the same time period, and periodic segments are mapped to cross-period segments covering a longer time span. Furthermore, these connection rules are iteratively recursively applied to embed short-term segments into daily periodic chains, and daily periodic chains are then embedded into long-term cross-period chains, ultimately forming a recursive chain structure containing different time scales, i.e., a time recursive chain.

[0099] Step S3014: Identify the transition nodes across segments in the time recursion chain, and use the transition nodes as key anchor points of the mental state time sequence chain, and construct the mental state time sequence chain.

[0100] In this embodiment, in the constructed time recursion chain, the intersection points of different time level segments are first identified, such as the switching positions between instantaneous segments and daily cycle segments, or the connection positions where daily cycle segments extend to cross-cycle segments, and these intersection points are marked as transition nodes. Subsequently, the transition nodes are classified and organized, and nodes that recur or have a continuous effect in multiple time windows are extracted to determine their key position in the time sequence chain. Furthermore, using these key nodes as anchor points, the original time recursion chain is reorganized into a hierarchical and coherent psychological state time sequence chain, so that the chain retains the detailed information between segments and highlights the cross-level connection logic. Through the identification and anchoring of transition nodes, key support points can be extracted from complex time recursion relationships to form a psychological state time sequence chain.

[0101] Step S302: Introduce a multi-scale segmentation window on the time series chain to extract the first fluctuation feature and the second evolution feature respectively.

[0102] In this embodiment, a multi-scale segmentation window is set on the constructed psychological state time series chain to segment and observe the same chain at different time granularities. Specifically, the chain is first segmented with a smaller time window to capture the rapid fluctuation signals of individuals in an instant or short period of time and extract the first fluctuation feature related to the short-term fluctuations of psychological state. Subsequently, the chain is segmented with a larger time window to track the continuous change trend across days, weeks, and even longer periods and extract the second evolutionary feature reflecting the long-term evolution of psychological state. Furthermore, by mapping the short-term fluctuation feature and the long-term evolutionary feature on a unified time chain, the fine-grained dynamics and macro-level extension patterns of psychological state can be preserved at the same time.

[0103] It should be noted that the first fluctuation characteristic is a short-term fluctuation characteristic, and the second evolutionary characteristic is a long-term evolutionary characteristic.

[0104] Step S303: Perform interactive reasoning between the first fluctuation feature and the second evolution feature to generate a candidate set of psychological state inflection points.

[0105] like Figure 3 As shown, the specific steps of step S303 are as follows:

[0106] Step S3031: Project the first fluctuation feature and the second evolution feature onto a unified time correlation plane to establish a mapping relationship between them.

[0107] In this embodiment, the first fluctuation feature and the second evolution feature extracted from different time scale windows are aligned according to their respective timestamps, and the two are mapped to a unified time correlation plane through interpolation and time normalization methods. Specifically, on this plane, with time as the main axis, the fine-grained data of short-term fluctuation features are arranged to correspond with the macro-trend of long-term evolution features, so that each short-term feature point can find its corresponding position in its long-term context. Subsequently, the mapping relationship between the two is established in this correlation plane, and by marking the intersection points and corresponding segments, a dynamic connection map between short-term fluctuations and long-term evolution is formed.

[0108] Step S3032: Identify the coupling segments of the first wave feature and the second evolution feature on the time correlation plane, and record the start and end nodes of their interaction.

[0109] In this embodiment, on a unified time correlation plane, the short-term fluctuation curve of the first fluctuation feature and the long-term trend line of the second evolution feature are overlapped and analyzed to identify the segments in which the two show a significant correlation in time, which are defined as coupling segments. Specifically, firstly, the direction and magnitude of change of the two features are scanned by a sliding time window. When the upward or downward trend of the short-term fluctuation is consistent with the corresponding trend of the long-term evolution, the window is determined to be a coupling segment. Subsequently, the start node and end node are marked in each coupling segment, and an index is established to record the duration and stage boundaries of the interaction between the two.

[0110] Step S3033: Arrange the coupled segments sequentially to generate interaction chains, and mark the node with the highest frequency of occurrence in each interaction chain.

[0111] In this embodiment, after identifying the coupled segments and recording the nodes, the coupled segments are first arranged sequentially according to time sequence, and continuous interaction chains are formed by connecting the start and end nodes of adjacent segments. Specifically, each chain represents the continuous coupling relationship between short-term fluctuations and long-term evolution over a period of time. Subsequently, the nodes in each interaction chain are statistically analyzed, the number of times each node appears in different chains is recorded, and the node with the highest frequency is identified. This node, because it appears repeatedly in multiple interaction chains, reflects its central position in the interaction process at different time scales and is explicitly marked.

[0112] Step S3034: Gather all the most frequent nodes into a candidate set and use it as a candidate set for psychological turning points.

[0113] In this embodiment, after marking the high-frequency nodes in each interaction chain, the nodes with the highest frequency in all chains are first collected and deduplicated to form a unified node set. Then, the nodes in this set are rearranged according to the time axis order and the segment level to ensure that the candidate set reflects the continuity of time distribution and maintains consistency across scales. Furthermore, the nodes in the set are aggregated and statistically analyzed to confirm their coverage in different chains and time windows, and nodes that appear by chance and lack cross-chain consistency are filtered out. Finally, the remaining nodes are included in the psychological state inflection point candidate set to identify the key moments when an individual's psychological state changes significantly from a multimodal fusion perspective.

[0114] Step S304: Reorganize the causal chain of the candidate set and screen out the turning points with continuous triggering tendency as potential psychological crisis trigger points.

[0115] The specific steps of step S304 are as follows:

[0116] Step S3041: Arrange the nodes in the candidate set of psychological state turning points according to time order and causal dependency relationship to generate an initial causal chain, and mark the causal strength between nodes in the chain.

[0117] In this embodiment, the confirmed nodes in the inflection point candidate set are arranged in order according to their timestamps to ensure that the causal chain has a strict temporal sequence. Subsequently, a causal dependency determination mechanism is introduced between the nodes. By performing correlation analysis on the modal feature change direction, duration and triggering conditions of adjacent nodes, the possible causal relationships between the nodes are determined, and a causal strength identifier is assigned to each pair of nodes with a dependency relationship. Specifically, the assignment of causal strength comprehensively considers the tightness of the time interval between nodes, the amplitude of feature changes and the consistency level across modalities, thereby forming an initial causal chain with weight information.

[0118] Step S3042: Identify the break points in the initial causal chain where the causal strength is below a threshold, and introduce virtual transition nodes or alternative nodes to complete the causal break.

[0119] In this embodiment, after the initial causal chain is constructed, the causal strength of adjacent nodes in the chain is detected segment by segment. If the causal strength is found to be lower than a preset threshold, it is determined that there is a causal break at that point. Subsequently, in order to avoid logical interruptions in the chain, virtual transition nodes are generated using verified typical patterns or substitution features between adjacent modes in historical segments, and these nodes are inserted into the break points to fill the missing logical links. At the same time, when there are highly similar candidate nodes that can be used as substitutes, substitute nodes can be directly introduced to complete the connection of the chain. Finally, the causal chain completed in the above manner forms a continuous causal path as a whole.

[0120] Step S3043: Perform multiple rounds of traversal on the completed causal chain, check the stability of the causal path in each consecutive time window, and record the causal sub-chains that appear repeatedly in multiple windows.

[0121] In this embodiment, after the causal chain is broken and repaired, multiple time windows with different lengths and starting points are first set, and the causal paths contained in the chain are traversed round by round to check their continuity and consistency at different time scales. Specifically, in each round of traversal, the causal node sequences within the window are compared and aggregated to confirm which paths can maintain a stable causal pointing relationship in the process of extension between adjacent windows or across windows. Subsequently, these causal sub-chains that repeatedly appear in multiple window divisions are recorded separately and given a repeatability mark to indicate that the sub-chain has robustness across time periods.

[0122] Step S3044: Extract the core nodes in the repeated causal subchain as candidate key nodes, and perform cross-comparison between different subchains to remove nodes that do not have cross-chain coherence.

[0123] In this embodiment, firstly, nodes located at the hub of the causal path are extracted from the repeated causal subchains recorded in step S3043. Nodes that appear repeatedly in multiple windows and subchains and have stable pointing relationships with upstream and downstream nodes are given priority to form an initial set of candidate key nodes. Subsequently, cross-comparison is carried out between different subchains. Specifically, the relative order of the same candidate node in each subchain, the consistency of adjacent causal pairs, and the commutativity with alternative nodes are compared. If a node cannot maintain the same causal direction or shows significant order drift in most subchains, it is determined that it does not have cross-chain coherence and is removed. Further, the remaining candidate nodes are re-aggregated based on the number of subchains covered and the cross-window recurrence. Nodes that only appear briefly in a single scene or local window are removed, and stable nodes that can be reproduced across chains and scales are retained as the candidate key node set for subsequent identification of potential psychological crisis trigger points.

[0124] Step S3045: Mark the selected set of key nodes as potential psychological crisis triggers.

[0125] In this embodiment, after the cross-comparison and screening of candidate key nodes are completed, the remaining set of nodes is further uniformly labeled and given the identifier of potential psychological crisis trigger points. Specifically, the nodes in the set are first reordered according to the timeline to ensure that their temporal position in the psychological state evolution process is clear and traceable. Then, each node is matched with its corresponding causal chain segment so that each trigger point has a reference background of upstream and downstream causal relationships. Furthermore, through the coherence test between nodes, it is confirmed that these nodes all show stable key turning points in cross-chain comparison.

[0126] Step S4: When a psychological crisis trigger point is identified, generate early warning information related to the actual life situation and output corresponding intervention suggestions.

[0127] The specific steps of step S4 are as follows:

[0128] Step S401: Associate the psychological crisis trigger point with the target individual's environmental data, including time, geographical location, and social scene information.

[0129] In this embodiment, after identifying potential psychological crisis trigger points, they are mapped and associated with the target individual's external environmental data. The environmental data includes three types of elements: time stamps, geographical locations, and social scenarios. Specifically, firstly, the timestamps of the trigger points are aligned with the individual's schedule records or activity trajectories to extract the environmental context of the corresponding time period. Then, the trigger points are associated with geographical location information to identify whether they occur in a specific spatial location, such as a workplace, living environment, or unfamiliar scene. Furthermore, the trigger points are matched with social scenario information to analyze the individual's interpersonal interaction state at that moment, such as solo activities, two-person conversations, or group gatherings.

[0130] Step S402: Construct a situation mapping table based on the environmental data, and project the psychological crisis trigger points to three situational dimensions: daily life, work interaction and social relationship.

[0131] In this embodiment, after obtaining the time, geographical location, and social scene information corresponding to the psychological crisis trigger point, a situation classification rule is first set to encode environmental elements in three dimensions: daily life, work interaction, and social relationships. Then, based on the time attribute and geographical location of the trigger point, it is mapped to the daily life dimension to characterize the psychological vulnerabilities that individuals may have in their daily life, travel, or leisure activities. Next, combined with the occurrence of the trigger point in the workplace or work cycle, it is projected to the work interaction dimension to reflect the psychological risks related to task pressure, collaboration rhythm, or work environment. Finally, by analyzing the type of social scene and the participating roles of the trigger point, it is classified into the social relationship dimension to reveal the potential connection between interpersonal interaction and emotional fluctuations.

[0132] Step S403: Identify the situational fragment coupled with the psychological crisis trigger point in the situational mapping table, and generate an event chain for the situational fragment. The event chain consists of triggering conditions, behaviors, and extended scenarios.

[0133] In this embodiment, the marked psychological crisis trigger points are retrieved from the constructed situation mapping table to identify the associated situational fragments in the dimensions of daily life, work interaction, or social relationships. Specifically, the time, space, and social roles corresponding to the trigger points are cross-referenced to identify typical situational units coupled with them. Subsequently, the triggering conditions associated with the trigger points are extracted within the situational unit, such as abnormal fluctuations in physiological indicators, sudden workload, or interpersonal tension. Furthermore, the behavioral manifestations that individuals may take in this situation, such as language expression, activity patterns, or social interaction methods, are recorded together with the triggering conditions. Finally, the external extended scenarios that may be triggered by the behavior are deduced, such as emotional escalation, social isolation, or task interruption, and the triggering conditions, behaviors, and extended scenarios are organized into a complete event chain in chronological order.

[0134] Step S404: Convert the event chain into an early warning information template, and embed a set of corresponding intervention instructions into the early warning information template to form early warning information related to real-life situations.

[0135] In this embodiment, after obtaining the event chain consisting of triggering conditions, behaviors, and extended scenarios, the event chain is first semantically structured, its core elements are extracted and mapped to a standardized information template framework, thereby generating a reusable warning information template. Specifically, fixed slots are set in the template to respectively carry the description of the triggering conditions, behavioral performance tags, and the deduction results of the extended scenarios, so as to ensure that the information content is clear in hierarchy and logically consistent. Subsequently, a set of intervention instructions is embedded in the corresponding slots of the template. The intervention instructions include immediate adjustment measures based on individual physiological state, alternative activity suggestions based on behavioral patterns, and communication and mitigation strategies for social situations. Finally, the completed warning information template is instantiated as warning information associated with the actual life situation of the target individual.

[0136] Example 2

[0137] Please see Figure 4 Another embodiment of the present invention provides a crisis early warning system based on multimodal data fusion, comprising: a data acquisition module, a modal feature determination module, an inference module, and an early warning module;

[0138] The data acquisition module is used to collect multimodal data of the target individual, including physiological signal data, behavioral data, and semantic data;

[0139] The modality feature determination module is used to compare the multimodal data with preset group baseline data, detect conflict signals between different modalities, and determine the dominant modality feature through counterfactual simulation.

[0140] The reasoning module constructs a psychological state evolution model based on the dominant modality features to infer potential psychological crisis triggers.

[0141] The early warning module is used to generate early warning information related to the actual life situation and output corresponding intervention suggestions when a psychological crisis trigger point is identified.

[0142] The inference module includes: a feature extraction unit, an inference unit, and a recombination and filtering unit;

[0143] The feature extraction unit extracts a first fluctuation feature and a second evolution feature based on the dominant mode feature;

[0144] The reasoning unit is used to perform interactive reasoning between the first fluctuation feature and the second evolution feature to generate a candidate set of psychological state inflection points.

[0145] The reorganization and screening unit is used to reorganize the causal chain of the candidate set and screen out turning points with continuous triggering tendency as potential psychological crisis trigger points.

[0146] In addition, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0147] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A crisis early warning method based on multimodal data fusion, characterized in that, include: Collect multimodal data of the target individual, including physiological signal data, behavioral data, and semantic data; The multimodal data is compared with preset group baseline data to detect conflict signals between different modes, and the dominant mode characteristics are determined through counterfactual simulation. Based on the dominant modality features, a state evolution model is constructed to infer potential crisis trigger points; When a crisis trigger is identified, early warning information related to the actual life situation is generated, and corresponding intervention suggestions are output.

2. The crisis early warning method based on multimodal data fusion as described in claim 1, characterized in that, The multimodal data is compared with preset population baseline data to detect conflict signals between different modalities, and the dominant modality characteristics are determined through counterfactual simulation, including: The multimodal data of the target individuals are mapped to a unified time axis, and a multimodal control matrix is ​​established based on the preset population baseline data. In the multimodal comparison matrix, the difference signals between different modes are identified. When the degree of difference exceeds a threshold, the corresponding modal signal is marked as a conflict signal. For the conflicting signal, a corresponding counterfactual sample is generated, which is used to simulate the state inference result when the modality signal is missing; By comparing the inference results of the counterfactual samples with the actual multimodal inference results, the dominant mode is determined based on the magnitude of the deviation, and the features of the dominant mode are output.

3. The crisis early warning method based on multimodal data fusion as described in claim 2, characterized in that, By comparing the inference results of the counterfactual samples with the actual multimodal inference results, the dominant mode is determined based on the magnitude of the deviation, and the features of the dominant mode are output, including: The difference between the inference results of the counterfactual samples and the actual multimodal inference results is measured to obtain the deviation value corresponding to each mode; A deviation sequence is constructed based on the distribution of the deviation values, and stable deviation intervals and abnormal deviation peaks are identified in the deviation sequence. Using the stable deviation range as a reference, the mode corresponding to the abnormal deviation peak is marked as the candidate dominant mode; Cross-comparison is performed among candidate dominant modes. If there is an alternation of bias between modes, the mode that maintains the best performance within a preset time window is selected as the final dominant mode, and the corresponding features are output.

4. The crisis early warning method based on multimodal data fusion as described in claim 3, characterized in that, The state evolution model constructed based on dominant modal characteristics, and the reasoning of potential crisis trigger points, include: The dominant modal features are serialized in chronological order to establish a state time sequence chain; A multi-scale segmentation window is introduced on the time series chain to extract the first fluctuation feature and the second evolution feature respectively; The first fluctuation feature and the second evolution feature are used to perform interactive reasoning to generate a candidate set of situation inflection points; The candidate set is reorganized into causal chains to screen out turning points with a tendency to trigger continuously, which are then used as potential crisis trigger points.

5. The crisis early warning method based on multimodal data fusion as described in claim 4, characterized in that, The step of serializing the dominant modal features in chronological order to establish a state time sequence chain includes: The dominant modal features are initially sorted according to the acquisition timestamp, and redundant features with overlapping or missing segments are removed. Temporal hierarchical labels are attached to the sorted dominant modality feature sequence to divide the features into micro-instantaneous segments, daily cycle segments, and cross-cycle segments. Based on the time layer labels, the connection relationship between different segments is reconstructed to generate a time recursive chain; In the time recursion chain, the transition nodes across segments are identified, and these transition nodes are used as key anchor points of the state time sequence chain to construct the state time sequence chain.

6. The crisis early warning method based on multimodal data fusion as described in claim 4, characterized in that, The first fluctuation feature and the second evolution feature are used for interactive reasoning to generate a candidate set of situation inflection points, including: The first fluctuation feature and the second evolution feature are projected onto a unified time correlation plane to establish a mapping relationship between them. On the time-related plane, identify the coupled segments of the first wave feature and the second evolution feature, and record the start and end nodes of their interaction; The coupled segments are arranged sequentially to generate interaction chains, and the node with the highest frequency of occurrence is marked in each interaction chain; All the nodes with the highest frequency are aggregated into a candidate set, which is then used as a candidate set for turning points in the situation.

7. The crisis early warning method based on multimodal data fusion as described in claim 4, characterized in that, The candidate set is reorganized into causal chains to screen out inflection points with a tendency to trigger continuously, as potential crisis trigger points, including: Arrange the nodes in the candidate set of the situation inflection point according to the time order and causal dependency relationship to generate an initial causal chain, and mark the causal strength between the nodes in the chain; In the initial causal chain, identify the break points where the causal strength is below a threshold, and introduce virtual transition nodes or alternative nodes to complete the causal break points; The completed causal chain is traversed multiple times to check the stability of the causal path within each consecutive time window, and the causal sub-chains that repeat in multiple windows are recorded. The core nodes in the repeated causal subchains are extracted as candidate key nodes, and cross-comparison is performed between different subchains to remove nodes that do not have cross-chain coherence. The selected set of key nodes is marked as potential crisis trigger points.

8. The crisis early warning method based on multimodal data fusion as described in claim 7, characterized in that, When a crisis trigger point is identified, early warning information related to the actual life situation is generated, including: The crisis trigger point is associated with the target individual's environmental data, which includes time, geographical location, and social context information. Based on the environmental data, a scenario mapping table is constructed, and crisis trigger points are projected to three scenario dimensions: daily life, work interaction, and social relationships. Identify situational fragments coupled with crisis trigger points in the situational mapping table, and generate event chains for the situational fragments. The event chains consist of triggering conditions, behaviors, and extended scenarios. The event chain is transformed into an early warning information template, and a set of corresponding intervention instructions is embedded in the early warning information template to form early warning information that is related to real-life situations.

9. A crisis early warning system based on multimodal data fusion, used to implement the crisis early warning method based on multimodal data fusion as described in any one of claims 1-8, characterized in that, include: The system comprises a data acquisition module, a modal feature determination module, an inference module, and an early warning module. The data acquisition module is used to collect multimodal data of the target individual, including physiological signal data, behavioral data, and semantic data; The modality feature determination module is used to compare the multimodal data with preset group baseline data, detect conflict signals between different modalities, and determine the dominant modality feature through counterfactual simulation. The reasoning module constructs a state evolution model based on the dominant modality features and infers potential crisis trigger points. The early warning module is used to generate early warning information related to real-life situations and output corresponding intervention suggestions when a crisis trigger point is identified.

10. The crisis early warning system based on multimodal data fusion as described in claim 9, characterized in that, The reasoning module includes: a feature extraction unit, a reasoning unit, and a recombination and filtering unit; The feature extraction unit extracts a first fluctuation feature and a second evolution feature based on the dominant mode feature; The reasoning unit is used to perform interactive reasoning between the first fluctuation feature and the second evolution feature to generate a candidate set of situation inflection points; The reorganization and screening unit is used to reorganize the causal chain of the candidate set and screen out the turning points with continuous triggering tendency as potential crisis trigger points.

Images