Event analysis method and device based on multi-modal data, equipment and medium

Abstract

The invention relates to the technical field of intelligent decision making, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a multi-modal data-based event analysis method, which comprises the following steps of: obtaining attribute data, monitoring data, environment data and biological characteristic data of a user, and performing standardization processing on the attribute data, the monitoring data, the environment data and the biological characteristic data; combining semantic analysis and structured fusion of the text records to form a unified input data set; and performing multi-modal fusion modeling on the unified input data set by using an intelligent decision model, generating a potential event analysis result of an individual state, generating a visual report containing potential event levels, main influence factors and intervention suggestions based on the result, and finally sending the report to a user terminal. According to the method, the multi-source structured data and the multi-source unstructured data are integrated, and the intelligent decision-making model is introduced to realize feature level fusion and semantic enhancement modeling, so that the accuracy and timeliness of individual state analysis are effectively improved, and the personalized expression of an analysis result is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent decision-making technology, and in particular to an event analysis method, device, equipment and storage medium based on multimodal data. Background Art

[0002] In current multi-domain intelligent analysis systems, particularly in healthcare and fintech scenarios, risk identification and early warning for individual status still primarily rely on structured indicators and manual empirical rules. Traditional systems typically use a small amount of attribute data as input variables and employ static modeling. This makes it difficult to fully integrate data from multiple sources and effectively reflect the dynamic changes and semantic context of user status.

[0003] In the healthcare business, the identification and analysis of individual user status is widely used in scenarios such as health management, auxiliary assessment, and early warning intervention. However, most existing systems still rely on static models built based on structured data, mainly referring to attribute information such as the user's age, gender, and medical history, and lack the ability to comprehensively process dynamic monitoring data (such as heart rate, sleep, exercise, etc.) and external environmental data (such as air quality, temperature and humidity, etc.). In addition, when it comes to unstructured text information such as user subjective descriptions, medical interview texts, and electronic medical records, the system generally lacks semantic understanding and structured extraction mechanisms, resulting in a large amount of information closely related to individual status being unable to participate in subsequent analysis, limiting the accuracy and personalization of status recognition.

[0004] In the fintech sector, risk assessment for health-related scenarios also faces technical limitations. For example, in health underwriting, risk pricing, or long-term health insurance modeling, traditional systems primarily rely on structured historical data to execute rule-matching or scoring mechanisms. This makes it difficult to fully integrate multi-source, heterogeneous user status data, especially lacking dynamic adaptation mechanisms for real-time data changes. Furthermore, the lack of intelligent semantic analysis for user-submitted questionnaire texts or supplementary explanations makes it difficult to convert them into structured features for modeling, resulting in low information utilization and weak assessment capabilities. Summary of the Invention

[0005] The main purpose of the present invention is to provide an event analysis method, device, equipment and storage medium based on multimodal data, aiming to solve the technical problem that the existing technology cannot achieve unified fusion and in-depth analysis of multimodal structured and unstructured data, resulting in incomplete individual status recognition and difficulty in personalized analysis.

[0006] To achieve the above objectives, the present invention provides an event analysis method based on multimodal data, comprising:

[0007] Obtain user attribute data, monitoring data, environmental data, and biometric data and perform standardization processing to generate an initial input data set;

[0008] Performing semantic analysis on the user's text records, extracting key information and fusing it with the initial input data set to generate a unified input data set;

[0009] Performing multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual states;

[0010] Based on the potential event analysis results, generate a visual report including the potential event level, main influencing factors and intervention suggestions;

[0011] The visualization report is sent to a user terminal.

[0012] Furthermore, to achieve the above-mentioned object, the present invention provides an event analysis device based on multimodal data, comprising:

[0013] Multimodal data acquisition module, used to obtain user attribute data, monitoring data, environmental data and biometric data and perform standardization processing to generate the initial input data set;

[0014] A semantic information fusion module is used to perform semantic analysis on the user's text records, extract key information and fuse it with the initial input data set to generate a unified input data set;

[0015] A multimodal in-depth analysis module, configured to perform multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual states;

[0016] A result visualization module is used to generate a visualization report including the potential event level, main influencing factors and intervention suggestions based on the potential event analysis results;

[0017] The report delivery module is used to send the visualization report to the user terminal.

[0018] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and an event analysis program based on multimodal data stored in the memory and executable on the processor. When the event analysis program based on multimodal data is executed by the processor, the steps of the event analysis method based on multimodal data as described above are implemented.

[0019] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an event analysis program based on multimodal data is stored. When the event analysis program based on multimodal data is executed by a processor, the steps of the event analysis method based on multimodal data as described above are implemented.

[0020] Beneficial effects: The present invention relates to the field of intelligent decision-making technology and can be applied to business scenarios such as financial technology and medical health. It discloses an event analysis method based on multimodal data, including: obtaining the user's attribute data, monitoring data, environmental data and biometric data and performing standardization processing to generate an initial input data set; performing semantic analysis on the user's text records, extracting key information and fusing it with the initial input data set to generate a unified input data set; performing multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual status; based on the potential event analysis results, generating a visualization report containing potential event levels, major influencing factors and intervention recommendations; and sending the visualization report to the user terminal. The present invention constructs a unified input expression by fusing multi-source heterogeneous structured data with semantic text information, and uses an intelligent decision-making model for joint modeling and feature extraction, thereby effectively improving the accuracy and timeliness of individual status analysis, enhancing the ability to identify potential events and the personalized expression of analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0022] Figure 1 Schematic diagram of an application environment of an event analysis method based on multimodal data in one embodiment of the present invention;

[0023] Figure 2 1 is a flow chart of an embodiment of a method for event analysis based on multimodal data according to the present invention;

[0024] Figure 3 Schematic diagram of functional modules of a preferred embodiment of an event analysis device based on multimodal data of the present invention;

[0025] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0026] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] The event analysis method based on multimodal data provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can obtain the user's attribute data, monitoring data, environmental data and biometric data through the user terminal and perform standardization processing to generate an initial input data set; perform semantic analysis on the user's text records, extract key information and fuse it with the initial input data set to generate a unified input data set; perform multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual status; based on the potential event analysis results, generate a visualization report containing potential event levels, main influencing factors and intervention recommendations; and send the visualization report to the user terminal. The present invention constructs a unified input expression by fusing multi-source heterogeneous structured data with semantic text information, and uses an intelligent decision-making model for joint modeling and feature extraction, thereby effectively improving the accuracy and timeliness of individual status analysis, enhancing the ability to identify potential events and the personalized expression of analysis results. Among them, the user terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0029] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of an event analysis method based on multimodal data provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0030] like Figure 2 As shown, the event analysis method based on multimodal data proposed by the present invention includes the following steps:

[0031] S10, obtaining the user's attribute data, monitoring data, environmental data, and biometric data and performing normalization processing to generate an initial input data set;

[0032] In this embodiment, to accurately assess individual user status, data from multiple sources must be structured and preprocessed uniformly. First, the user's attribute data, monitoring data, environmental data, and biometric data are obtained and standardized to construct an initial input dataset for subsequent modeling. This process addresses the inability of existing analysis systems to uniformly process multimodal data, taking into account the diversity of data sources.

[0033] Attribute data refers to static or periodically updated demographic information and historical behavior records about users. Common data fields include age, gender, occupation, long-term health records, or financial records. This data typically comes from local databases, historical platform records, online questionnaires, or user self-reported forms. These fields are valuable in assessing user stability characteristics or long-term drift trends.

[0034] Monitoring data refers to information collected by wearable devices or IoT terminals that reflects changes in a user's current state, such as continuously recorded heart rate fluctuations, blood pressure trends, exercise intensity, or sleep cycles. This type of data typically has time series properties, so processing requires operations such as sample alignment, outlier removal, and window aggregation to extract feature expressions that can be used for modeling.

[0035] Environmental data is obtained from external sensing systems related to the user's location, such as air quality monitoring nodes, meteorological data interfaces, or geolocation tracking modules. This data can reflect the impact of the user's physical environment on their behavior and physiological state. For example, factors such as high temperature, humidity, and pollution may affect health status or behavioral responses. Environmental data is typically expressed in the form of spatially distributed parameters, such as the Air Quality Index (AQI), PM2.5 concentration, longitude and latitude coordinates, and local average humidity.

[0036] Biometric data primarily includes identity and individual characteristics collected by biometric devices or medical testing tools, such as fingerprints, voiceprint models, gene fragments, and phenotypic codes. This type of data is typically complex, high-dimensional, and subject to confidentiality requirements. Therefore, prior to processing, operations such as anonymization, feature extraction, and vectorization are required.

[0037] After acquiring various data types, standardization is necessary to achieve unified modeling. Standardization not only includes basic operations such as numerical normalization, data format unification, and encoding alignment, but also includes processing strategies such as time series alignment, spatial coordinate mapping, noise removal, and missing data completion. This process requires different preprocessing strategies for different modalities, but all data should ultimately be converted into structured data objects that can be uniformly represented as the initial input dataset.

[0038] Among them, attribute data usually adopts discrete variable encoding methods such as one-hot encoding, label encoding or bin mapping. Monitoring data requires the construction of a sliding time window for statistical feature extraction and the data to be normalized in segments. Environmental data is combined with the user's geographic coordinates to perform geocoding conversion to form an input structure aligned with the spatial reference system. Biometric data is often converted to vectors through deep embedding models or compression mapping to generate feature representations with unified dimensions. Ultimately, these data from different sources are aligned and spliced in a unified coordinate space to generate an initial input data set with consistent structure and complementary semantics.

[0039] In practical implementation, attribute data can be obtained through static information forms filled out during user registration, or by connecting to existing user data interfaces in business systems. This process can be combined with a data cleansing module to complete or standardize any irregular user entries. Monitoring data can be collected in real time through wearable device SDK integration, with metrics such as heart rate, blood pressure, and step count aligned with timestamps and downsampling performed through a caching system. Environmental data can be introduced by integrating third-party API services, such as the National Meteorological Service or local air quality platforms, to periodically capture corresponding metrics based on the user's geographic location. This data must be aligned with the user's location information using a spatial indexing mechanism. Biometric data processing relies on dedicated collection modules, such as fingerprint recognition modules, biochip readers, or medical testing data platforms. After collection, identity desensitization and security vector compression must be implemented using privacy protection mechanisms. Standardization strategies can automatically classify data based on data type. The system can be configured with a hierarchical data preprocessing pipeline, where attribute data is processed by an encoding module, monitoring data by a time series processing module, environmental data by a geographic mapping module, and biometric data by an embedding vector generation module. After processing by these modules, the intermediate results are fed into a unified feature concatenation layer to construct the final initial input dataset. The dataset is cached in the analysis pipeline for subsequent analysis modules to call.

[0040] It can also be adapted to different terminal forms, such as completing some pre-processing tasks at the edge computing node and transmitting the data in a unified format to the cloud analysis center; it can also perform some feature conversion on the data collection side and embed vectorized processing into the biometric terminal, thereby reducing the cloud computing burden.

[0041] Example: In the healthcare sector, a user's historical diagnostic information, heart rate fluctuations over the past seven days, PM2.5 concentrations in their living environment, and specific mutation sites from genetic testing can be introduced as different modal data. This initial input dataset comprehensively reflects the user's current status and potential risk clues, supporting the system to provide more accurate status analysis and intervention recommendations in the next stage.

[0042] In the field of financial technology, occupation and insurance history from attribute data, health behavior changes from monitoring data, city residence level from environmental data, and identity recognition results from biometric data can all be used as inputs for assessing customer health preferences and pricing factors. After unified processing, they are fed into the scoring model. The multi-dimensional structure of the initial input dataset makes it easier for the model to discover user individual characteristics, improving the accuracy of intelligent pricing and personalized policy delivery.

[0043] By constructing an initial input dataset based on multimodal and multi-source data, we achieve unified processing of structured data, temporal data, spatial data, and high-dimensional biological information. This not only improves the integrity of data fusion but also provides a unified input interface for subsequent deep modeling. Through standardized processing, the impact of data bias on modeling accuracy is significantly reduced, giving the system excellent portability and data scalability.

[0044] S20, performing semantic analysis on the user's text records, extracting key information and fusing it with the initial input dataset to generate a unified input dataset;

[0045] In this embodiment, to achieve semantic fusion of multimodal data, it is necessary to introduce deep parsing and interactive expression of natural language data on top of structured numerical data. Text records primarily come from descriptive content during user interactions, including medical consultation notes, insurance application statements, service feedback text, and user report summaries. While these texts are highly expressive, they often contain important supplementary information about individual status and are highly context-dependent and semantically unstructured.

[0046] The semantic parsing process of text records usually consists of multiple sub-stages. First, named entity recognition is performed on the text content through a pre-trained language model to extract key entity phrases containing user status information, such as physiological indicators (such as "increased blood pressure" and "heart rate fluctuations"), operational instructions (such as "reducing high-salt diets"), and potential event descriptions (such as "recent chest tightness"). The language model can adopt a natural language understanding model based on the Transformer structure, such as BERT, RoBERTa, or ERNIE, which has strong semantic modeling and entity context recognition capabilities.

[0047] After entity extraction, semantic relationship analysis techniques are used to construct the logical structure between entities, forming structured textual supplementary information. This process utilizes technical tools such as dependency parsing, sentence-level nested relationship analysis, and relationship classifiers to identify semantic relationships such as "cause and effect," "condition," and "suggestion," transforming the original text into structured information units that can be used for data fusion.

[0048] After information extraction and structuring are complete, the temporal and spatial attributes of the supplementary text information need to be identified and aligned to enable integration with the original numerical data. For example, a record of "multiple insomnia nights last week" should be parsed as an abnormal event for a monitoring indicator within a time range and connected to the sleep data entry for the corresponding time period in the unified input dataset. Spatial attributes such as "experienced breathing discomfort while living at high altitude" need to be matched with the geographic coordinates in the environmental data to achieve cross-modal data synchronization.

[0049] Finally, the aligned text supplementary information is spliced with the attribute data, monitoring data, environmental data, and biometric data in the initial input dataset to form a unified input dataset with semantically enhanced expressive capabilities. To prevent semantic information from being overwhelmed by mainstream numerical features, the splicing method can adopt feature-level alignment mechanisms or attention-enhanced fusion methods, so that the text semantics are embedded in the overall feature space, maintaining independent expression and contextual control capabilities.

[0050] The processing of text records can adopt a three-stage process. In the first stage, the original text is converted into language encoding through the input standardization module, invalid symbols are removed, and time and unit expressions are standardized; in the second stage, the trained medical, insurance or service scenario-specific language model is used to perform entity recognition and relationship labeling; in the third stage, structured triples or embedding matrices are generated based on the recognition results, and time alignment and position matching with the structured data are performed.

[0051] The text entity vector can be jointly modeled by co-embedding with the feature space of the attribute data, so that "taking antihypertensive drugs" in the text and the "hypertension" label in the attribute data form a feature co-association; the sentence containing the description of the time series event can also be mapped to the time window of the monitoring data, and the indicator changes in the sliding window can be matched with the text description to achieve sequence-level semantic linkage.

[0052] If applied to edge devices, text parsing and structuring can also be performed locally, and only the structured results can be uploaded to the cloud to participate in unified input data generation; cross-modal comparative learning mechanisms can also be introduced during model training to make the text semantic expression more in line with the target data characteristics.

[0053] Example: In a healthcare scenario, a doctor might record that "a patient has repeatedly complained of dizziness and a low heart rate in the morning over the past three days." The system uses semantic analysis to label "low heart rate in the morning" as an abnormal fluctuation in a time-series monitoring indicator. It then locates the heart rate indicator during the morning hours in the monitoring data and associates it with the corresponding physiological state. This process provides the analysis model with interpretability and semantic awareness.

[0054] In the fintech scenario, when users apply for insurance, they fill in "recent adjustment to night shift work schedule". The system extracts "work schedule change" through semantic structuring and matches it to the sleep segment abnormalities in the monitoring data. It also links the occupational information "night operator" in the attribute data to identify the potential impact of changes in their living status on risk factors, providing higher semantic dimension support for subsequent model building.

[0055] By introducing semantic parsing and structured integration of text records, the expressive dimensionality of unified input datasets can be significantly improved, enhancing the system's ability to understand individual states in complex scenarios. Information such as behavioral patterns, anomaly perception, and user tendencies described in text can be effectively converted into usable features for modeling, addressing areas that are difficult to capture with structured data and effectively improving overall analysis accuracy and personalized recommendation capabilities.

[0056] S30, performing multimodal data fusion processing on the unified input data set through an intelligent decision model to generate a potential event analysis result of the individual state;

[0057] In this embodiment, multimodal data fusion processing plays a key role in the unified representation and joint modeling of heterogeneous information sources in the intelligent decision-making model. It is particularly suitable for fusing multiple types of data inputs, such as numerical, textual, temporal, and spatial data. The unified input dataset contains attribute data, monitoring data, environmental data, biometric data, and textual supplementary information, with obvious modal differences and structural inconsistencies. To achieve accurate modeling of individual states, it is necessary to construct differentiated encoding pathways according to feature modalities in the intelligent decision-making architecture, and perform fusion modeling in the joint feature space.

[0058] First, the convolutional neural network module processes the spatial distribution relationships within attribute and biometric data, adapting to the underlying structural correlation patterns between static attribute items (such as age, gender, and genetic structure). The convolution kernel performs sliding convolution and local aggregation operations across the feature dimensional space, extracting low-level and high-level features to form a multi-layer spatial semantic nesting.

[0059] Secondly, the recurrent neural network module is used to model the time series features in the monitoring data, capturing the changing trends and continuity of individual states over time. Typical implementations include long short-term memory networks (LSTMs) or gated recurrent units (GRUs). These can model the periodicity, mutation points, and trend shifts of daily continuous monitoring data (such as heart rate, blood pressure, and exercise volume), and generate high-dimensional feature representations of the time series.

[0060] Environmental data and supplementary textual information, as externally relevant modalities, are fused and expressed through an attention mechanism. Environmental data typically includes highly variable factors such as geographic location and meteorological indicators, while supplementary textual information provides a semantically enhanced representation of individual states. A multi-head attention mechanism learns the importance weights of environmental factors for different feature channels, while simultaneously identifying the modulatory role of various semantic markers in the text in modeling the current state. This generates a weight matrix for the environmental and textual associations, which serves as the modulation input for subsequent joint modeling.

[0061] Subsequently, the spatial correlation features and time series dependency features output by the three modules are combined with the context- and text-related weight matrices and fed into the multimodal fusion module. This fusion module uses feature-level concatenation, gated fusion, or tensor crossover to achieve multimodal feature fusion. Furthermore, through dimensionality reduction operations (such as principal component analysis, linear transformation, and projection networks), a unified multimodal joint feature vector is generated, resulting in highly compressed and expressive input features.

[0062] Finally, the multimodal joint feature vector is fed into the model's output layer, where a classifier or regressor outputs the potential event analysis results for the current individual state. This analysis is output as an event category label, risk level label, or associated probability distribution, representing the potential event types and occurrence trends assessed by the model based on the current multimodal state characteristics.

[0063] A unified multimodal modeling network based on the Transformer architecture can be used to uniformly encode four types of input: attributes, monitoring, environment, and text, and then feed them into a multi-layer self-attention mechanism for processing, achieving fully connected feature perception across modalities. In lightweight scenarios, a CNN-LSTM dual-path architecture can be used to model spatial information and time series separately, then combine them at the fusion layer, and introduce external text and environmental modulation factors through the attention mechanism.

[0064] During the training phase, multimodal input data can be used to introduce strategies such as masking training and modality contrast loss to improve the model's robustness to modality loss or offset. During the inference phase, if some modal data is unavailable, an approximate embedding of the missing modality can be estimated by combining existing modal inputs with historical data, enabling fault-tolerant inference.

[0065] A multi-task learning structure can also be used. Based on the main task outputting potential event categories, additional tasks can be set to output auxiliary labels such as risk levels and trigger factors to improve the generalization ability of the overall modeling structure and the interpretability of the results.

[0066] Example: In a healthcare scenario, a user recently detected unstable nighttime heart rate. Attribute data indicates they are in a high-risk age group, text records describe them as "frequently anxious and insomniac," and environmental data indicates severe noise pollution in their residential area. Fusion modeling can output a potential event, "Currently in an Abnormal Mental Stress State," and determine its potential for developing into a sleep disorder.

[0067] In a financial scenario, a user's profile may indicate a high credit score, but recent transaction monitoring data indicates frequent small transfers. Supplemental text mentions "temporary economic pressure," and environmental information indicates their industry has been severely impacted by sudden policy adjustments. Fusion modeling can identify potential abnormal financial behavior in this user, facilitating dynamic adjustments to financial service strategies.

[0068] By integrating the expressive power of multimodal input data, intelligent decision-making models can identify individual state signals across time, space, behavior, and text. They also integrate multi-source heterogeneous information to form highly expressive joint feature vectors, effectively improving the accuracy and stability of potential event identification. This is especially true for complex scenarios where the same potential event is triggered by different modalities. Cross-dimensional modeling can be achieved, thus avoiding analytical errors caused by misjudgments or missing information.

[0069] S40, generating a visual report including the potential event level, main influencing factors and intervention suggestions based on the potential event analysis results;

[0070] In this embodiment, the potential event analysis result is a quantitative analysis output generated by the intelligent decision-making model for multimodal input data, which usually includes the probability distribution of the individual's current state that may evolve into an adverse event, the activation feature dimensions and their corresponding importance scores, etc. The visualization report built around the analysis result must include three core contents: potential event level, main influencing factors and intervention recommendations. These three parts need to be generated through steps such as parsing the model output features, semantic mapping of the original input features and linkage reasoning with the external intervention knowledge base, and finally combined with the front-end rendering module to generate user-readable and interactive diversified display results.

[0071] Potential incident levels are generated based on the probability values or scoring metrics from the analysis results, which are then mapped to pre-defined risk categories, such as low, medium, or high risk. The mapping rules can be linear, quantile, or soft distribution normalization strategies, typically based on the output confidence, probability, or log-likelihood metrics in the model loss function. These levels are not only used for user perception but also provide conditional input for subsequent recommendations and response processes.

[0072] The generation of key influencing factors relies on parsing the activated feature dimensions in the model output. These feature dimensions are derived from the vector representation structure of the multimodal fusion process in the previous stage, and some dimensions have significant weights when generating potential event analysis results. By parsing the original input fields corresponding to the activated feature dimensions (for example, high weights correspond to fields such as "heart rate fluctuation" and "PM2.5 concentration") and combining the specific values of these fields with their temporal distribution patterns, it is possible to determine whether they are important variables affecting the level of the potential event.

[0073] The generation of intervention recommendations is based on a predefined intervention strategy library. The intervention strategy library is built on the basis of rule mapping, data-driven or expert systems, and contains response measures under the conditions of a combination of event levels and influencing factors. For example, for "high respiratory pressure levels dominated by environmental factors", intervention plans such as "using air purifiers and reducing travel during peak hours" can be recommended. The content of recommendations is not only recommended based on the level of the event, but also dynamically selects the more executable recommendations based on the user's personalized attributes (such as age, gender, and basic status), and can be labeled with recommendation priority and implementation cycle.

[0074] The final visualization report must structure the aforementioned levels, influencing factors, and recommendations before being fed into the front-end rendering module. The rendering process integrates the text summary with graphical structures (such as dynamic pie charts, heat maps, and time-series trend graphs) to enhance users' intuitive understanding of their status and their willingness to respond. The report format should accommodate differences in rendering capabilities between mobile and web platforms, with adaptive layouts and interactive feedback channels.

[0075] Graphical user interface component libraries (such as ECharts and D3.js) can be used to generate complex report pages that include multiple risk level indicators, dynamic indicator trend lines, and keyword annotation capabilities. Potential event levels are displayed through color-coded risk index cards, while key influencing factors are displayed through weighted ranking tables, expandable explanation cards, or graphical heat distribution. Intervention suggestions are presented as conditional buttons, link recommendations, or timed execution plan entries.

[0076] You can also incorporate user behavior records (click history, reading time, etc.) to dynamically adjust the report display order. For example, prioritize risk factor content that users have viewed or interacted with multiple times, and optimize the visual focus layout in the report through A / B strategies.

[0077] The intervention recommendation strategy library can be stored in the form of rule templates, with each strategy containing the applicable level range, trigger factor matching expression and recommended text content; a reinforcement learning model can also be used to dynamically optimize the recommended content, and the response effect model can be trained in combination with user feedback data to achieve personalized adjustments in future iterations.

[0078] Example: In a healthcare scenario, a user's potential event analysis results in a "high risk of abnormal breathing" and a "medium-high risk" rating. Key influencing factors include "frequently elevated heart rate at night," "persistently exceeding PM2.5 concentration standards," and the presence of the keyword 'shortness of breath' in test records. Based on the user's age of 60 or older, the system recommends interventions such as "increase air purifier use during sleep, avoid going out during morning exercise, and undergo lung function screening." These recommendations are displayed through dynamic charts showing trend curves, heat maps displaying pollution maps, and recommendation panels presenting textual suggestions. The resulting report is then sent via the WeChat mini-program, with user confirmation and feedback.

[0079] In a financial scenario, a potential analysis of an individual's status indicates "low short-term debt repayment ability," resulting in a "medium risk" rating. Key influencing factors include "fluctuations in credit card repayment frequency," "abnormal income-to-expenditure ratio," and the presence of the term 'funds tight' in text records. Based on the user's monthly income and debt structure, the system recommends "temporarily reducing credit limits, enabling installment financing, and recommending asset optimization consultation." This report is delivered within the app, and user behavior (such as clicking to accept or requesting adjustments) provides feedback on the effectiveness of these recommendations, which is then used to adjust subsequent strategies.

[0080] By analyzing the three structured levels of content generated by potential event analysis results—levels, influencing factors, and recommendations—it can help users achieve a complete closed loop from result recognition and cause understanding to action plans, thereby intervening in potential adverse trends early in the evolution of the status. Compared to static score displays, visual reports guide users to participate in status interpretation and intervention in an interactive manner, significantly enhancing users' behavioral response willingness and execution effectiveness. At the same time, the multi-layered information display method helps provide traceable and structured reference analysis content to other system modules (such as medical intervention decision-making systems and credit assessment platforms).

[0081] S50: Send the visualization report to the user terminal.

[0082] In this embodiment, the action of sending the visualization report to the user terminal is essentially the process of transmitting data content from the model generation end to the user interaction end. Its technical logic not only includes the packaging of content format and the selection of transmission protocol, but also covers key links such as terminal adaptation, permission control, and maintenance of data interactivity. First, after the intelligent decision-making model completes the potential event analysis and generates the potential event level, main influencing factors, and intervention recommendations, this structured information is formatted into a unified visualization output format, such as a JSON structure, an HTML page fragment, or a set of graphics rendering instructions. These contents are processed by the rendering engine on the server side to form the final report data package.

[0083] To adapt to different user terminals (e.g., mobile phones, tablets, and desktop devices), report content must undergo client-side rendering adaptation before transmission, including content partitioning, responsive layout, and interactive logic injection. Component-based front-end frameworks (e.g., React or Vue) are often used to integrate the report's static content with real-time data binding mechanisms, ensuring that the display is refreshed in real time after the terminal receives the data.

[0084] During transmission, a data channel must be selected that is compatible with the target terminal's communication capabilities and content structure. For example, static content can be transmitted directly to the browser or app rendering module via HTTPS, while content that requires real-time interaction can be pushed via WebSocket or MQTT protocols. This ensures that interactive commands in the data visualization results (such as clicking on a risk point to display detailed information) are synchronously transmitted and trigger a view update. If privacy data or policy recommendations are involved, content permission tags must be included before transmission so that the terminal can parse and control display based on permissions.

[0085] On the user terminal, the system captures user operation behaviors in real time through event monitoring interfaces (such as DOM event listeners and Native event callbacks), such as adjusting screening parameters, dragging time ranges, confirming intervention suggestions, etc., and encodes these interactive behaviors and feeds them back to the back-end service through the specified protocol as input for further risk judgment or user behavior modeling.

[0086] During implementation, static rendered content can be delivered using a RESTful interface, suitable for low-interaction or graphic summary reports. Alternatively, persistent connections (such as WebSocket) can be used to maintain the persistence of report delivery channels, suitable for scenarios where data updates are frequent or user actions require real-time response. The front-end can encapsulate reporting components as adaptive component sets, dynamically enabling different content presentation modes based on the terminal screen resolution and input method (touch or mouse).

[0087] Visual reports can also be encrypted and compressed before being sent, and then transmitted to the terminal client after verifying the user's identity through the permission verification middleware, preventing unauthorized access or content tampering. When users click or slide interactive operations, the front end can generate operation event logs and feed them back to the server for behavior prediction, model optimization, or policy updates.

[0088] Through front-end dynamic components and data interaction channels, users can not only passively receive status judgments but also participate in data filtering, feedback, and personalized selection, thereby building a closed-loop risk awareness and response chain. The system is also highly adaptable and secure, supporting cross-platform multi-terminal rendering, and ensuring the integrity and privacy of data transmission through permission verification and transmission encryption.

[0089] The present invention relates to the field of intelligent decision-making technology and can be applied to business scenarios such as financial technology and medical health. It discloses an event analysis method based on multimodal data, including: obtaining the user's attribute data, monitoring data, environmental data and biometric data and performing standardization processing to generate an initial input data set; performing semantic parsing on the user's text records, extracting key information and fusing it with the initial input data set to generate a unified input data set; performing multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual status; based on the potential event analysis results, generating a visualization report containing potential event levels, major influencing factors and intervention suggestions; and sending the visualization report to the user terminal. The present invention constructs a unified input expression by fusing multi-source heterogeneous structured data with semantic text information, and uses an intelligent decision-making model for joint modeling and feature extraction, thereby effectively improving the accuracy and timeliness of individual status analysis, enhancing the ability to identify potential events and the personalized expression of analysis results.

[0090] In one embodiment, the above step S10 includes:

[0091] S101, obtaining user attribute data from a local database or an external interface;

[0092] S102, collecting user monitoring data from wearable devices or IoT sensors;

[0093] S103, obtaining the user's environmental data through environmental monitoring equipment;

[0094] S104, obtaining the user's biometric data through a biometric recognition device;

[0095] S105, formatting the user basic information and historical records in the attribute data to generate encoded attribute data;

[0096] S106, performing sliding window feature extraction on the time series change indicators in the monitoring data to generate monitoring data after feature extraction;

[0097] S107, performing geocoding mapping on the spatial distribution parameters in the environmental data to generate mapped environmental data;

[0098] S108, performing feature vectorization processing on the heterogeneous information in the biometric data to generate vectorized biometric data;

[0099] S109 , cross-modal alignment and concatenation of the encoded attribute data, the feature-extracted monitoring data, the mapped environmental data, and the vectorized biometric data to generate the initial input data set.

[0100] In this embodiment, the local database refers to a data storage system deployed on an enterprise, institution, or individual terminal, which is used to store user historical information, basic attributes, behavior logs, or other structured data. This type of database is usually directly controlled and maintained by the system, has high access speed and data security, and is suitable for scenarios with high requirements for data real-time performance or privacy protection needs. For example, in a medical and health system, the electronic health record (EHR) database in the hospital's internal information system can be regarded as a local database, which records the user's gender, date of birth, medical history summary, and other information. The system can call it locally without the need for network transmission.

[0101] An external interface refers to an interface mechanism for accessing third-party data sources through network communication, usually in the form of an application programming interface (API). These interfaces can be used to dynamically call structured or semi-structured data from external platforms, cloud services, or other systems. External interfaces are often used in cross-system collaboration or real-time data synchronization scenarios, such as obtaining users' social security information through government interfaces, pulling user-authorized financial account attribute information through financial open platform APIs, or accessing data collected by users from sensors on multiple devices through health data open platforms. Compared with local databases, external interfaces have stronger connectivity and data coverage capabilities, but their access latency, authorization mechanisms, and data quality control are usually more complex.

[0102] Attribute data mainly refers to basic information related to the user's identity, background and historical health status, such as relatively stable information such as age, gender, occupation, etc., which are usually provided by existing health record databases, user information systems or third-party authentication platforms. As an important part of attribute data, medical history records can include past hospitalization records, chronic disease diagnosis information, etc., and are highly sensitive and valuable for analysis in medical and health scenarios. Monitoring data is reflected in the trajectory of changes in the user's physiological state over a certain period of time. It usually comes from wearable devices (such as bracelets, blood pressure monitors) or networked monitoring terminals (such as smart mattresses), and contains data entries in a timestamp sequence, such as heart rate, blood pressure, exercise volume and sleep status.

[0103] Environmental data emphasizes the role of external factors in analyzing individual status. This data is typically collected by environmental monitoring equipment and includes outdoor air quality index, current geographic location coordinates (such as GPS), and climate parameters such as temperature and humidity. This data can be obtained synchronously through IoT platforms or mobile terminals. Biometric data originates from terminal devices with collection capabilities (such as fingerprint scanners, voiceprint recognition modules, and genetic testing chips). Its essence is high-dimensional, unstructured information, which needs to be converted into model-readable structured input through a unified vector mapping method. After data acquisition, the standardization process for these four types of data is technically complex.

[0104] The encoding processing of attribute data mainly adopts discrete value mapping, numerical normalization or one-hot encoding to convert non-numeric fields (such as gender and occupation) into corresponding dimensions in the structured data matrix; due to the time series characteristics of monitoring data, statistical features need to be extracted based on sliding windows, such as the maximum value, mean, standard deviation, volatility and other indicators within the window, in order to compress the original data and retain the time series trend; environmental data needs to be encoded and mapped based on geographic information system (GIS) technology for collection points, such as converting GPS coordinates into specific administrative area codes or pollution level category labels; biometric data such as voiceprints or gene sequences are originally unstructured signals and need to be generated into vector expressions through deep feature extraction models (such as voiceprint CNN or gene feature Autoencoder).

[0105] After standardization is completed, the four types of data need to be aligned under a unified spatial representation to ensure their consistency in time granularity, field structure, and input dimension, and then spliced together to form an initial input data set with a unified structure for subsequent multimodal fusion of the model.

[0106] The implementation of the initial input dataset can be flexibly adjusted in different application scenarios. For example, in high-frequency sampling scenarios, the step size and window width of the sliding window can be optimized based on the frequency of user device data. For example, if heart rate is sampled at one-minute intervals, a 10-minute sliding window can be used to extract the mean and variance. Attribute data such as occupation fields can be mapped and encoded using the International Standard Classification of Occupations (ISCO). For voiceprint data, a convolutional neural network can be used to generate voiceprint vectors based on the spectrogram, while genetic data can be compressed into fixed-dimensional vectors through one-dimensional convolution or variable-length sequences.

[0107] In resource-constrained scenarios, raw data can be uploaded to cloud servers for standardized processing and splicing, or lightweight feature extraction modules can be executed on local edge devices. For example, vectorized models deployed using TensorFlow Lite can be used to implement edge biometric processing and synchronize structured data to the model analysis module through message queues.

[0108] The alignment method can also be adjusted according to the input form of the target model. For example, if the target model uses the Transformer structure, the splicing method must ensure the separation of each type of modal data in positional encoding; if the CNN structure is used, each modality should be filled into a unified matrix dimension.

[0109] This implementation reduces the dimensional complexity of the original data through structural standardization and vectorized representation, while preserving temporal, spatial, and individual variation characteristics, enhancing the model's expressive power in nonlinear learning. Cross-modal alignment further ensures spatiotemporal consistency across different data, enabling the model to accurately perceive potential connections between variables during training and inference, thereby improving the accuracy of potential event analysis.

[0110] In one embodiment, the above step S20 includes:

[0111] S201, performing named entity recognition on a user's text record using a pre-trained language model to extract entity information from the text record;

[0112] S202, performing semantic relationship extraction processing on the entity information to generate textual supplementary information including entity types and association relationships;

[0113] S203, performing cross-modal spatiotemporal alignment of the text supplementary information with the attribute data and monitoring data in the initial input data set to generate aligned attribute data, aligned monitoring data, and aligned text supplementary information;

[0114] S204 , performing multimodal feature splicing on the aligned attribute data, aligned monitoring data, aligned text supplementary information, and the environmental data and biometric data in the initial input data set to generate the unified input data.

[0115] In this embodiment, text records can be sourced from unstructured or semi-structured sources such as electronic medical records, physical examination reports, user-completed health questionnaires, service interaction logs, or financial activity logs. These types of text often contain content with medical or behavioral semantics. By using a pre-trained language model for text processing, we can maximize the extraction of valuable information for individual status analysis.

[0116] A pre-trained language model refers to a neural network language representation model that is pre-trained on a large-scale corpus. Typical examples include BERT, RoBERTa, or the medical-specific ClinicalBERT and BioBERT. This type of model has powerful semantic representation capabilities and can identify entity terms and their types in sentences. In this step, the named entity recognition module converts user text records into structured entity labels, including but not limited to health risk-related labels (such as "high blood pressure"), physiological indicators (such as "blood sugar value 7.2mmol / L"), and behavioral operation recommendations (such as "recommended salt-reducing diet"). These entity information is extracted as standardized fields.

[0117] Semantic relationship extraction further processes this entity information. Through deep dependency parsing or structural modeling methods based on graph neural networks, it identifies semantic relationships between different entities, such as the causal relationship between "abnormal blood sugar levels" and "diabetes risk." This process transforms named entities and their contextual relationships into structured supplementary information, including fields such as entity type, trigger words, and associated dimensions, thereby producing intermediate results that better meet the input requirements of deep learning models.

[0118] The cross-modal spatiotemporal alignment mechanism addresses the temporal and semantic differences between textual information and data from different sources in the initial input dataset. By constructing a timestamp index and contextual semantic mapping rules, the supplementary textual information is aligned with attribute data (such as age and occupation) and monitoring data (such as heart rate series). This alignment ensures that the data in the subsequent splicing phase is consistent across time windows and feature dimensions, preventing information misalignment.

[0119] Finally, the aligned textual supplementary information is combined with the environmental data (such as air quality) and biometric data (such as gene loci or fingerprint vectors) in the initial input dataset for multimodal feature concatenation. This concatenation can be performed at the feature or representation level, using common methods such as vector concatenation, weighted fusion, or tensor combination. This unified input dataset serves as the foundational input structure for subsequent intelligent decision-making model processing.

[0120] In one implementation, the named entity recognition module uses a sequence labeling model fine-tuned based on the BERT architecture. A dictionary is constructed for specialized terminology in the medical or financial fields to enhance the model's boundary recognition capabilities. The model input is user-submitted questionnaire text or system log output, and the output is a labeled entity sequence, such as "[diabetes]_disease name" or "[fasting blood glucose]_physiological indicator."

[0121] In another implementation, semantic relationship extraction uses a Transformer-based relational classifier, which takes as input entity pairs and context sentence vectors and outputs the semantic type between entities (e.g., causal, correlation, or recommendation). In practice, the syntactic structure between "elevated blood pressure" and "medication control" is deconstructed and identified as a "regulatory recommendation" type.

[0122] In terms of alignment strategy, a sliding time window mechanism can be used to align monitoring data with similar timestamps with the descriptive information in the text; at the same time, the Euclidean distance or cosine similarity between the sentence vector and the structured indicator vector is introduced for semantic alignment judgment.

[0123] In terms of splicing strategies, feature flattening and normalization mechanisms can be used to unify attribute data, aligned monitoring data, text features, environmental vectors, and biological features into standard dimensions, and then spliced together as a unified input.

[0124] Example: In the healthcare sector, when a user uploads a medical examination record containing the description "recent persistent high blood pressure and frequent headaches," the named entity recognition module identifies "high blood pressure" as a disease entity and "headache" as a symptom entity. The semantic extraction module then identifies the causal relationship between "high blood pressure" and "headache." Combined with abnormal blood pressure trends in the user's monitoring data, the system aligns the text information with the real-time monitoring data, concatenates them, and feeds them into a deep learning model for analysis.

[0125] In the financial field, when the system recognizes the expression "credit card overdue and has been repaid in installments" in the user's notes, it can extract "credit card overdue" as the behavior entity and "installment repayment" as the operation instruction; and perform time alignment and splicing analysis with the user's financial attribute data (such as credit score, repayment period) to assist the system in generating individual credit risk assessments and intelligent repayment recommendations.

[0126] This embodiment utilizes semantic parsing of text records and a structured information fusion mechanism to enable the model to process individual health or behavioral status, moving beyond numerical or sensor data. Instead, it can fully utilize the subjective descriptions, medical context, and behavioral preferences contained in unstructured language content. This enhances the model's ability to understand the context of potential events and significantly improves overall analytical accuracy and interpretability.

[0127] In one embodiment, the above step S30 includes:

[0128] S301, extracting spatial correlation features of attribute data and biometric data in the unified input data set through a convolutional neural network module in the intelligent decision-making model;

[0129] S302, extracting time series dependency features of the monitoring data in the unified input data set through a recurrent neural network module in the intelligent decision-making model;

[0130] S303, dynamically assigning weights to the environment data and text supplementary information in the unified input data set through the attention mechanism module in the intelligent decision-making model to generate an environment-text association weight matrix;

[0131] S304: Inputting the spatial correlation features, time series dependency features, and environment and text correlation weight matrix into the multimodal fusion module in the intelligent decision-making model to perform feature-level fusion and dimensionality reduction processing to generate a multimodal joint feature vector;

[0132] S305: Input the multimodal joint feature vector into the output layer of the intelligent decision-making model to generate a potential event analysis result of the individual state.

[0133] In this embodiment, the core of this phase is to perform deep modeling on the fused, unified input dataset and select appropriate neural network structures based on the characteristics of different data modalities to complete the mining, fusion, and analysis of high-dimensional features. This process is based on an intelligent decision-making model and is constructed in modules, each corresponding to the characteristic patterns of different types of input data.

[0134] The unified input dataset includes user attribute data, biometric data, monitoring data, environmental data, and supplementary textual information. These data sources exhibit highly heterogeneous temporal, spatial, and semantic representations. Attribute data includes static structured fields such as gender, age, and occupation, while biometric data is primarily derived from high-dimensional representations with strong spatial distributions, such as fingerprints and gene expression. This type of data often contains significant patterns in local regions, so a convolutional neural network module is used for processing. Convolutional neural networks use sliding convolution kernels to identify local dependencies in spatial distributions, thereby extracting spatial correlation features from this data, such as expression commonalities between different gene segments or regional density features in fingerprint patterns.

[0135] Monitoring data primarily consists of time-series data such as heart rate, blood pressure, and exercise intensity, which exhibit strong temporal dependencies. The temporal characteristics of this data are difficult to extract using traditional methods, so a recurrent neural network module is used to model its temporal evolution. By sharing parameters across time steps and maintaining state memory, recurrent neural networks can identify patterns in a user's short- and long-term physiological changes and extract their time-series dependency characteristics.

[0136] Environmental data and supplementary textual information are characterized by discontinuous context and diverse semantic orientations. This leads to asymmetric correlations between different environmental variables (such as air quality and geographic location) and linguistic expressions. To address this, an attention mechanism module cross-models the relationship between environmental data and textual content, dynamically learning the importance of each type of data for potential event analysis and outputting a weight matrix. This module enables active focusing based on environmental events or semantic context, highlighting key information.

[0137] After modeling the three aforementioned features in the spatial, temporal, and semantic dimensions, they are fed into the multimodal fusion module. This module transforms the representation of data from different modalities into a unified one at the feature level, resolving issues of inconsistent dimensions and scales. Fusion can employ gating mechanisms, weighted concatenation, or low-rank projection, ultimately compressing these features into a fixed-dimensional multimodal joint feature vector.

[0138] This feature vector, serving as a high-dimensional semantic representation, is fed into the model's output layer for subsequent classification or regression inference, generating an analysis of potential events in the individual's state. This analysis can be presented as multiple labels (e.g., high risk, medium risk, normal) or as continuous probability predictions to support downstream interpretation, recommendation, and presentation.

[0139] In one implementation, attribute data and biometric data are first embedded as two-dimensional tensors and then fed into a standard convolutional neural network architecture, such as a stack of two 3×3 convolutional layers and a max pooling layer, to extract local spatial patterns. The output spatial feature vector is flattened into a one-dimensional vector for later use.

[0140] For monitoring data, a bidirectional long short-term memory (Bi-LSTM) network is used to model the previous and next states of the time series to enhance the ability to identify sudden changes in indicators. The network outputs the state vector of the last time step as a time-dependent representation of the monitoring data.

[0141] Before input, contextual data and supplementary textual information are separately encoded as vectors. The supplementary textual information is fed into the BERT model to output sentence vectors, while the contextual data is normalized and encoded as continuous-valued vectors. These two vectors are fed into an interactive layer with an attention gating mechanism. Attention weights are calculated based on the current task objective, and the original input is weighted and reconstructed, outputting a weight matrix representing its semantic strength.

[0142] The outputs of all submodules are fed into a fusion network, which includes a multi-layer perceptron (MLP) for feature fusion, normalization, and dimensionality reduction. The network ultimately outputs a 128-dimensional multimodal joint feature vector. This vector is fed into the output layer, which can use a Softmax function for classification prediction or a Sigmoid function with mean squared error loss for numerical regression of risk probability.

[0143] This embodiment uses spatial structure to identify local commonalities between individual attributes and biometrics, uses time series modeling to analyze trends in continuously monitored data, and integrates subjective descriptions with the external context through semantic focus, ultimately forming a unified analytical representation. This structure significantly enhances the fine-grained recognition capabilities and contextual interpretability of latent state analysis, making it particularly suitable for addressing the generalization difficulties caused by high-dimensional, heterogeneous data.

[0144] In one embodiment, the above step S40 includes:

[0145] S401, determining the potential event level using a preset level classification strategy based on the probability distribution in the potential event analysis result;

[0146] S402: Analyze the potential event analysis results to identify characteristic dimensions that meet a predefined importance screening strategy, and based on the characteristic dimensions, correlate the monitoring data, environmental data, and attribute data in the unified input data set to determine the main influencing factors corresponding to the characteristic dimensions.

[0147] S403, matching corresponding intervention suggestions through a predefined intervention strategy library based on the potential event level and main influencing factors;

[0148] S404: Input the potential event level, main influencing factors and intervention suggestions into a visualization rendering engine to generate a static visualization report including static charts, heat maps and / or text summaries.

[0149] In this example, model analysis results are converted into understandable and actionable visual representations to help users identify the severity, impact sources, and response strategies of potential state risks. Ultimately, reports are output in the form of static charts, heat maps, or text summaries. This generation process relies on four key mechanisms: level determination, feature analysis, strategy matching, and visual rendering.

[0150] First, the probability distribution from the potential event analysis results is processed and a grading strategy is used to determine the potential event's level. The probability distribution can be a softmax multi-classification vector output by the intelligent decision-making model or a single sigmoid output value. The grading strategy can use an interval mapping approach, for example, dividing the risk probability into three levels: low, medium, and high. This can be done using a fixed threshold or quantiles based on the data distribution, thereby converting continuous predictions to discrete levels.

[0151] After the level division, the stage of parsing and interpreting the feature dimensions begins. The feature dimensions referred to here refer to the various dimensions in the multimodal joint feature vector output by the middle layer of the intelligent decision-making model during the potential event analysis process. Each dimension represents the importance or relevance of a certain type of input signal. According to the predefined importance screening strategy, several dimensions with high contribution rankings can be selected as significant influencing sources. The screening strategy can be based on weight thresholds, gradient return values, or SHAP values. Subsequently, the system establishes an index relationship between the selected feature dimensions and the monitoring data, environmental data, and attribute data in the unified input data set, and traces back to their source fields to extract the main influencing factors. The expression of the influencing factors can be a combination of the indicator name and the deviation direction, such as "PM2.5 concentration is high" and "heart rate fluctuations are large at night".

[0152] After identifying the levels and influencing factors, the system matches the appropriate intervention suggestions from the intervention strategy library based on the combined conditions of the two. The strategy library is a predefined structure that contains a set of rules for mapping multiple groups of influencing factors and level combinations to intervention measures. The matching process takes into account the age, gender, and biometric data in the user attributes to ensure that the output intervention suggestions have personalized adaptation capabilities. The content of the suggestions may include behavioral suggestions (such as "increase outdoor activity time"), medical suggestions (such as "regular electrocardiogram testing is recommended"), or daily adjustment suggestions (such as "adjust indoor humidity").

[0153] Finally, all information, including potential event levels, key influencing factors, and personalized intervention recommendations, is fed into a visualization rendering engine to generate a user-friendly static visualization report. The rendering engine can encode level information using color blocks, present influencing factors using bar charts or heat maps, and display recommendations using modular text summaries. The generated static report can be packaged in HTML, PDF, or image formats for display or archiving.

[0154] In one implementation, the model outputs a continuous probability of a potential event, and the grading strategy uses a fixed threshold method. For example, less than 0.3 is low risk, between 0.3 and 0.7 is medium risk, and above 0.7 is high risk. The grading results are color-coded (such as green, yellow, and red) for chart rendering.

[0155] During the feature screening phase, the system extracts the top five weighted dimensions from the 128-dimensional joint feature vector and, using an intermediate-layer weight tracking algorithm, maps them back to the original input fields. For example, dimension 17 has the highest weight, and tracking it correlates with "sleep interruption frequency," identifying it as one of the primary influencing factors.

[0156] The intervention strategy library is stored in a key-value pair structure, where the key is a combination of risk level and impact factor, and the value is a set of recommended items. The matching mechanism is based on exact or fuzzy matching, and user attributes are used for secondary filtering. For example, for male users with "lack of exercise," "increase aerobic exercise" is recommended, while for older female users, "increase walking frequency" is recommended.

[0157] The rendering engine uses Matplotlib or ECharts to display the grade information as a color block background, displays the influencing factors as variation indicators in bar charts, and encapsulates the intervention recommendations into paragraph text to form a logically unified layout with the chart area.

[0158] Example: In the healthcare sector, a user uploaded physiological indicator data and daily life descriptions for the past seven days. The model predicted a high risk of cardiovascular abnormalities. The risk was determined to be high, with the most heavily weighted feature dimensions being "severe heart rate fluctuations at night" and "large fluctuations in PM2.5 concentrations." The system matched the strategy library and generated intervention recommendations such as "avoid travel during high-pollution weather" and "reducing high-intensity exercise at night." The system then presented this information as a text summary with a dynamic trend chart.

[0159] In the fintech sector, after a client's financial behavior log and environmental information are entered into the system, the model analyzes the client's exposure to moderate financial risk. High-weighted dimensions correspond to "shortened transaction intervals" and "changed income sources." The strategy library then recommends "allocating short-term, low-risk financial instruments" and prompts the client to strengthen their repayment plan. The system generates a visual report combining the assessment level, financial indicator trends, and recommended strategies, and sends it to the client to assist in developing a plan to adjust their financial behavior.

[0160] This embodiment improves the interpretability and operability of potential risk results by hierarchically classifying the probability distribution in the potential event analysis results, combining the importance screening of feature dimensions with the cross-modal association of the original input data, and further matching the intervention strategy library based on the user's individual attributes and visually rendering the output. It realizes a closed-loop communication from model prediction to multi-dimensional impact analysis to personalized intervention recommendations, which helps users intuitively understand the potential status, identify the source of impact and take effective intervention actions, thereby enhancing the practical value and response efficiency of the analysis results.

[0161] In one embodiment, after the above step S40, the method further includes:

[0162] S405, performing piecewise encryption on the attribute data, monitoring data, environmental data, biometric data, and text supplementary information in the unified input data set to generate multiple encrypted data blocks;

[0163] S406, binding the encrypted data block with the user's unique identifier to generate a data shard hash chain;

[0164] S407, distributing and storing the data shard hash chain to the blockchain network nodes, and generating a storage verification record;

[0165] S408, after verifying the access rights of the data requester through the smart contract of the blockchain network, the encrypted data block is decrypted and retrieved.

[0166] In this embodiment, after generating a visual report containing potential event levels, main influencing factors and intervention recommendations, in order to achieve security protection of individual user privacy information, the encryption and access control process of multimodal data is further executed. First, it is necessary to perform structured segmentation on the various types of data content in the unified input data set that have been used for model analysis, including attribute data, monitoring data, environmental data, biometric data and text supplementary information generated by text semantic parsing. These data types may include static identity information (such as age, gender), dynamic physiological monitoring values (such as heart rate, sleep), sensitive scene identifiers (such as genetic sequences, life trajectories) and behavioral suggestion trigger conditions that may be obtained through semantic reasoning, so they need to be treated as sensitive data units.

[0167] During the encryption phase, a sharding encryption strategy is used to split and independently encrypt data with minimal granularity. Each sharded data type can be encrypted using a symmetric encryption method (such as AES), combined with an initialization vector and key derivation algorithm to generate multiple encrypted data blocks. To ensure data uniqueness and traceability, each data shard is bound to a unique user identifier to generate a hash chain. This identifier is calculated from the irreversible hash value of the user's attribute data, ensuring that the original identity information cannot be reversed.

[0168] The generated hash chain and its corresponding data blocks are then written into a distributed blockchain storage system. Within this system, each data write operation forms an immutable chain structure, automatically recording the data structure, timestamp, and source identifier. The system pre-sets data call rules through a smart contract mechanism. When a data request is made, the accessor's identity and the purpose of the call are first verified to ensure compliance with the contract terms. Only after verification is successful is the target data block decrypted.

[0169] In practical implementation, the encryption policy module can be embedded into the output process node of the visualization report generation module to implement immediate encryption of data after use. AES-256 can be used as the encryption algorithm, combined with multi-party secure computation methods such as Shamir's Secret Sharing to implement threshold-based key distribution. The encrypted data blocks are stored as hash index tables on blockchain nodes built on a consortium chain architecture, such as those using Hyperledger Fabric or Quorum.

[0170] A user's unique identifier can be generated by calculating a hash value from static fields in the user's registration information (such as mobile phone number or ID number), or by using a UUID generated internally by the system and persistently bound. The hash chain structure of data sharding uses a Merkle Tree approach, supporting segmented verification and rapid auditing.

[0171] During the data access process, smart contracts are responsible for executing policy-based access decisions. For example, hospital access requests must include a verified digital certificate and the operation request must match the patient's authorization record. Once access rights are successfully determined by the contract, the system decrypts the required data block using the bound encryption key, retrieves it, and transmits it.

[0172] Example: In the healthcare business, a platform generates an individual status analysis report for an elderly user. The report identifies that insufficient exercise and sleep fluctuations have a significant impact on the user's potential health risk level. After the system completes report generation, it immediately encrypts the fields containing the user's genetic information, continuous blood pressure monitoring data, and natural language medical record text. The generated data hash chain is written to the blockchain node of the alliance hospital, and is set to allow only medical institutions with user authorization tokens to query it. After that, if another medical service system attempts to call the data, it must pass smart contract verification and provide a legal authorization certificate before access.

[0173] In the fintech sector, an insurance company builds a potential claims warning model by analyzing customer behavior logs, device usage, and semantic feedback records, generating a risk level and recommended list for each user. To prevent the misuse of this user data after model training, the system encrypts and shards the user's consumption history, device fingerprints, and call behavior, and writes this data to the blockchain platform using a hash chain bound to the user's identity. If the business department subsequently wishes to reload part of the user's training data, the system contract must verify the legitimacy and authorization status of the request.

[0174] This embodiment implements sharded encryption of the multimodal data used for analysis within a unified input dataset. After encryption, it binds the encrypted data to a user ID to generate a data hash chain. This is then combined with blockchain nodes for distributed storage and access verification. This effectively implements hierarchical protection of user sensitive information and a fully auditable data operation security mechanism. This not only enhances the privacy protection of individual data but also strengthens the credibility of data access control in multi-terminal, cross-platform, and cross-subject interaction scenarios.

[0175] In one embodiment, the above step S50 includes:

[0176] S501, adding an interactive logic module to the static charts, heat maps and text summaries in the visualization report to generate dynamic interactive components;

[0177] S502, binding the dynamic interactive component with the monitoring data in the unified input data set to generate an interactive view;

[0178] S503, performing cross-terminal layout adaptation on the interactive view to generate an interactive report matching the user terminal type;

[0179] S504: Send the interactive report to the user terminal, and capture the interactive operation instruction through the event monitoring interface of the user terminal to trigger the update of the interactive view and the adjustment of the intervention suggestion.

[0180] In this embodiment, after completing the generation of individual status analysis results and the construction of static visualization reports, in order to enhance user participation, feedback efficiency, and the ability to dynamically update intervention recommendations, it is necessary to convert the static visualization results into interactive and actionable content. This process first performs component-level enhancements on static charts, heat maps, and text summaries, embeds them into an interactive logic module, and constructs them into dynamic interactive components. The interactive logic module should support, but is not limited to, three basic functions: one is data drilling, which is used to trigger the presentation of detailed information of a specific dimension after clicking on a chart area; the second is view zooming, which is suitable for local zooming or overall bird's-eye view switching of geographic information heat maps; and the third is filter condition response, which allows users to specify time periods, data types, or threshold conditions to re-filter the view.

[0181] After generating the interactive logic module, it is bound to the built and real-time updated monitoring data in the unified input dataset to form an interactive view. This view can automatically refresh based on the binding mechanism. For example, when a wearable device uploads a new batch of heart rate or sleep quality data, the line chart or dynamic heat map on the front-end page will immediately reflect the corresponding value changes.

[0182] Subsequently, to accommodate the differences in display resolution, interactive capabilities, and transmission performance across different types of terminals (such as smartphones, tablets, web browsers, and embedded devices), the interactive view needs to be adapted across terminals. This adaptation process dynamically selects layout templates, chart scaling parameters, and response event types based on terminal resolution, input method, and system type, ultimately generating a complete, interactive report that matches the target terminal.

[0183] Ultimately, the constructed interactive report is sent to the target terminal via supported data transmission channels, such as HTTP(S), WebSocket, and MQTT. On the terminal side, the event listening interface continuously captures user actions (clicks, swipes, filter selections, etc.), triggering the redrawing of the front-end page and invoking the back-end recommendation engine or policy library to perform a secondary update of the intervention suggestions, ensuring the real-time and adaptability of the recommended content.

[0184] In practice, front-end frameworks (such as React and Vue.js) can be used to build interactive logic modules, converting static chart components into dynamic ones and binding graphical elements to user behavior event listeners. Data drilling logic can be based on tree structures or drill-down path mapping, with view scaling achieved through SVG transformations or Canvas redrawing, and filter condition responses updated bidirectionally through bound variables.

[0185] Monitoring data binding can establish a continuous connection between the front-end and back-end data streams through the WebSocket channel. The back-end pushes new physiological monitoring values at a certain frequency. The front-end view component subscribes to data changes and triggers the chart update function to achieve true dynamic data rendering.

[0186] Cross-device layout adaptation leverages CSS media queries and JavaScript device detection to conditionally switch elements such as page structure, font size, and graphic dimensions, enabling, for example, a vertical scrolling layout on mobile devices and a multi-column layout on desktops. To improve transmission efficiency, charts can be encoded using a vector compression encoding format before transmission, reducing bandwidth overhead.

[0187] During the data transmission phase, HTTPS is preferred for channel security. WebSocket bidirectional communication can also be enabled for scenarios requiring push and return data. On the user terminal side, event listeners, such as addEventListener('click',...), are bound to capture interaction commands. Data binding updates the chart state, submits the filtered information to the server, and the server matches new intervention recommendations and returns them to the terminal page.

[0188] Example: In the healthcare business scenario, a chronic disease management platform serves people at high risk of hypertension. The platform first continuously collects monitoring data such as heart rate, blood pressure, number of steps, and sleep time from the user's bound smart wearable device; and obtains the user's basic attribute data (such as age, gender, occupation) and historical medical history information through the local health record interface; simultaneously, through user authorization, it accesses the air quality sensor's API to obtain the real-time environmental data of the user's residence, including PM2.5, humidity, and air pressure; the system also reads biometric data provided by fingerprint recognition terminals and genetic testing service providers (such as family susceptibility gene mutations). At the same time, the user fills out the medical notes and medication logs for the past three months on the App. The system uses a pre-trained language model to perform named entity recognition on the text content, extracts key information such as "continuous high blood pressure exceeding 140" and "daily medication without interruption", and constructs an entity relationship diagram through a semantic parsing model to further generate structured text supplementary information, such as "abnormal high blood pressure has no direct correlation with medication compliance." After cross-modal spatiotemporal alignment of all structured information, it is then integrated with attribute data, monitoring data, environmental parameters, and biometric encoding vectors to form a complete, unified input dataset. This dataset is then fed into a multi-layered intelligent decision-making model. A CNN extracts spatial static associations, a RNN captures the temporal dependencies of physiological fluctuations, and an attention module calculates the response weights between environmental changes and textual descriptions. Finally, a fusion module performs feature compression and outputs a potential event analysis of the individual's status. The analysis results indicate that the user has had a potential warning of "high cardiovascular risk" over the past two weeks. The system parses the model output and determines a risk score of 0.86, with key feature dimensions being "excessive PM2.5 levels," "increased nighttime blood pressure fluctuations," and "no recovery after midday exercise." The system automatically correlates the user's historical data and determines that the primary contributing factors are elevated environmental pollution levels and abnormal nocturnal sympathetic nervous system responses. After matching the intervention strategy library, recommendations are pushed, including "replacing air purification equipment," "adjusting exercise time to before 3 p.m.," and "retesting blood pressure at a fixed time at night." This information is presented to the user's app front-end via dynamic charts and heat maps. Users can click on a specific day's blood pressure trend chart to view the impact of air quality on their physiological responses, filter different time windows, and view the system's recommended adjustment plans. All visualizations are pushed to the user's terminal in an interactive format via an encrypted WebSocket channel. Following user actions, the system synchronously updates the chart status and recommended intervention strategies, encrypts the data in shards, and records it on-chain to ensure access authorization, integrity, and immutability of health information.

[0189] In a financial business scenario, a credit risk management platform provides personalized risk analysis services for credit loan users in first-tier cities. The system first obtains basic user information from the financial institution's internal database, such as age, job type, past default history, and loan amount history. It then accesses their public credit report and social security payment records through an external interface. Real-time user behavior data (such as bank card transaction records and POS device payment records) is collected as monitoring data and combined with location data and weather conditions provided by the phone's GPS to form an environmental data dimension. Users can also choose to bind voiceprint verification and finger vein recognition as biometric data sources to ensure accurate attribution of user behavior data. The system also accesses text feedback submitted by users on the credit complaint platform, such as explanatory text such as "Salary payments suspended on March 10 due to company financial adjustments" or "Voluntarily applied for an extension." Using a financial semantic recognition model, the system extracts keywords such as "income interruption" and "Voluntarily extended" and identifies their correlation with time points and transaction behavior, creating a supplementary field called "Abnormal short-term and medium-term liquidity but stable subjective intention." This information, after undergoing structured encoding, geographic data mapping, and feature extraction, forms a unified input dataset, which is fed into a deep neural network architecture specifically designed for risk modeling in the financial industry. The convolutional module extracts the distribution of risk labels from the applicant's historical behavioral patterns, while the recurrent network captures trends in account activity and inflow / outflow structure over the past 60 days. The attention mechanism identifies a strong match between frequent geographic location changes and the description of "business trip and job transfer" in the text, indicating that the risk is not malicious debt evasion. The final analysis results indicate that this customer falls into a medium-risk category characterized by "rising short-term credit pressure but stable long-term credit intentions." The model identifies "high-frequency cross-regional transactions," "increasing credit card overdrafts," and "textual descriptions of intermittent income" as key feature dimensions. Based on this information, the system recommends risk control interventions such as "temporarily reducing credit limits," "reducing collection frequency and adding automated negotiation channels," and "promoting installment wealth management products to users as an alternative to short-term loan cash flow." This information is rendered into a risk distribution map, a behavioral impact path map, and intervention plan cards using a visualization engine. These are then adapted to the terminal type and sent to the bank's internal approval system and the user's app. Users can review their credit trends online and interact with credit specialists to adjust recommended solutions. The system will simultaneously encrypt the interactive behavior and analysis data and store them through the alliance chain. Risk control personnel can access and view them after the permission verification is passed, effectively ensuring data auditability and compliance.

[0190] This embodiment enhances static visualization reports into dynamic components with user interaction capabilities, binding them to monitoring data from a unified input dataset in real time. Combined with terminal adaptation strategies and interactive monitoring mechanisms, the system enables real-time updates of report content and dynamic optimization of intervention recommendations. This not only increases user engagement but also enhances the system's responsiveness to individual status changes, making the resulting behavioral recommendations more timely and personalized, significantly improving the implementation of multimodal analysis results and the efficiency of actual interventions.

[0191] In one embodiment, a device for analyzing events based on multimodal data is provided, and the device for analyzing events based on multimodal data corresponds one-to-one to the method for analyzing events based on multimodal data in the above embodiment. Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the multimodal data-based event analysis device of the present invention. It includes a multimodal data acquisition module 10, a semantic information fusion module 20, a multimodal in-depth analysis module 30, a result visualization module 40, and a report delivery module 50. Each functional module is described in detail below:

[0192] The multimodal data acquisition module 10 is used to obtain the user's attribute data, monitoring data, environmental data and biometric data and perform normalization processing to generate an initial input data set;

[0193] Semantic information fusion module 20, used to perform semantic analysis on the user's text records, extract key information and fuse it with the initial input data set to generate a unified input data set;

[0194] A multimodal in-depth analysis module 30 is configured to perform multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual states;

[0195] A result visualization module 40 is used to generate a visualization report including the potential event level, main influencing factors and intervention suggestions based on the potential event analysis results;

[0196] The report delivery module 50 is configured to send the visualization report to a user terminal.

[0197] In one embodiment, the multimodal data acquisition module 10 is specifically configured to:

[0198] Obtain user attribute data from the local database or external interface;

[0199] Collect user monitoring data from wearable devices or IoT sensors;

[0200] Obtain user environmental data through environmental monitoring equipment;

[0201] Obtaining biometric data of users through biometric devices;

[0202] Format encoding of user basic information and historical records in the attribute data to generate encoded attribute data;

[0203] Performing sliding window feature extraction on the time series change indicators in the monitoring data to generate monitoring data after feature extraction;

[0204] Performing geocoding mapping on the spatial distribution parameters in the environmental data to generate mapped environmental data;

[0205] Performing feature vectorization processing on heterogeneous information in the biometric data to generate vectorized biometric data;

[0206] The encoded attribute data, the feature-extracted monitoring data, the mapped environmental data, and the vectorized biometric data are cross-modally aligned and spliced to generate the initial input data set.

[0207] In one embodiment, the semantic information fusion module 20 is specifically configured to:

[0208] Performing named entity recognition on the user's text records using a pre-trained language model to extract entity information from the text records;

[0209] Performing semantic relationship extraction on the entity information to generate textual supplementary information including entity types and association relationships;

[0210] Performing cross-modal spatiotemporal alignment of the text supplementary information with the attribute data and monitoring data in the initial input data set to generate aligned attribute data, aligned monitoring data, and aligned text supplementary information;

[0211] The aligned attribute data, aligned monitoring data, aligned text supplementary information and the environmental data and biometric data in the initial input data set are subjected to multimodal feature splicing to generate the unified input data.

[0212] In one embodiment, the multimodal depth analysis module 30 is specifically configured to:

[0213] Extracting spatial correlation features of attribute data and biometric data in the unified input data set through a convolutional neural network module in the intelligent decision-making model;

[0214] Extracting time series dependency features of the monitoring data in the unified input data set through a recurrent neural network module in the intelligent decision-making model;

[0215] Dynamically assign weights to the environmental data and text supplementary information in the unified input data set through the attention mechanism module in the intelligent decision-making model to generate an environmental and text association weight matrix;

[0216] Inputting the spatial correlation features, time series dependency features, and environment and text correlation weight matrix into the multimodal fusion module in the intelligent decision-making model, performing feature-level fusion and dimensionality reduction processing, and generating a multimodal joint feature vector;

[0217] The multimodal joint feature vector is input into the output layer of the intelligent decision-making model to generate a potential event analysis result of the individual state.

[0218] In one embodiment, the result visualization module 40 is specifically configured to:

[0219] Determining the potential event level using a preset level classification strategy based on the probability distribution of the potential event analysis results;

[0220] Analyzing the characteristic dimensions that meet the predefined importance screening strategy from the potential event analysis results, and correlating the monitoring data, environmental data, and attribute data in the unified input data set based on the characteristic dimensions to determine the main influencing factors corresponding to the characteristic dimensions;

[0221] Based on the potential event level and main influencing factors, matching corresponding intervention suggestions through a predefined intervention strategy library;

[0222] The potential event level, main influencing factors and intervention recommendations are input into a visualization rendering engine to generate a static visualization report including static charts, heat maps and / or text summaries.

[0223] In one embodiment, the result visualization module 40 is specifically configured to:

[0224] Performing fragmented encryption on the attribute data, monitoring data, environmental data, biometric data, and text supplementary information in the unified input data set to generate multiple encrypted data blocks;

[0225] Binding the encrypted data block to the user's unique identifier to generate a data shard hash chain;

[0226] Distribute and store the data shard hash chain to the blockchain network nodes and generate storage verification records;

[0227] After verifying the access rights of the data requester through the smart contract of the blockchain network, the encrypted data block is decrypted and retrieved.

[0228] In one embodiment, the report delivery module 50 is specifically configured to:

[0229] Adding interactive logic modules to the static charts, heat maps and text summaries in the visualization report to generate dynamic interactive components;

[0230] Binding the dynamic interactive component with the monitoring data in the unified input data set to generate an interactive view;

[0231] Adapting the interactive view to a cross-terminal layout to generate an interactive report that matches the user terminal type;

[0232] The interactive report is sent to the user terminal, and the interactive operation instruction is captured through the event monitoring interface of the user terminal, triggering the update of the interactive view and the adjustment of the intervention suggestion.

[0233] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an event analysis method based on multimodal data.

[0234] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a method for event analysis based on multimodal data.

[0235] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0236] Obtain user attribute data, monitoring data, environmental data, and biometric data and perform standardization processing to generate an initial input data set;

[0237] Performing semantic analysis on the user's text records, extracting key information and fusing it with the initial input data set to generate a unified input data set;

[0238] Performing multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual states;

[0239] Based on the potential event analysis results, generate a visual report including the potential event level, main influencing factors and intervention suggestions;

[0240] The visualization report is sent to a user terminal.

[0241] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0242] Obtain user attribute data, monitoring data, environmental data, and biometric data and perform standardization processing to generate an initial input data set;

[0243] Performing semantic analysis on the user's text records, extracting key information and fusing it with the initial input data set to generate a unified input data set;

[0244] Performing multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual states;

[0245] Based on the potential event analysis results, generate a visual report including the potential event level, main influencing factors and intervention suggestions;

[0246] The visualization report is sent to a user terminal.

[0247] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0248] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0249] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0250] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An event analysis method based on multimodal data, characterized in that: The following steps are involved: Obtain user attribute data, monitoring data, environmental data, and biometric data and perform standardization processing to generate an initial input data set; Performing semantic analysis on the user's text records, extracting key information and fusing it with the initial input data set to generate a unified input data set; Performing multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual states; Based on the potential event analysis results, generate a visual report including the potential event level, main influencing factors and intervention suggestions; The visualization report is sent to a user terminal.

2. The event analysis method based on multimodal data according to claim 1, characterized in that: Obtain user attribute data, monitoring data, environmental data, and biometric data and perform standardization to generate an initial input data set, including: Obtain user attribute data from the local database or external interface; Collect user monitoring data from wearable devices or IoT sensors; Obtain user environmental data through environmental monitoring equipment; Obtaining biometric data of users through biometric devices; Format encoding of user basic information and historical records in the attribute data to generate encoded attribute data; Performing sliding window feature extraction on the time series change indicators in the monitoring data to generate monitoring data after feature extraction; Performing geocoding mapping on the spatial distribution parameters in the environmental data to generate mapped environmental data; Performing feature vectorization processing on heterogeneous information in the biometric data to generate vectorized biometric data; The encoded attribute data, the feature-extracted monitoring data, the mapped environmental data, and the vectorized biometric data are cross-modally aligned and spliced to generate the initial input data set.

3. The event analysis method based on multimodal data according to claim 1, characterized in that: Perform semantic analysis on the user's text records, extract key information, and fuse it with the initial input dataset to generate a unified input dataset, including: Performing named entity recognition on the user's text records using a pre-trained language model to extract entity information from the text records; Performing semantic relationship extraction on the entity information to generate textual supplementary information including entity types and association relationships; Performing cross-modal spatiotemporal alignment of the text supplementary information with the attribute data and monitoring data in the initial input data set to generate aligned attribute data, aligned monitoring data, and aligned text supplementary information; The aligned attribute data, aligned monitoring data, aligned text supplementary information and the environmental data and biometric data in the initial input data set are subjected to multimodal feature splicing to generate the unified input data.

4. The event analysis method based on multimodal data according to claim 1, characterized in that: The unified input data set is subjected to multimodal data fusion processing through an intelligent decision-making model to generate potential event analysis results of individual states, including: Extracting spatial correlation features of attribute data and biometric data in the unified input data set through a convolutional neural network module in the intelligent decision-making model; Extracting time series dependency features of the monitoring data in the unified input data set through a recurrent neural network module in the intelligent decision-making model; Dynamically assign weights to the environmental data and text supplementary information in the unified input data set through the attention mechanism module in the intelligent decision-making model to generate an environmental and text association weight matrix; Inputting the spatial correlation features, time series dependency features, and environment and text correlation weight matrix into the multimodal fusion module in the intelligent decision-making model, performing feature-level fusion and dimensionality reduction processing, and generating a multimodal joint feature vector; The multimodal joint feature vector is input into the output layer of the intelligent decision-making model to generate a potential event analysis result of the individual state.

5. The event analysis method based on multimodal data according to claim 1, characterized in that: Based on the potential event analysis results, a visual report containing the potential event level, main influencing factors and intervention recommendations is generated, including: Determining the potential event level using a preset level classification strategy based on the probability distribution of the potential event analysis results; Analyzing the characteristic dimensions that meet the predefined importance screening strategy from the potential event analysis results, and correlating the monitoring data, environmental data, and attribute data in the unified input data set based on the characteristic dimensions to determine the main influencing factors corresponding to the characteristic dimensions; Based on the potential event level and main influencing factors, matching corresponding intervention suggestions through a predefined intervention strategy library; The potential event level, main influencing factors and intervention recommendations are input into a visualization rendering engine to generate a static visualization report including static charts, heat maps and / or text summaries.

6. The event analysis method based on multimodal data according to claim 1, characterized in that: Based on the potential event analysis results, a visual report is generated containing the potential event level, main influencing factors and intervention recommendations, which also includes: Performing fragmented encryption on the attribute data, monitoring data, environmental data, biometric data, and text supplementary information in the unified input data set to generate multiple encrypted data blocks; Binding the encrypted data block to the user's unique identifier to generate a data shard hash chain; Distribute and store the data shard hash chain to the blockchain network nodes and generate storage verification records; After verifying the access rights of the data requester through the smart contract of the blockchain network, the encrypted data block is decrypted and retrieved.

7. The event analysis method based on multimodal data according to claim 1, characterized in that: Sending the visualization report to the user terminal includes: Adding interactive logic modules to the static charts, heat maps and text summaries in the visualization report to generate dynamic interactive components; Binding the dynamic interactive component with the monitoring data in the unified input data set to generate an interactive view; Adapting the interactive view to a cross-terminal layout to generate an interactive report that matches the user terminal type; The interactive report is sent to the user terminal, and the interactive operation instruction is captured through the event monitoring interface of the user terminal, triggering the update of the interactive view and the adjustment of the intervention suggestion.

8. An event analysis device based on multimodal data, characterized in that: The event analysis device based on multimodal data includes: Multimodal data acquisition module, used to obtain user attribute data, monitoring data, environmental data and biometric data and perform standardization processing to generate the initial input data set; A semantic information fusion module is used to perform semantic analysis on the user's text records, extract key information and fuse it with the initial input data set to generate a unified input data set; A multimodal in-depth analysis module, configured to perform multimodal data fusion processing on the unified input data set through an intelligent decision-making model to generate potential event analysis results of individual states; A result visualization module is used to generate a visualization report including the potential event level, main influencing factors and intervention suggestions based on the potential event analysis results; The report delivery module is used to send the visualization report to the user terminal.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and an event analysis program based on multimodal data stored in the memory and capable of running on the processor. When the event analysis program based on multimodal data is executed by the processor, the steps of the event analysis method based on multimodal data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores an event analysis program based on multimodal data, and when the event analysis program based on multimodal data is executed by the processor, the steps of the event analysis method based on multimodal data as described in any one of claims 1 to 7 are implemented.

Images