Multi-modal project data analysis method

Abstract

The invention belongs to the technical field of data processing, and discloses a multi-modal project data analysis method, which comprises the following steps: collecting heterogeneous modal project data, carrying out denoising and timestamp correction on each modal project data based on an adaptive multi-scale filtering technology, carrying out cross-modal time alignment, and outputting a multi-modal signal. Constructing a multi-level Boltzmann machine energy network, and modeling the joint probability distribution of the multi-modal signals; a dynamic complexity measurement mechanism based on topological entropy is designed and used for representing project structure complexity, and a topological entropy dynamic field is generated; the topological entropy dynamic field is used as a constraint condition, a multi-dimensional covariant field frame is used, nonlinear coupling and space-time propagation between multi-modal signals are simulated, the multi-modal signals are abstracted into a string vibration mode, local topological defects in the topological entropy dynamic field are recognized, and an abnormal fluctuation map is generated; and the analysis of the complex multi-modal data in the project operation process is more intelligent and controllable.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more particularly, to a multimodal project data analysis method. Background Art

[0002] Patent publication number CN117710117A discloses a multimodal large-scale model-based insurance data analysis method and system. The system obtains image documents submitted by the insured and uses the multimodal large-scale model to extract specified information from the image documents, including the document type, audit items, and fees. The multimodal large-scale model determines whether the items in the extracted specified information are compensation items. If so, the compensation category and compensation ratio are determined. Based on the determined compensation category and compensation ratio, the compensation amount for the image document is calculated. The commercial insurance business processing and compensation results of the image document are presented to commercial insurance staff for confirmation and modification. A desensitized compensation report is generated using the confirmed and modified commercial insurance business processing and compensation results of the image document. This invention eliminates the need for manual intervention for submitted image documents, allowing them to be directly processed and converted into structured data with high accuracy.

[0003] The existing modal project data analysis methods have the following main problems:

[0004] Most existing multimodal data analyses adopt static fusion strategies, which are unable to characterize the complex coupling behaviors between different modalities that evolve over time, and are difficult to reflect the dynamic mutual influence of different information sources during the project process; most existing methods use rule thresholds, sliding statistics or machine learning classifiers for anomaly detection, lack modeling and identification of topological structure change trends in multimodal embedding graphs, are prone to missing or misjudging key evolution nodes, cannot utilize structural information between multimodal modes to enhance the robustness of anomaly detection, have difficulty identifying deep changes in the structural hierarchy, and lack a unified judgment benchmark for the comprehensive judgment of heterogeneous source data.

[0005] Existing graph analysis methods are unable to determine the critical turning point of a certain node during the time evolution process, cannot achieve early warning and explainable tracking of the project system evolution process, lack the ability to understand the structural changes in the graph from the perspective of non-stationary dynamic evolution, and are difficult to determine where and when irreversible evolutionary trends or risk surge points are faced.

[0006] In view of this, the present invention proposes a multimodal project data analysis method to solve the above problems. Summary of the Invention

[0007] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned objectives, the present invention provides the following technical solution: a multimodal project data analysis method, comprising: S1. Collect heterogeneous modal project data, denoise and timestamp correct the data of each modality based on adaptive multi-scale filtering technology, perform cross-modal time alignment, and output multimodal signals; S2. Construct a multi-level Boltzmann machine energy network to model the joint probability distribution of multimodal signals; design a dynamic complexity measurement mechanism based on topological entropy to represent the complexity of project structure and generate a topological entropy dynamic field; S3. Using the topological entropy dynamic field as a constraint, and utilizing the multidimensional covariant field framework, we simulate the nonlinear coupling and spatiotemporal propagation between multimodal signals, abstract the multimodal signals into string vibration modes, identify local topological defects in the topological entropy dynamic field, and generate abnormal fluctuation maps. S4. Drawing on the selective regulation characteristics of glial cells on neural signals and combining them with abnormal fluctuation maps, the weighted coupling strength of each modal signal is dynamically adjusted to perform inter-modal feedback regulation. S5. Based on the weighted multimodal signals, a multimodal causal structure self-construction algorithm is designed. Combined with the topological entropy dynamic field and abnormal fluctuation map, abnormal events are identified, and cross-modal spatiotemporal causal paths are tracked. The identified abnormal events are then transmitted to the data analysis terminal.

[0008] Preferably, the method for collecting heterogeneous modality project data includes: Deploy n data acquisition channels with different perception types at the project site. These channels include image acquisition channels, voice acquisition channels, structured data interface channels, time series signal acquisition channels, and unstructured text acquisition channels to collect heterogeneous modality project data. Heterogeneous modal project data includes image modality, sound modality, equipment operating parameter modality, time series signal modality data and project document modality data; based on data source, physical type and semantic attributes, the modality type of heterogeneous modal project data is identified and labeled to generate a data structure with modality labels.

[0009] Preferably, the method for outputting a multimodal signal includes: A multiscale filter bank is constructed based on the modal type of the heterogeneous modal project data. The multiscale filter bank includes a filter structure based on wavelet transform and empirical mode decomposition to extract signal trends and suppress modal-specific noise. The sampling frequency, noise characteristics, and spectral distribution of each heterogeneous modal project data are analyzed, and the filter scale and adaptive denoising are dynamically adjusted based on the modal-specific error feedback mechanism. Anomaly detection is performed in timestamp order to identify missing values, mutation points and pseudo-synchronization segments, and timestamp correction is performed using interpolation compensation methods. The heterogeneous modal project data with timestamps are input into the cross-modal time alignment algorithm, and nonlinear alignment is performed on the heterogeneous modal project data with different sampling frequencies and synchronization delays, ultimately generating a multimodal signal under a unified time axis.

[0010] Preferably, the method for modeling the joint probability distribution of multimodal signals includes: The multimodal signal is converted into a unified embedding representation, with different embedding strategies adopted for different modalities. After normalization of all modal embeddings, the representation is used as the input of the Boltzmann machine. A multi-layer Boltzmann machine network structure is constructed, where each layer includes learnable latent variable nodes, which are symmetrically connected with the nodes of the previous and next layers to form an undirected graph structure, forming an energy network that depicts the semantic joint distribution of multimodal signals. The energy function of the energy network is designed to characterize the joint activation relationship between modal embedding and latent variables. A regulation factor based on the topological entropy dynamic field is introduced to adjust the sensitivity of the connection weights between modalities. The parameter distribution for joint modeling and reasoning is obtained, and finally the joint probability distribution model of multimodal signals in the joint modal space is output.

[0011] Preferably, the method for generating a topological entropy dynamic field comprises: Based on multimodal signals, a unified multimodal embedding graph is constructed. The nodes in the multimodal embedding graph represent the embedding units of each modal signal, and the edges represent the explicit or implicit associations between modal signals. For any node in the multimodal embedding graph, the topological entropy of the node is defined within a preset sliding time window. The time-varying topological entropy of each node in a multimodal embedded graph is constructed as a spatiotemporal continuous distribution tensor. A wave-like diffusion mechanism is introduced to model the propagation behavior, and the governing partial differential equations for the propagation of topological entropy are defined. The governing partial differential equations are numerically solved using a discretization method based on the graph structure to generate a dynamic field of topological entropy at any time. To identify sudden change points in the topological structure, bifurcation theory is used to determine the stability of node structures in the dynamic field of topological entropy. The local Jacobian matrix of each node in the current window is calculated, and the spectral radius is extracted to determine whether the spectral radius meets the topological bifurcation condition. When the topological bifurcation condition is met, the node is determined to be a topological bifurcation point; all identified topological bifurcation points are combined with their mode type, timestamp and propagation direction to be plotted as a topological entropy anomaly response map.

[0012] Preferably, the method of abstracting the multimodal signal into a string vibration mode includes: Based on denoised and time-aligned multimodal signal data, a sliding time window and modal spatial distribution information are used to construct a multidimensional covariant tensor field. The topological entropy dynamic field is introduced as a modulation constraint into the construction of the multidimensional covariant tensor field to calculate the covariance, mutual information, and high-order statistics between different modes. According to the covariance, mutual information, and high-order statistics between different modes in different time periods, the covariance coefficients are dynamically assigned entropy-guided weighting factors. Under the constraint of topological entropy, the coupling mapping relationship between multimodal signals is modeled by nonlinear kernel functions to generate coupling paths containing time, space and modal dimensions; the coupling paths are abstracted into a multidimensional string vibration model under controlled boundary conditions.

[0013] Preferably, the method for generating an abnormal fluctuation map comprises: Each modal signal is represented as an excitable and coupled vibrating string-like element, and energy propagation and resonance behavior are simulated in the covariant field. The dominant modal frequency, standing wave nodes and phase jump areas are extracted based on the multidimensional string vibration model. Local topological defects that exhibit energy aggregation and sudden propagation in the covariant field are identified, and an abnormal fluctuation map containing abnormal response paths and disturbance area identifiers is constructed.

[0014] Preferably, the method for performing inter-modal feedback adjustment includes: Based on the abnormal fluctuation map, each node in the multimodal embedding graph corresponding to the multimodal signal is identified and labeled to determine the node area with structural abnormalities; for each node, the characteristic parameters are extracted and the regulation strength index of the node at the current moment is calculated; Drawing on the regulatory mechanism of glial cells in the nervous system for synaptic signal transmission, the modal coupling weights between a node and its adjacent nodes are selectively adjusted based on the regulatory strength index of each node. When the abnormal fluctuation trend in a region of any modality is greater than a preset abnormal fluctuation trend threshold, the connection strength between this modality and other modalities in the corresponding region is enhanced. When the participation of any mode in the abnormal response is less than the preset participation threshold, the weight of the mode in the coupling structure is reduced; combined with the topological structure evolution trajectory recorded in the abnormal fluctuation map, the feedback path structure between the modes is dynamically updated to generate a feedback adjustment curve in which the coupling strength between the modes changes over time.

[0015] Preferably, the method for identifying abnormal events includes: The characteristic variables of each mode in the multimodal signal are represented as graph nodes. The change order relationship between the characteristic variables, trigger frequency and signal gain are combined to construct a set of candidate causal edges. The candidate edges are constrained and screened to eliminate the edge relationships in the low-complexity region where the structural complexity is less than the preset structural complexity threshold, thus forming an effective candidate causal network under structural modulation. For each pair of candidate nodes, we calculate three indicators, including information gain, transfer entropy, and coupling strength in the abnormal fluctuation graph, to construct a causal relationship scoring function. Combined with the local gradient dynamic weighting of the topological entropy field, we obtain the credibility weight of the multimodal causal edge. Based on the credibility weight, the causal candidate edges are structured, and a multimodal causal structure map is generated through the maximum weighted directed acyclic graph extraction algorithm. The node clusters where the fluctuation propagation paths are concentrated and the local areas where the propagation energy concentration is greater than the preset propagation energy concentration threshold are identified in the multimodal causal structure map. Combined with the amplitude transition and phase disturbance at the corresponding position in the abnormal fluctuation map, it is determined whether the area constitutes an abnormal event.

[0016] Preferably, the method for tracing a cross-modal spatiotemporal causal path includes: The corresponding node of the identified abnormal event in the multimodal causal structure graph is used as the tracing starting point, and the modal type, event intensity, timestamp information, and upstream and downstream causal edge information of the tracing starting point are recorded. For the case of concurrent occurrence of different abnormal events, the topological adjacency analysis in the abnormal fluctuation graph is used to construct an abnormal source set containing different starting points. For each anomaly starting point node, a cross-modal causal path scoring function is designed to dynamically score the causal path starting from this node. The path scoring function is obtained by weighting the credibility score of the causal edge, the structural coupling strength between modes, and the anomaly propagation strength of the corresponding position in the anomaly fluctuation map. Based on the scoring results of the path scoring function, a cross-modal spatiotemporal causal path is recursively searched in chronological order starting from the starting node. During the path expansion process, edges with path scores higher than the preset path scoring threshold are preferentially selected. When mode switching occurs, the modal coupling strength and abnormal propagation gradient are used for guidance. If the path score is lower than the preset path score threshold or a loop structure appears, the current path tracking is terminated and the path pruning operation is performed; all cross-modal spatiotemporal causal paths that meet the conditions are integrated into a cross-modal spatiotemporal causal path graph.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses nodes to represent modal embedding units and edges to represent explicit or implicit associations, thus realizing for the first time the integrated modeling of different modalities at the structural level, enhancing the semantic connectivity and interactive parsing capabilities between data, and laying a unified structural foundation for subsequent topological modeling and dynamic analysis. By constructing a topological entropy spatiotemporal tensor field and combining it with a wave-diffusion-like partial differential equation, the propagation path of topological complexity in the graph is simulated, and potential behaviors such as information flow diffusion and structural coupling enhancement are effectively captured, so that the originally abstract system evolution has an observable and deducible dynamic description. By calculating the spectral radius of the local Jacobian matrix and combining the criterion of drastic changes in topological entropy over time, a topological bifurcation point identification model is constructed to accurately capture the critical state or sudden transition of the system structure, significantly improving the early discovery and root cause location capabilities of project operation anomalies. The identified topological bifurcation points are combined with modal type, propagation direction and time label to form an abnormal response map, which provides an intuitive decision-making basis for causal analysis of multimodal events, tracing the source of key nodes and strategic intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the structure of the multimodal project data analysis method of the present invention; Figure 2 Schematic diagram of the multimodal project data analysis system flow of the present invention; Figure 3 This is a schematic diagram of the method for outputting multimodal signals provided by the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] Example 1 See also Figure 1 and Figure 3 As shown, this embodiment 1 further illustrates the multimodal project data analysis method proposed by the present invention, including: In practical applications such as complex engineering projects, urban operation systems, industrial manufacturing, and management information systems, information from multiple modalities, including images, audio, time-series sensors, structured logs, and text documents, is often involved. This multimodal data reflects the system's operational status, environmental changes, and human-computer interaction behaviors, and is highly heterogeneous, dynamic, and spatiotemporally correlated. Therefore, how to fuse and analyze multimodal data to accurately identify anomalies, infer causal relationships, and support project evaluation and decision-making is a core issue in current research on multimodal intelligent analysis technologies.

[0021] Existing multimodal project data analysis methods are typically based on static fusion strategies, using simple splicing, weighting, or machine learning models to jointly model data from different modalities. These methods struggle to capture the nonlinear coupling mechanisms between modalities that evolve over time and overlook the mutual feedback and dynamic influences between different information sources during project operation. This results in analytical models being slow to respond to unexpected events, delayed effects, or structural mutations, impacting their early warning and tracking capabilities in practical applications.

[0022] Existing anomaly detection technologies primarily rely on sliding statistical features, threshold judgments, or static classifiers. These are susceptible to noise in multimodal scenarios, lack structural sensitivity, and struggle to effectively model system state changes by incorporating the topological evolution of multimodal embedding graphs. This leads to the risk of missed anomalies, high false positive rates, and difficulty interpreting them in real-world projects. This is especially true when there is asynchrony between modalities, heterogeneous coupling, or missing modalities, making it difficult for traditional methods to develop a stable and robust analysis path.

[0023] Existing graph structure analysis methods have significant limitations in temporal modeling. Most graph methods are unable to accurately determine the critical transition states of nodes or structures in the graph during temporal evolution, struggle to characterize bifurcation points or non-stationary dynamic features in structural changes, and lack the ability to model the changing trends of graph topology from a quasi-dynamic perspective. Furthermore, they are unable to integrate semantic hierarchies across multiple modalities for causal path tracing. This makes it difficult to promptly identify potential risks or key drivers during project system evolution, limiting the predictive power and interpretability of intelligent analysis systems.

[0024] The present invention is proposed in this context and proposes a multimodal project data analysis method, comprising: S1. Collect heterogeneous modal project data, denoise and timestamp correct the data of each modality based on adaptive multi-scale filtering technology, perform cross-modal time alignment, and output multimodal signals; S2. Construct a multi-level Boltzmann machine energy network to model the joint probability distribution of multimodal signals; design a dynamic complexity measurement mechanism based on topological entropy to represent the complexity of project structure and generate a topological entropy dynamic field; S3. Using the topological entropy dynamic field as a constraint, and utilizing the multidimensional covariant field framework, we simulate the nonlinear coupling and spatiotemporal propagation between multimodal signals, abstract the multimodal signals into string vibration modes, identify local topological defects in the topological entropy dynamic field, and generate abnormal fluctuation maps. S4. Drawing on the selective regulation characteristics of glial cells on neural signals and combining them with abnormal fluctuation maps, the weighted coupling strength of each modal signal is dynamically adjusted to perform inter-modal feedback regulation. S5. Based on the weighted multimodal signals, a multimodal causal structure self-construction algorithm is designed. Combined with the topological entropy dynamic field and abnormal fluctuation map, abnormal events are identified, and cross-modal spatiotemporal causal paths are tracked. The identified abnormal events are then transmitted to the data analysis terminal.

[0025] Methods for collecting heterogeneous modality project data include: Deploy n data acquisition channels with different perception types at the project site. These channels include image acquisition channels, voice acquisition channels, structured data interface channels, time series signal acquisition channels, and unstructured text acquisition channels to collect heterogeneous modality project data. Heterogeneous modal project data includes image modality, sound modality, equipment operating parameter modality, time series signal modality data and project document modality data; based on data source, physical type and semantic attributes, the modality type of heterogeneous modal project data is identified and labeled to generate a data structure with modality labels.

[0026] Methods for outputting multimodal signals include: A multiscale filter bank is constructed based on the modal type of the heterogeneous modal project data. The multiscale filter bank includes a filter structure based on wavelet transform and empirical mode decomposition to extract signal trends and suppress modal-specific noise. The sampling frequency, noise characteristics, and spectral distribution of each heterogeneous modal project data are analyzed, and the filter scale and adaptive denoising are dynamically adjusted based on the modal-specific error feedback mechanism. Anomaly detection is performed in timestamp order to identify missing values, mutation points and pseudo-synchronization segments, and timestamp correction is performed using interpolation compensation methods. The heterogeneous modal project data with timestamps are input into the cross-modal time alignment algorithm, and nonlinear alignment is performed on the heterogeneous modal project data with different sampling frequencies and synchronization delays, ultimately generating a multimodal signal under a unified time axis.

[0027] Methods for modeling the joint probability distribution of multimodal signals include: Convert multimodal signals into a unified embedding representation, using different embedding strategies for different modalities. The embedding strategies include using convolutional feature extraction to generate visual embeddings for image modalities, using recursive neural network structures to generate sequence embeddings for sound modalities, and using semantic encoders to generate context vectors for project document modalities. All modal embeddings are normalized and used as input to the Boltzmann machine. Construct a multi-level Boltzmann machine network structure, where each layer includes learnable latent variable nodes, which form an undirected graph structure with the nodes of the previous and next layers through symmetrical connections, forming an energy network that depicts the joint distribution of the semantics of multimodal signals; The energy function of the energy network is designed to characterize the joint activation relationship between modal embedding and latent variables. A regulation factor based on the topological entropy dynamic field is introduced to adjust the sensitivity of the connection weights between modalities. The parameter distribution for joint modeling and reasoning is obtained, and finally the joint probability distribution model of multimodal signals in the joint modal space is output.

[0028] Methods for generating dynamic fields of topological entropy include: Based on multimodal signals, a unified multimodal embedding graph is constructed. The nodes in the multimodal embedding graph represent the embedding units of each modal signal, and the edges represent the explicit or implicit associations between modal signals. For any node in the multimodal embedding graph, the topological entropy of the node is defined within a preset sliding time window. The topological entropy of a node is: ;in, Representation node At the time point The topological entropy of Representation node The set of adjacent nodes of ; Representation node and adjacent nodes Between the time points The normalized edge weights of ; Indicates the nodes; Indicates the nodes; and Indicates the index of the node; An index representing a time point; The time-varying topological entropy of each node in the multimodal embedding graph is constructed as a spatiotemporal continuous distribution tensor, and a wave-like diffusion mechanism is introduced to model the propagation behavior, defining the governing partial differential equation for the propagation of topological entropy. The governing partial differential equation is: ; represents the structural diffusion coefficient, which controls the diffusion speed of topological entropy propagation on the multimodal embedding graph; It represents the heterogeneous coupling regulation factor, which controls the degree of intervention between heterogeneous modes on the evolution of topological entropy; is the graph Laplacian operator, which represents the spatial diffusion trend of topological entropy on the multimodal embedded graph; is the modal coupling tensor, which represents the position , time point Under , the coupling strength between different modes; Represent the spatial positions of nodes in a multimodal embedded graph; numerically solve the governing partial differential equations using a discretization method based on the graph structure to generate a dynamic field of topological entropy at any time; To identify sudden change points in the topological structure, bifurcation theory is used to determine the stability of node structures in the dynamic field of topological entropy. The local Jacobian matrix of each node in the current window is calculated, and the spectral radius is extracted to determine whether the spectral radius meets the topological bifurcation condition. The topological bifurcation condition is: in, Represents the maximum value of the modulus of all eigenvalues ​​of the Jacobian matrix; Representation node The propagation sensitivity Jacobian matrix of represents the derivative of topological entropy with respect to time; Indicates the preset topological entropy increase threshold; It is a logical symbol. In a logical expression, it means that two conditions are met at the same time. In a topological bifurcation condition, it means that two conditions must be met at the same time. When the topological bifurcation condition is met, the node is determined to be a topological bifurcation point; all identified topological bifurcation points are combined with their mode type, timestamp and propagation direction to be plotted as a topological entropy anomaly response map.

[0029] The following problems with existing technologies are solved: Most existing multimodal data analysis methods use static fusion strategies, which are unable to characterize the complex coupling behavior between different modalities that evolve over time, and are difficult to reflect the dynamic interaction between different information sources during the project process; most existing methods use rule thresholds, sliding statistics, or machine learning classifiers for anomaly detection, lacking the ability to model and identify the topological structure change trends in multimodal embedded graphs, and are prone to missing or misjudging key evolutionary nodes. They are unable to utilize the structural information between multiple modalities to enhance the robustness of anomaly detection, have difficulty identifying deep changes in the structural hierarchy, and lack a unified judgment standard for comprehensive judgment of heterogeneous source data. Existing graph analysis methods are unable to determine the critical turning point state of a node in the temporal evolution process, cannot achieve early warning and explainable tracking of the evolution process of the project system, lack the ability to understand the structural changes in the graph from the perspective of non-stationary dynamic evolution, and have difficulty in determining where and when irreversible evolutionary trends or risk surge points are faced.

[0030] Compared to existing technologies, this approach offers several advantages: by representing modal embedding units with nodes and explicit or implicit relationships with edges, it achieves the first integrated structural modeling of different modalities, enhancing semantic connectivity and interactive parsing capabilities between data and laying a unified structural foundation for subsequent topological modeling and dynamic analysis. By constructing a topological entropy spatiotemporal tensor field and combining it with a wave-diffusion-like partial differential equation, it simulates the propagation path of topological complexity in the graph, effectively capturing potential behaviors such as information flow diffusion and enhanced structural coupling, and providing an observable and deducible dynamic description of the previously abstract system evolution. By calculating the spectral radius of the local Jacobian matrix and combining it with a criterion for the dramatic change of topological entropy over time, a topological bifurcation point identification model is constructed, enabling accurate capture of critical states or sudden transitions in the system structure, significantly improving the ability to early detect and locate the root cause of project operational anomalies. The identified topological bifurcation points, combined with modal type, propagation direction, and time labels, converge to form an anomaly response map, providing an intuitive decision-making basis for multimodal event causal analysis, key node tracing, and strategic intervention.

[0031] Methods for abstracting multimodal signals into string vibration modes include: Based on denoised and time-aligned multimodal signal data, a sliding time window and modal spatial distribution information are used to construct a multidimensional covariant tensor field. The topological entropy dynamic field is introduced as a modulation constraint into the construction of the multidimensional covariant tensor field to calculate the covariance, mutual information, and high-order statistics between different modes. Based on the covariance, mutual information, and high-order statistics between different modes in different time periods, the covariance coefficient is dynamically assigned an entropy-guided weighting factor to highlight the coupling characteristics of topologically complex regions. Under the constraint of topological entropy, the coupling mapping relationship between multimodal signals is modeled by nonlinear kernel functions to generate coupling paths containing time, space and modal dimensions; the coupling paths are abstracted into a multidimensional string vibration model under controlled boundary conditions.

[0032] Methods for generating abnormal fluctuation maps include: Each modal signal is represented as an excitable and coupled vibrating string-like element, and energy propagation and resonance behavior are simulated in the covariant field. The dominant modal frequency, standing wave nodes and phase jump areas are extracted based on the multidimensional string vibration model. Local topological defects that exhibit energy aggregation and sudden propagation in the covariant field are identified, and an abnormal fluctuation map containing abnormal response paths and disturbance area identifiers is constructed.

[0033] Methods for inter-modal feedback adjustment include: Based on the abnormal fluctuation map, each node in the multimodal embedding graph corresponding to the multimodal signal is identified and labeled to determine the node area with structural anomalies. For each node, characteristic parameters are extracted; the characteristic parameters include mode type, timestamp, number of adjacent nodes and abnormal fluctuation amplitude, and the regulation strength index of the node at the current moment is calculated. Drawing on the regulatory mechanism of glial cells in the nervous system for synaptic signal transmission, the modal coupling weights between a node and its adjacent nodes are selectively adjusted based on the regulatory strength index of each node. When the abnormal fluctuation trend in a region of any modality is greater than a preset abnormal fluctuation trend threshold, the connection strength between this modality and other modalities in the corresponding region is enhanced. When the participation of any mode in the abnormal response is less than the preset participation threshold, the weight of the mode in the coupling structure is reduced; combined with the topological structure evolution trajectory recorded in the abnormal fluctuation map, the feedback path structure between the modes is dynamically updated to generate a feedback adjustment curve in which the coupling strength between the modes changes over time.

[0034] Methods for identifying abnormal events include: The characteristic variables of each mode in the multimodal signal are represented as graph nodes. The change order relationship between the characteristic variables, trigger frequency and signal gain are combined to construct a set of candidate causal edges. The candidate edges are constrained and screened to eliminate the edge relationships in the low-complexity region where the structural complexity is less than the preset structural complexity threshold, thus forming an effective candidate causal network under structural modulation. For each pair of candidate nodes, we calculate three indicators, including information gain, transfer entropy, and coupling strength in the abnormal fluctuation graph, to construct a causal relationship scoring function. Combined with the local gradient dynamic weighting of the topological entropy field, we obtain the credibility weight of the multimodal causal edge. Based on the credibility weight, the causal candidate edges are structured, and a multimodal causal structure map is generated through the maximum weighted directed acyclic graph extraction algorithm. The node clusters where the fluctuation propagation paths are concentrated and the local areas where the propagation energy concentration is greater than the preset propagation energy concentration threshold are identified in the multimodal causal structure map. Combined with the amplitude transition and phase disturbance at the corresponding position in the abnormal fluctuation map, it is determined whether the area constitutes an abnormal event.

[0035] Methods for tracing cross-modal spatiotemporal causal paths include: The corresponding node of the identified abnormal event in the multimodal causal structure graph is used as the tracing starting point, and the modal type, event intensity, timestamp information, and upstream and downstream causal edge information of the tracing starting point are recorded. For the case of concurrent occurrence of different abnormal events, the topological adjacency analysis in the abnormal fluctuation graph is used to construct an abnormal source set containing different starting points. For each anomaly starting point node, a cross-modal causal path scoring function is designed to dynamically score the causal path starting from this node. The path scoring function is obtained by weighting the credibility score of the causal edge, the structural coupling strength between modes, and the anomaly propagation strength of the corresponding position in the anomaly fluctuation map. Based on the scoring results of the path scoring function, a cross-modal spatiotemporal causal path is recursively searched in chronological order starting from the starting node. During the path expansion process, edges with path scores higher than the preset path scoring threshold are preferentially selected. When mode switching occurs, the modal coupling strength and abnormal propagation gradient are used for guidance. If the path score is lower than the preset path score threshold or a loop structure appears, the current path tracking is terminated and the path pruning operation is performed; all cross-modal spatiotemporal causal paths that meet the conditions are integrated into a cross-modal spatiotemporal causal path graph.

[0036] The preset abnormal fluctuation trend threshold is set by the staff based on the results of historical data analysis. The historical analysis process includes the system collecting the average value of multiple abnormal fluctuation trends as a reference, which can be adjusted by the staff according to actual conditions during the operation of the system; similarly, the preset participation threshold, preset structural complexity threshold, preset propagation energy concentration threshold and preset path score threshold are set.

[0037] In this embodiment, the nodes represent the modal embedding units and the edges represent the explicit or implicit association relationships, which realizes the integrated modeling of different modalities at the structural level for the first time, enhances the semantic connectivity and interactive analysis capabilities between data, and lays a unified structural foundation for subsequent topological modeling and dynamic analysis. By constructing the topological entropy space-time tensor field and combining it with the wave-diffusion partial differential equation, the propagation path of topological complexity in the graph is simulated, and potential behaviors such as information flow diffusion and structural coupling enhancement are effectively captured, so that the originally abstract system evolution has an observable and deducible dynamic description. By calculating the spectral radius of the local Jacobian matrix and combining the criterion of drastic changes in topological entropy over time, a topological bifurcation point identification model is constructed to accurately capture the critical state or sudden change of the system structure, significantly improving the early detection and root cause location capabilities of project operation anomalies. The identified topological bifurcation points are combined with the modal type, propagation direction and time label to form an abnormal response map, which provides an intuitive decision-making basis for multimodal event causal analysis, key node tracing and strategic intervention.

[0038] Example 2 See also Figure 2 As shown, for the parts not described in detail in this embodiment, please refer to the description of Example 1. A multimodal project data analysis system is provided, including: The data processing and presentation module collects heterogeneous modal project data, denoises and timestamps each modal project data based on adaptive multi-scale filtering technology, performs cross-modal time alignment, and outputs multimodal signals. The multimodal energy optimization module constructs a multi-level Boltzmann machine energy network to model the joint probability distribution of multimodal signals; designs a dynamic complexity measurement mechanism based on topological entropy to represent the complexity of project structure and generate a topological entropy dynamic field; The multi-dimensional covariant field construction and analysis module uses the topological entropy dynamic field as a constraint condition and utilizes the multi-dimensional covariant field framework to simulate the nonlinear coupling and spatiotemporal propagation between multimodal signals, abstracting the multimodal signals into string vibration modes, identifying local topological defects in the topological entropy dynamic field, and generating abnormal fluctuation maps. The multimodal information flow adaptive module draws on the selective regulation characteristics of glial cells on neural signals and combines them with abnormal fluctuation maps to dynamically adjust the weight coupling strength of each modal signal to establish inter-modal feedback regulation; The multimodal causal tracking module designs a multimodal causal structure self-construction algorithm based on the weighted multimodal signal, combines the topological entropy dynamic field and the abnormal fluctuation map, identifies abnormal events, tracks the cross-modal spatiotemporal causal path, and transmits the identified abnormal events to the data analysis terminal.

[0039] Since the electronic device described in this embodiment is an electronic device used to implement the multimodal project data analysis method in the embodiment of the present application, based on the multimodal project data analysis method described in the embodiment of the present application, those skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application will not be described in detail here. As long as those skilled in the art implement the electronic device used in the multimodal project data analysis method in the embodiment of the present application, it falls within the scope of protection of this application.

[0040] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0041] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal project data analysis method, characterized in that: include: S1. Collect heterogeneous modal project data, denoise and timestamp correct the data of each modality based on adaptive multi-scale filtering technology, perform cross-modal time alignment, and output multimodal signals; S2. Construct a multi-level Boltzmann machine energy network to model the joint probability distribution of multimodal signals; Design a dynamic complexity measurement mechanism based on topological entropy to represent the complexity of project structure and generate a dynamic field of topological entropy; S3. Using the topological entropy dynamic field as a constraint, and utilizing the multidimensional covariant field framework, we simulate the nonlinear coupling and spatiotemporal propagation between multimodal signals, abstract the multimodal signals into string vibration modes, identify local topological defects in the topological entropy dynamic field, and generate abnormal fluctuation maps. S4. Drawing on the selective regulation characteristics of glial cells on neural signals and combining them with abnormal fluctuation maps, the weighted coupling strength of each modal signal is dynamically adjusted to perform inter-modal feedback regulation. S5. Based on the weighted multimodal signals, a multimodal causal structure self-construction algorithm is designed. Combined with the topological entropy dynamic field and abnormal fluctuation map, abnormal events are identified, and cross-modal spatiotemporal causal paths are tracked. The identified abnormal events are then transmitted to the data analysis terminal.

2. The multimodal project data analysis method according to claim 1, characterized in that: The method for collecting heterogeneous modality project data includes: Deploy n data acquisition channels with different perception types at the project site to collect heterogeneous modal project data; heterogeneous modal project data includes image modality, sound modality, equipment operation parameter modality, time series signal modality data and project document modality data; perform modality type identification and labeling on heterogeneous modality project data based on data source, physical type and semantic attributes to generate a data structure with modality labels.

3. The multimodal project data analysis method according to claim 2, characterized in that: The method for outputting a multimodal signal comprises: Multi-scale filter groups are constructed according to the modal types of heterogeneous modal project data to extract signal trends and suppress modal-specific noise; anomaly detection is performed in timestamp order to identify missing values, mutation points and pseudo-synchronization segments, and timestamp correction is performed using interpolation compensation method; heterogeneous modal project data with timestamps are input into the cross-modal time alignment algorithm to generate multimodal signals under a unified time axis.

4. The multimodal project data analysis method according to claim 3, characterized in that: The method for modeling the joint probability distribution of multimodal signals includes: Convert multimodal signals into a unified embedding representation, adopt different embedding strategies for different modalities, and standardize all modal embeddings before using them as input for the Boltzmann machine. Build a multi-level Boltzmann machine network structure to form an energy network. The energy function of the energy network is designed, and a regulation factor based on the topological entropy dynamic field is introduced to adjust the sensitivity of the connection weights between each modality; the parameter distribution for joint modeling and reasoning is obtained, and finally the joint probability distribution model of multimodal signals in the joint modal space is output.

5. The multimodal project data analysis method according to claim 4, characterized in that: The method for generating a topological entropy dynamic field comprises: Based on multimodal signals, a unified multimodal embedding graph is constructed. For any node in the multimodal embedding graph, the topological entropy of the node is defined within a preset sliding time window. The time-varying topological entropy of each node in the multimodal embedding graph is constructed as a spatiotemporal continuous distribution tensor, and a quasi-wave diffusion mechanism is introduced to model the propagation behavior, defining the governing partial differential equation for the propagation of topological entropy. The governing partial differential equation is numerically solved using a discretization method based on the graph structure to generate a dynamic field of topological entropy at any time. In order to identify sudden change points in the topological structure, bifurcation theory is used to determine the stability of the node structure in the topological entropy dynamic field; the local Jacobian matrix of each node in the current window is calculated, and the spectral radius is extracted to determine whether the spectral radius meets the topological bifurcation condition; when the topological bifurcation condition is met, the node is determined to be a topological bifurcation point; all identified topological bifurcation points are combined with their mode type, timestamp and propagation direction to plot as a topological entropy anomaly response map.

6. The multimodal project data analysis method according to claim 5, characterized in that: The method for abstracting a multimodal signal into a string vibration mode includes: Based on multimodal signal data, a multidimensional covariant tensor field is constructed using a sliding time window and modal space distribution information; the topological entropy dynamic field is introduced as a modulation constraint into the construction process of the multidimensional covariant tensor field, and the covariance, mutual information and high-order statistics between different modes are calculated; the coupling mapping relationship between multimodal signals is modeled by a nonlinear kernel function, and a coupling path containing time, space and modal dimensions is generated; the coupling path is abstracted into a multidimensional string vibration model under controlled boundary conditions.

7. The multimodal project data analysis method according to claim 6, characterized in that: The method for generating an abnormal fluctuation spectrum includes: Each modal signal is represented as a vibrating string element, and energy propagation and resonance behavior are simulated in the covariant field. Based on the multidimensional string vibration model, the dominant modal frequencies, standing wave nodes and phase jump areas are extracted. Local topological defects showing energy aggregation and sudden propagation in the covariant field are identified, and an abnormal fluctuation map containing abnormal response paths and disturbance area identifiers is constructed.

8. The multimodal project data analysis method according to claim 7, characterized in that: The method for performing inter-modal feedback adjustment includes: Based on the abnormal fluctuation map, each node in the multimodal embedding graph corresponding to the multimodal signal is identified and labeled to determine the node area with structural abnormalities; for each node, the characteristic parameters are extracted and the regulation strength index of the node at the current moment is calculated; Drawing on the regulatory mechanism of glial cells on synaptic signal transmission in the nervous system, the modal coupling weights between nodes and adjacent nodes are selectively adjusted according to the regulatory strength index of each node. Combined with the topological structure evolution trajectory recorded in the abnormal fluctuation map, the feedback pathway structure between each mode is dynamically updated to generate a feedback regulation curve in which the coupling strength between modalities changes over time.

9. The multimodal project data analysis method according to claim 8, characterized in that: The method for identifying abnormal events includes: The characteristic variables of each mode in the multimodal signal are represented as graph nodes. The change order relationship between the characteristic variables, trigger frequency and signal gain are combined to construct a set of candidate causal edges. The candidate edges are constrained and screened to eliminate the edge relationships in the low-complexity region where the structural complexity is less than the preset structural complexity threshold, thus forming an effective candidate causal network under structural modulation. A causal relationship scoring function is constructed, and combined with the local gradient dynamic weighting of the topological entropy field, the credibility weight of the multimodal causal edge is obtained. Based on the credibility weight, the causal candidate edge is structured and a multimodal causal structure graph is generated through the maximum weighted directed acyclic graph extraction algorithm. Identify the node clusters where the wave propagation paths are concentrated in the multimodal causal structure map and the local areas where the propagation energy concentration is greater than the preset propagation energy concentration threshold. Combined with the amplitude transition and phase disturbance at the corresponding position in the abnormal wave map, determine whether the area constitutes an abnormal event.

10. The multimodal project data analysis method according to claim 9, characterized in that: The method for tracing cross-modal spatiotemporal causal paths includes: The nodes corresponding to the identified abnormal events in the multimodal causal structure graph are used as the tracing starting points. For the case where different abnormal events occur concurrently, a set of abnormal sources containing different starting points is constructed through topological adjacency analysis in the abnormal fluctuation graph. For each abnormal starting point node, a cross-modal causal path scoring function is designed to dynamically score the causal path starting from this node. Based on the scoring results of the path scoring function, a cross-modal spatiotemporal causal path is recursively searched in chronological order starting from the starting point. During the path extension process, edges with path scores higher than the preset path score threshold are preferentially selected, and the modal coupling strength and abnormal propagation gradient are used as reference for guidance when modal switching occurs. If the path score is lower than the preset path score threshold or a loop structure appears, the current path tracking is terminated and path pruning is performed. All cross-modal spatiotemporal causal paths that meet the conditions are integrated into a cross-modal spatiotemporal causal path graph.

Images