Multimodal project data analysis method

Abstract

The present application belongs to the technical field of data processing, and discloses a multi-modal project data analysis method, which comprises the following steps: collecting heterogeneous modal project data; based on adaptive multi-scale filtering technology, denoising and timestamp correction are performed on each modal project data, and cross-modal time alignment is performed, and a multi-modal signal is output; a multi-level Boltzmann machine energy network is constructed, and a joint probability distribution of the multi-modal signal is modeled; a dynamic complexity measurement mechanism based on topological entropy is designed, which is used for representing project structure complexity, and a topological entropy dynamic field is generated; the topological entropy dynamic field is used as a constraint condition, a multi-dimensional covariant field framework is used, nonlinear coupling and space-time propagation among the multi-modal signals are simulated, the multi-modal signals are abstracted into string vibration modes, local topological defects in the topological entropy dynamic field are identified, and an abnormal fluctuation atlas is generated; and the analysis of complex multi-modal data in the project running process is more intelligent and controllable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, more particularly, the present application relates to a multi-modal project data analysis method. BACKGROUND

[0002] The patent with the patent publication number CN117710117A discloses an insurance data analysis method and system based on a multi-modal large model, obtains a picture document submitted by an insured person, extracts specified information in the picture document through a multi-modal large model, including document type, audit items and fees; determines whether the extracted specified information is a claim item through the multi-modal large model, if the item in the specified information is a claim item, determines the claim category and claim ratio; calculates the claim amount of the picture document according to the determined claim category and claim ratio; presents the commercial insurance business processing and claim result of the picture document to the commercial insurance staff for confirmation and modification of the commercial insurance business processing and claim result of the picture document; generates a desensitized claim report using the confirmed and modified commercial insurance business processing and claim result of the picture document. The present application can no longer require manual intervention for the submitted picture document, and directly automatically process and directly convert into structured data; high accuracy.

[0003] The existing modal project data analysis method mainly has the following problems:

[0004] Most existing multi-modal data analysis methods use static fusion strategies, which cannot depict the complex coupling behavior between different modalities over time, and cannot reflect the dynamic mutual influence of different information sources in the project process; most existing methods use rule thresholds, sliding statistics or machine learning classifiers for anomaly detection, lack modeling and identification of the trend of topological structure changes in multi-modal embedding graphs, are prone to missing or misjudging critical evolution nodes, cannot enhance the robustness of anomaly detection using the structure information between modalities, and are difficult to identify deep changes in the structure level, and lack a unified judgment standard for comprehensive judgment of heterogeneous source data.

[0005] The existing technology graph analysis method cannot determine the critical turning state of a node in the time evolution process, cannot realize early warning and interpretable tracking of the project system evolution process, lacks the ability to understand the structure transition in the graph from the perspective of non-stationary dynamics evolution, and is difficult to determine where and when to face irreversible evolution trends or risk explosion points.

[0006] In view of the above, the present application proposes a multi-modal project data analysis method to solve the above problems. SUMMARY

[0007] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme: a multi-modal project data analysis method, comprising:

[0008] S1, collect heterogeneous modal project data, denoise and timestamp correct each modal project data based on adaptive multi-scale filtering technology, and perform cross-modal time alignment to output multi-modal signals;

[0009] S2, construct a multi-level Boltzmann machine energy network to model the joint probability distribution of the multi-modal signals; design a dynamic complexity measurement mechanism based on topological entropy to represent the project structure complexity and generate a topological entropy dynamic field;

[0010] S3, using the topological entropy dynamic field as a constraint condition, simulating the nonlinear coupling and space-time propagation between multi-modal signals by using a multi-dimensional covariant field framework, abstracting the multi-modal signals into string vibration modes, identifying local topological defects in the topological entropy dynamic field, and generating abnormal fluctuation atlas;

[0011] S4, referring to the selective regulation characteristics of glial cells on neural signals, combining the abnormal fluctuation atlas, and dynamically adjusting the weight coupling strength of each modal signal for inter-modal feedback regulation;

[0012] S5, based on the multi-modal signals after weight adjustment, designing a multi-modal causal structure self-construction algorithm, combining the topological entropy dynamic field and the abnormal fluctuation atlas, identifying abnormal events, and tracking cross-modal spatiotemporal causal paths, and transmitting the identified abnormal events to a data analysis terminal.

[0013] Preferably, the method for collecting heterogeneous modal project data comprises:

[0014] Deploying n data acquisition channels with different sensing types on the project site, the data acquisition channels including image acquisition channels, voice acquisition channels, structured data interface channels, time series signal acquisition channels, and unstructured text acquisition channels, to collect heterogeneous modal project data;

[0015] The heterogeneous modal project data includes image modal, sound modal, device operating parameter modal, time series signal modal data, and project document modal data; based on data source, physical type, and semantic attribute, the heterogeneous modal project data is identified and labeled by modal type to generate a data structure with modal labels.

[0016] Preferably, the method for outputting multi-modal signals comprises:

[0017] According to the modal type of the heterogeneous modal project data, a multi-scale filter bank is constructed, including filter structures based on wavelet transform and empirical mode decomposition, to extract signal trends and suppress modal-specific noise; analyze the sampling frequency, noise characteristics, and frequency spectrum distribution of each heterogeneous modal project data, and dynamically adjust the filtering scale and adaptive denoising based on the modal-specific error feedback mechanism;

[0018] The abnormality detection is performed in time stamp order, missing values, mutation points and pseudo-synchronous fragments are identified, and an interpolation compensation method is used for time stamp correction; the heterogeneous modal item data with time stamp is input into the cross-modal time alignment algorithm, the heterogeneous modal item data with different sampling frequencies and synchronization delays are nonlinearly aligned, and finally the multi-modal signal under the unified time axis is generated.

[0019] Preferably, the method for modeling the joint probability distribution of the multi-modal signal comprises:

[0020] The multi-modal signal is converted into a unified embedding representation, different embedding strategies are used for different modalities, and after standardization processing of all modal embeddings, the unified embedding representation is input into the Boltzmann machine; a multi-level Boltzmann machine network structure is constructed, each level includes a learnable hidden variable node, and the node is connected to the node of the previous layer and the node of the next layer by a symmetric connection to form a undirected graph structure, forming an energy network for depicting the joint distribution of the semantic of the multi-modal signal;

[0021] An energy function of the energy network is designed to depict the joint activation relationship between the modal embedding and the hidden variable, a regulation factor based on the topological entropy dynamic field is introduced to adjust the sensitivity of the connection weight between the modalities; a parameter distribution for joint modeling and reasoning is obtained, and finally a joint probability distribution model of the multi-modal signal in the joint modal space is output.

[0022] Preferably, the method for generating the topological entropy dynamic field comprises:

[0023] Based on the multi-modal signal, a unified multi-modal embedding graph is constructed, the nodes in the multi-modal embedding graph represent the embedding units of the modal signals, and the edges represent the explicit or implicit association relationship between the modal signals; for any node in the multi-modal embedding graph, the topological entropy of the node is defined under a preset sliding time window;

[0024] The topological entropy of each node in the multi-modal embedding graph is constructed as a spatiotemporal continuous distribution tensor, and a wave-like diffusion mechanism is introduced to model the propagation behavior, and a control partial differential equation for topological entropy propagation is defined; the control partial differential equation is numerically solved by using a discretization method on the graph structure to generate the topological entropy dynamic field at any time;

[0025] In order to identify the sudden change points in the topological structure, the node structure stability is judged by using the bifurcation theory on the topological entropy dynamic field; the local Jacobian matrix of each node under the current window is calculated, and the spectral radius is extracted to determine whether the spectral radius satisfies the topological bifurcation condition;

[0026] When the topological bifurcation condition is satisfied, it is determined that the node is a topological bifurcation point; all identified topological bifurcation points are combined with the modal type, time stamp and propagation direction to draw a topological entropy abnormal response atlas.

[0027] Preferably, the method of abstracting the multi-modal signal into a string vibration mode comprises:

[0028] Based on the multi-modal signal data after denoising and time alignment, a multi-dimensional covariant tensor field is constructed using a sliding time window and modal space distribution information. A topological entropy dynamic field is introduced as a modulation constraint in the construction process of the multi-dimensional covariant tensor field, and the covariance, mutual information and high-order statistics between different modalities are calculated. According to the covariance, mutual information and high-order statistics between different modalities in different time periods, the covariant coefficients are dynamically given an entropy-oriented weighting factor.

[0029] Under the constraint of topological entropy, the coupling mapping relationship between multi-modal signals is modeled by a nonlinear kernel function to generate a coupling path containing time, space and modal dimensions. The coupling path is abstracted as a multi-dimensional string vibration model under controlled boundary conditions.

[0030] Preferably, the method of generating an abnormal fluctuation atlas comprises:

[0031] Each modality signal is represented as a vibration-like string element that can be excited and coupled. The energy propagation and resonance behavior in the covariant field are simulated, and the dominant modal frequency, standing wave node and phase jump region are extracted based on the multi-dimensional string vibration model. Local topological defects showing energy aggregation and sudden propagation in the covariant field are identified, and an abnormal fluctuation atlas containing abnormal response paths and disturbance region identifiers is constructed.

[0032] Preferably, the method of performing inter-modal feedback regulation comprises:

[0033] Based on the abnormal fluctuation atlas, each node in the multi-modal embedding graph corresponding to the multi-modal signal is identified and labeled, and the node region with structural abnormalities is determined. For each node, the feature parameters are extracted, and the adjustment intensity index of the node at the current time is calculated.

[0034] Referring to the regulation mechanism of glial cells in the nervous system for synaptic signal transmission, the modal coupling weight between the node and the adjacent node is selectively adjusted according to the adjustment intensity index of each node. When the abnormal fluctuation trend of any modality is greater than a preset abnormal fluctuation trend threshold, the connection strength between the modality in the corresponding region and other modalities is enhanced.

[0035] When the participation of any modality in the abnormal response is less than a preset participation threshold, the weight of the modality in the coupling structure is reduced. Combined with the topological structure evolution trajectory recorded in the abnormal fluctuation atlas, the feedback path structure between the modalities is dynamically updated, and a feedback regulation curve of the coupling strength between the modalities over time is generated.

[0036] Preferably, the method of identifying abnormal events comprises:

[0037] The characteristic variables of each mode in the multi-modal signal are represented as graph nodes, and a candidate edge set of causal relationship is constructed in combination with the change sequence relationship, triggering frequency and signal gain between the characteristic variables; the candidate edges are subjected to constraint screening to eliminate the edge relationship located in a low complexity region corresponding to a structure complexity less than a preset structure complexity threshold, so as to form an effective causal candidate network under structure modulation;

[0038] For each pair of candidate nodes, three indexes including an information gain value, a transition entropy value and a coupling strength in an abnormal fluctuation atlas are calculated, a causal relationship scoring function is constructed, and a reliability weight of a multi-modal causal edge is obtained in combination with dynamic weighting of a local gradient of a topological entropy field;

[0039] Based on the reliability weight, a causal candidate edge is subjected to structure construction, a multi-modal causal structure atlas is generated through a maximum weighted directed acyclic graph extraction algorithm, a node cluster in a fluctuation propagation path set in the multi-modal causal structure atlas and a local region with a propagation energy aggregation degree greater than a preset propagation energy aggregation threshold are identified, and whether the region constitutes an abnormal event is determined in combination with amplitude transition and phase disturbance at a corresponding position in the abnormal fluctuation atlas.

[0040] Preferably, the method for tracking a cross-modal spatiotemporal causal path comprises:

[0041] The identified abnormal event at a corresponding node in the multi-modal causal structure atlas is taken as a tracking starting point, and the modality type, event intensity, timestamp information and upstream and downstream causal edge information of the tracking starting point are recorded; for the case that different abnormal events occur concurrently, an abnormal source set containing different starting point nodes is constructed through topological adjacency analysis in the abnormal fluctuation atlas;

[0042] For each abnormal starting point node, a cross-modal causal path scoring function is designed to dynamically score the causal path starting from the node, and the path scoring function is weightedly obtained through the reliability score of the causal edge, the structure coupling strength between the modalities and the abnormal propagation strength at a corresponding position in the abnormal fluctuation atlas;

[0043] According to the scoring result of the path scoring function, a cross-modal spatiotemporal causal path is recursively searched in time sequence from the starting point node, in the path expansion process, edges with a path score higher than a preset path score threshold are preferentially selected, and when a modality switching occurs, the path is guided by referring to the modality coupling strength and the abnormal propagation gradient;

[0044] If the path score is lower than the preset path score threshold or a loop structure occurs, the current path tracking is terminated and a path pruning operation is performed; all cross-modal spatiotemporal causal paths meeting the conditions are integrated into a cross-modal spatiotemporal causal path atlas.

[0045] Compared with the prior art, the present application has the following beneficial effects:

[0046] This invention, by representing modal embedding units with nodes and explicit or implicit relationships with edges, achieves for the first time integrated modeling of different modalities at the structural level, enhancing the semantic connectivity and interactive parsing capabilities between data, and laying a unified structural foundation for subsequent topology modeling and dynamic analysis. By constructing a topological entropy spatiotemporal tensor field and combining it with wave-like diffusion partial differential equations, the propagation path of topological complexity in the graph is simulated, effectively capturing potential behaviors such as information flow diffusion and enhanced structural coupling, giving the originally abstract system evolution an observable and deducible dynamic description. By calculating the spectral radius of the local Jacobian matrix and combining it with the topological entropy criterion for drastic changes over time, a topological bifurcation point identification model is constructed, enabling accurate capture of critical states or sudden transitions in the system structure, significantly improving the early detection and root cause localization capabilities for project operational anomalies. The identified topological bifurcation points, combined with modality type, propagation direction, and time label, converge to form an anomaly response map, providing an intuitive decision-making basis for multimodal event causal analysis, key node tracing, and strategic intervention. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the multimodal project data analysis method of the present invention;

[0048] Figure 2 This is a schematic diagram of the multimodal project data analysis system of the present invention;

[0049] Figure 3 This is a schematic diagram of the method for outputting multimodal signals provided by the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1

[0052] Please see Figure 1 and Figure 3 As shown, this embodiment further illustrates the multimodal project data analysis method proposed in this invention, including:

[0053] In practical applications such as complex engineering projects, urban operation systems, industrial manufacturing and management information systems, information sources from multiple modalities such as images, audio, time series sensors, structured logs and text documents are often involved. These multi-modal data reflect the running state of the system, environmental changes and human-computer interaction behavior, and have high heterogeneity, dynamic and spatio-temporal correlation. Therefore, how to fuse and analyze multi-modal data to accurately identify abnormal states, infer causal relationships, and support project evaluation and decision-making is one of the core problems of current multi-modal intelligent analysis technology research.

[0054] Existing multi-modal project data analysis methods are usually based on static fusion strategies, using simple splicing, weighting or machine learning models to jointly model different modal data. This kind of method is difficult to describe the nonlinear coupling mechanism between modalities evolving over time, ignores the mutual feedback and dynamic influence between different information sources in the project running process, leading to slow response of the analysis model to sudden events, delay effects or structural mutations, affecting the early warning and tracking ability in practical applications.

[0055] Existing anomaly detection techniques mainly rely on sliding statistical features, threshold judgment or static classifiers, which are easily disturbed by noise in multi-modal scenarios, lack structural sensitivity, and are difficult to model system state changes in combination with the topological evolution information of multi-modal embedding graph. This leads to the risk of missing detection, high false alarm rate and difficulty in explanation in actual projects, especially in the case of asynchronous, heterogeneous coupling or missing modalities between modalities, traditional methods are difficult to form a stable and robust analysis path.

[0056] Existing graph structure analysis methods have obvious limitations in time modeling. Most graph methods cannot accurately determine the critical turning state of nodes or structures in the graph in the process of time evolution, and are difficult to describe the bifurcation point or non-stationary dynamic characteristics in the structure transition, and lack the ability to model the trend of graph topology structure change from the perspective of dynamics. It is even more impossible to combine the semantic hierarchical structure between multi-modalities to track the causal path, making it difficult to identify potential risks or key driving factors in the evolution process of the project system, limiting the predictability and explainability of intelligent analysis systems.

[0057] The present application is proposed in this background, and the present application proposes a multi-modal project data analysis method, comprising:

[0058] S1, collect heterogeneous modal project data, denoise and timestamp correction of each modal project data based on adaptive multi-scale filtering technology, and cross-modal time alignment, output multi-modal signal;

[0059] S2, construct a multi-level Boltzmann machine energy network to model the joint probability distribution of multi-modal signals; design a dynamic complexity measurement mechanism based on topological entropy to represent the project structure complexity and generate a topological entropy dynamic field;

[0060] S3, using the topological entropy dynamic field as a constraint condition, simulate the nonlinear coupling and spatiotemporal propagation between multi-modal signals using a multi-dimensional covariant field framework, abstract multi-modal signals as string vibration modes, identify local topological defects in the topological entropy dynamic field, and generate abnormal fluctuation atlas;

[0061] S4, drawing on the selective regulation characteristics of glial cells on neural signals, combining the abnormal fluctuation atlas, dynamically adjusting the weight coupling strength of each modal signal, and performing inter-modal feedback regulation;

[0062] S5, based on the multi-modal signals after weight adjustment, design a multi-modal causal structure self-construction algorithm, combine the topological entropy dynamic field and the abnormal fluctuation atlas, identify abnormal events, and track cross-modal spatiotemporal causal paths, and transmit the identified abnormal events to the data analysis terminal.

[0063] The method for collecting heterogeneous modal project data comprises:

[0064] Deploying n data collection channels with different sensing types on the project site, the data collection channels include image collection channels, voice collection channels, structured data interface channels, time series signal collection channels, and unstructured text collection channels, and collecting heterogeneous modal project data;

[0065] The heterogeneous modal project data includes image modal, sound modal, device operation parameter modal, time series signal modal data, and project document modal data; based on data source, physical type and semantic attribute, the heterogeneous modal project data is identified and labeled by modal type, and a data structure with modal label is generated.

[0066] The method for outputting multi-modal signals comprises:

[0067] According to the modal type of the heterogeneous modal project data, a multi-scale filter bank is constructed, the multi-scale filter bank includes a filter structure based on wavelet transform and empirical mode decomposition, the signal trend is extracted and the modal specific noise is suppressed; analyze the sampling frequency, noise characteristics and frequency spectrum distribution of each heterogeneous modal project data, and dynamically adjust the filtering scale and adaptive denoising based on the modal specific error feedback mechanism;

[0068] Anomaly detection is performed in timestamp order to identify missing values, mutation points and pseudo-synchronous segments, and an interpolation compensation method is used for timestamp correction; the heterogeneous modal item data with timestamps is input into a cross-modal time alignment algorithm, non-linear alignment is performed on the heterogeneous modal item data with different sampling frequencies and synchronization delays, and finally multi-modal signals under a unified time axis are generated.

[0069] The method for modeling the joint probability distribution of the multi-modal signal comprises:

[0070] The multi-modal signal is converted into a unified embedding representation, different embedding strategies are used for different modalities, the embedding strategies include using a convolution feature extraction method for the image modality to generate visual embedding, using a recurrent neural network structure for the sound modality to generate sequence embedding, using a semantic encoder for the item document modality to generate a context vector, and after standardizing all modality embeddings, inputting them into a Boltzmann machine;

[0071] A multi-level Boltzmann machine network structure is constructed, each layer includes a learnable hidden variable node, and is connected to the nodes of the previous layer and the next layer by a symmetric connection to form a undirected graph structure, forming an energy network that characterizes the joint distribution of the semantic of the multi-modal signal;

[0072] An energy function of the energy network is designed to characterize the joint activation relationship between the modality embedding and the hidden variable, a dynamic field based on topological entropy is introduced to adjust the sensitivity of the connection weight between modalities; the parameter distribution used for joint modeling and reasoning is obtained, and finally the joint probability distribution model of the multi-modal signal in the joint modal space is output.

[0073] The method for generating a topological entropy dynamic field comprises:

[0074] Based on the multi-modal signal, a unified multi-modal embedding graph is constructed, the nodes in the multi-modal embedding graph represent the embedding units of the modal signals, and the edges represent the explicit or implicit association relationship between the modal signals; for any node in the multi-modal embedding graph, the topological entropy of the node is defined under a preset sliding time window;

[0075] The topological entropy of the node is: ; wherein, represents the topological entropy of the node at the time point . represents the set of adjacent nodes of the node . represents the normalized edge weight between the node and the adjacent node at the time point . represents the th node; represents the a node; and denotes an index of a node; denotes an index of a time point;

[0076] The topology entropy of each node in the multi-modal embedding graph changing over time is constructed as a spatiotemporal continuous distribution tensor, and a wave-like diffusion mechanism is introduced to model the propagation behavior, and a control partial differential equation of topology entropy propagation is defined.

[0077] The control partial differential equation is: ; denotes a structure diffusion coefficient, which controls the diffusion speed of topology entropy propagation on the multi-modal embedding graph; denotes a heterogeneous coupling adjustment factor, which controls the intervention degree of topology entropy evolution between different modalities; is a graph Laplacian operator, which represents the spatial diffusion trend of topology entropy on the multi-modal embedding graph; is a modal coupling tensor, which represents the coupling strength between different modalities at position and time point ; denotes the spatial position of a node in the multi-modal embedding graph; a discrete method on the graph structure is used to numerically solve the control partial differential equation, and generate the topology entropy dynamic field at any time;

[0078] To identify the sudden change points in the topological structure, the bifurcation theory is used on the topology entropy dynamic field to determine the stability of the node structure; the local Jacobian matrix of each node in the current window is calculated, and the spectral radius is extracted to determine whether the spectral radius meets the topology bifurcation condition;

[0079] The topology bifurcation condition is: wherein, denotes the maximum modulus of all eigenvalues of the Jacobian matrix; denotes the propagation sensitivity Jacobian matrix of node ; denotes the derivative of topology entropy with respect to time; denotes a preset topology entropy increment threshold; is a logical symbol, which represents that two conditions are simultaneously true in a logical expression, and in the topology bifurcation condition, it is embodied that the two conditions must be met simultaneously;

[0080] When the topology bifurcation condition is met, it is determined that the node is a topology bifurcation point; all identified topology bifurcation points are combined with the modal type, time stamp and propagation direction to draw a topology entropy abnormal response graph.

[0081] The problems existing in the prior art are solved: the existing multi-modal data analysis mostly adopts a static fusion strategy, which cannot depict the complex coupling behavior between different modalities evolving over time, and cannot reflect the dynamic mutual influence of different information sources in the project process; most of the existing methods use rule thresholds, sliding statistics or machine learning classifiers for anomaly detection, lack modeling and identification of the change trend of the topological structure in the multi-modal embedding graph, are prone to missing or misjudging critical evolution nodes, cannot enhance the robustness of anomaly detection using the structural information between modalities, are difficult to identify deep changes in the structure level, and lack a unified judgment criterion for the comprehensive judgment of heterogeneous source data. The graph analysis method of the prior art cannot judge the critical turning state of a node in the time evolution process, cannot realize early warning and interpretable tracking of the project system evolution process, lacks the ability to understand the structure transition in the graph from the perspective of non-stationary dynamics evolution, and is difficult to judge where and when to face an irreversible evolution trend or risk surge point.

[0082] The beneficial effects of the prior art are: through the node representation modality embedding unit and the edge representation explicit or implicit association relationship, the integrated modeling of different modalities at the structure level is realized for the first time, the semantic connectivity and interactive analysis capability between data are enhanced, and a unified structure foundation is laid for subsequent topological modeling and dynamic analysis. By constructing a topological entropy space-time tensor field and combining a wave-like diffusion partial differential equation, the propagation path of topological complexity in the graph is simulated, potential behaviors such as information flow diffusion and structure coupling enhancement are effectively captured, and the originally abstract system evolution has an observable and deducible dynamic description. By calculating the spectral radius of the local Jacobian matrix and combining the topological entropy with the criterion for the sharp change of time, a topological bifurcation point identification model is constructed, the critical state or sudden change of the system structure is accurately captured, and the early discovery and root cause positioning capability for project operation anomalies are significantly improved. The identified topological bifurcation points are combined with the modality type, propagation direction and time label to form an abnormal response graph, which provides an intuitive decision basis for multi-modal event causal analysis, key node tracing and strategy intervention.

[0083] The method of abstracting multi-modal signals into string vibration modes comprises:

[0084] Based on the multi-modal signal data after denoising and time alignment, a multi-dimensional covariant tensor field is constructed using a sliding time window and modality space distribution information; the topological entropy dynamic field is introduced as a modulation constraint in the construction process of the multi-dimensional covariant tensor field, and the covariance, mutual information and high-order statistics between different modalities are calculated; according to the covariance, mutual information and high-order statistics between different modalities at different time periods, the covariant coefficients are dynamically given an entropy-oriented weighting factor, and the coupling characteristics of the topological complex region are highlighted;

[0085] Under the constraint of topological entropy, the coupling mapping relationship between multi-modal signals is modeled by a nonlinear kernel function to generate a coupling path containing time, space and modal dimensions; and the coupling path is abstracted as a multi-dimensional string vibration model under controlled boundary conditions.

[0086] The method for generating an abnormal fluctuation atlas comprises:

[0087] Each modal signal is represented as a vibration string element that can be excited and coupled, and the energy propagation and resonance behavior in the covariant field are simulated, the dominant modal frequency, standing wave node and phase jump region are extracted based on the multi-dimensional string vibration model, the local topological defects in the covariant field that exhibit energy aggregation and abrupt propagation are identified, and the abnormal fluctuation atlas containing abnormal response paths and perturbation region identifiers is constructed.

[0088] The method for inter-modal feedback regulation comprises:

[0089] Based on the abnormal fluctuation atlas, each node in the multi-modal embedding graph corresponding to the multi-modal signals is identified and labeled to determine the node region with structural abnormalities; for each node, a feature parameter is extracted; the feature parameter includes the modal type, the time stamp, the number of adjacent nodes and the abnormal fluctuation amplitude, and the regulation intensity index of the node at the current time is calculated.

[0090] Referring to the regulation mechanism of glial cells on synaptic signal transmission in the nervous system, the modal coupling weight between the node and the adjacent node is selectively adjusted according to the regulation intensity index of each node, and when the abnormal fluctuation trend of any modal region is greater than a preset abnormal fluctuation trend threshold, the connection strength between the modal in the corresponding region and other modes is enhanced;

[0091] When the participation of any modal in the abnormal response is less than a preset participation threshold, the weight of the modal in the coupling structure is reduced; the feedback path structure between the modes is dynamically updated in combination with the topological structure evolution trajectory recorded in the abnormal fluctuation atlas, and a feedback regulation curve of the coupling strength between the modes changing with time is generated.

[0092] The method for identifying abnormal events comprises:

[0093] The feature variables of each modal in the multi-modal signals are represented as graph nodes, and a candidate edge set of causal relationships is constructed in combination with the change sequence, triggering frequency and signal gain of the feature variables; the candidate edges are constrained and selected to eliminate the edge relationships located in low-complexity regions with a structural complexity less than a preset structural complexity threshold to form an effective causal candidate network under structural modulation.

[0094] For each pair of candidate nodes, three indicators including information gain value, transfer entropy value and coupling strength in the abnormal fluctuation atlas are calculated, a causal relationship scoring function is constructed, and the reliability weight of the multi-modal causal edge is obtained by combining the local gradient dynamic weighting of the topological entropy field;

[0095] Based on the reliability weight, the structure of the causal candidate edge is constructed, and the multi-modal causal structure atlas is generated by the maximum weighted directed acyclic graph extraction algorithm; the node cluster in the fluctuation propagation path set and the local area with a propagation energy aggregation degree greater than a preset propagation energy aggregation threshold in the multi-modal causal structure atlas are identified, and the amplitude transition and phase disturbance of the corresponding position in the abnormal fluctuation atlas are combined to determine whether the area constitutes an abnormal event.

[0096] The method for tracking the cross-modal spatiotemporal causal path includes:

[0097] The identified abnormal event in the corresponding node of the multi-modal causal structure atlas is taken as the tracking starting point, and the modality type, event intensity, timestamp information and upstream and downstream causal edge information of the tracking starting point are recorded; for the case of concurrent occurrence of different abnormal events, the abnormal source set containing different starting nodes is constructed by topological adjacency analysis in the abnormal fluctuation atlas;

[0098] For each abnormal starting node, a cross-modal causal path scoring function is designed to dynamically score the causal path starting from the node, and the path scoring function is obtained by weighting the reliability score of the causal edge, the structural coupling strength between the modalities and the abnormal propagation strength of the corresponding position in the abnormal fluctuation atlas;

[0099] According to the scoring results of the path scoring function, the cross-modal spatiotemporal causal path is recursively searched in time sequence from the starting node, and in the path expansion process, the edges with a path score higher than a preset path score threshold are preferentially selected, and the modal coupling strength and abnormal propagation gradient are referred to for guidance when a modal switch occurs;

[0100] If the path score is lower than the preset path score threshold or a loop structure occurs, the current path tracking is terminated and a path pruning operation is performed; all cross-modal spatiotemporal causal paths that meet the conditions are integrated into a cross-modal spatiotemporal causal path graph.

[0101] The preset abnormal fluctuation trend threshold is set by the staff based on the historical data analysis results, and the historical analysis process includes that the system collects the average value of multiple abnormal fluctuation trends as a reference, which can be adjusted by the staff during the system operation; similarly, the preset participation threshold, the preset structural complexity threshold, the preset propagation energy aggregation threshold and the preset path score threshold are set.

[0102] In this embodiment, the node represents the modal embedding unit, and the edge represents the explicit or implicit correlation. For the first time, different modalities are integrated at the structural level, enhancing the semantic connectivity and interaction analysis capability between data, and laying a unified structural foundation for subsequent topology modeling and dynamic analysis. By constructing a topology entropy spatiotemporal tensor field and combining it with a wave-like diffusion partial differential equation, the propagation path of topology complexity in the graph is simulated, effectively capturing potential behaviors such as information flow diffusion and structure coupling enhancement, making the originally abstract system evolution have observable and deducible dynamic description. By calculating the spectral radius of the local Jacobian matrix and combining the topology entropy with the criterion for the dramatic change of time, a topology bifurcation point identification model is constructed to accurately capture the critical state or sudden transition of the system structure, significantly improving the early detection and root cause positioning ability of project operation abnormalities. The identified topology bifurcation points, combined with modal types, propagation directions, and time labels, converge to form an abnormal response atlas, providing intuitive decision-making basis for multi-modal event causal analysis, key node tracing, and strategy intervention.

[0103] Embodiment 2

[0104] Please refer to Figure 2 The embodiment does not describe some parts in detail, which can be seen from the description of embodiment 1. A multi-modal project data analysis system is provided, which includes:

[0105] A data processing representation module collects heterogeneous modal project data, denoises and timestamps corrects each modal project data based on adaptive multi-scale filtering technology, and performs cross-modal time alignment to output multi-modal signals;

[0106] A multi-modal energy optimization module constructs a multi-level Boltzmann machine energy network to model the joint probability distribution of multi-modal signals; a dynamic complexity measurement mechanism based on topology entropy is designed to represent the project structure complexity and generate a topology entropy dynamic field;

[0107] A multi-dimensional covariant field construction analysis module uses a multi-dimensional covariant field framework to simulate the nonlinear coupling and spatiotemporal propagation between multi-modal signals, abstracts the multi-modal signals into string vibration patterns, identifies local topological defects in the topology entropy dynamic field, and generates an abnormal fluctuation atlas, with the topology entropy dynamic field as a constraint condition;

[0108] A multi-modal information flow adaptive module dynamically adjusts the weight coupling strength of each modal signal based on the abnormal fluctuation atlas, and constructs feedback regulation between modalities, inspired by the selective regulation characteristics of glial cells on neural signals;

[0109] A multi-modal causal tracing module designs a multi-modal causal structure self-construction algorithm based on the multi-modal signals adjusted by the weight, identifies abnormal events, traces the cross-modal spatiotemporal causal path, and transmits the identified abnormal events to the data analysis terminal, combined with the topology entropy dynamic field and the abnormal fluctuation atlas.

[0110] Since the electronic device introduced in the embodiment is the electronic device used for implementing the multi-modal project data analysis method in the embodiment, based on the multi-modal project data analysis method introduced in the embodiment, those skilled in the art can understand the specific implementation of the electronic device in the embodiment and various changes thereof, so how the electronic device implements the method in the embodiment will not be introduced in detail. As long as those skilled in the art implement the electronic device used for the multi-modal project data analysis method in the embodiment, it belongs to the scope of protection of the present application.

[0111] The above formulas are dimensionless values, the formulas are obtained by collecting a large amount of data to simulate the latest real situation, and the preset parameters and threshold values in the formulas are set by those skilled in the art according to the actual situation.

[0112] The above is only the preferred embodiment of the present application, the protection scope of the present application is not limited to the above embodiment, any technical solution under the idea of the present application belongs to the protection scope of the present application. It should be pointed out that, for ordinary technical operators in the technical field, some improvements and decorations without departing from the principle of the present application are also regarded as the protection scope of the present application.

Claims

1. A multi-modal project data analysis method, characterized in that, The method comprises the following steps: S1, collecting heterogeneous modal project data, denoising and timestamp correcting each modal project data based on adaptive multi-scale filtering technology, and performing cross-modal time alignment to output multi-modal signals; The method for collecting heterogeneous modal project data comprises: deploying n data collection channels with different perception types on the project site to collect heterogeneous modal project data; the heterogeneous modal project data comprises image modal, sound modal, equipment operation parameter modal, time series signal modal data and project document modal data; modal type identification and labeling are performed on the heterogeneous modal project data based on data source, physical type and semantic attribute to generate a data structure with modal labels; S2, constructing a multi-level Boltzmann machine energy network to model the joint probability distribution of the multi-modal signals; a dynamic complexity measurement mechanism based on topological entropy is designed to represent the project structure complexity and generate a topological entropy dynamic field; The method for generating the topological entropy dynamic field comprises: based on the multi-modal signals, a unified multi-modal embedding graph is constructed; for any node in the multi-modal embedding graph, the topological entropy of the node is defined under a preset sliding time window; the topological entropy of each node in the multi-modal embedding graph is constructed as a spatiotemporal continuous distribution tensor, and a wave-like diffusion mechanism is introduced to model the propagation behavior, and a control partial differential equation of topological entropy propagation is defined; the control partial differential equation is numerically solved by using a discretization method on the graph structure to generate the topological entropy dynamic field at any time; In order to identify the sudden change points in the topological structure, the bifurcation theory is used to judge the stability of the node structure on the topological entropy dynamic field; the local Jacobian matrix of each node under the current window is calculated, and the spectral radius is extracted to determine whether the spectral radius satisfies the topological bifurcation condition; when the topological bifurcation condition is satisfied, it is determined that the node is a topological bifurcation point; all identified topological bifurcation points are combined with the modal type, timestamp and propagation direction to draw a topological entropy abnormal response graph; S3, using the topological entropy dynamic field as a constraint condition, simulating the nonlinear coupling and spatiotemporal propagation among the multi-modal signals by using a multi-dimensional covariant field framework, abstracting the multi-modal signals as a string vibration mode, identifying local topological defects in the topological entropy dynamic field, and generating an abnormal fluctuation graph; The method for abstracting the multi-modal signals as a string vibration mode comprises: based on the multi-modal signal data, a multi-dimensional covariant tensor field is constructed by using a sliding time window and modal spatial distribution information; the topological entropy dynamic field is introduced into the multi-dimensional covariant tensor field construction process as a modulation constraint, and the covariance, mutual information and high-order statistics between different modalities are calculated; the coupling mapping relationship between the multi-modal signals is modeled by using a nonlinear kernel function to generate a coupling path containing time, space and modal dimensions; the coupling path is abstracted as a multi-dimensional string vibration model under controlled boundary conditions; S4, referring to the selective regulation characteristics of glial cells on neural signals, combining the abnormal fluctuation graph, dynamically adjusting the weight coupling strength of each modal signal, and performing inter-modal feedback regulation; The method for performing inter-modal feedback regulation comprises: Based on the abnormal fluctuation atlas, each node in the multi-modal embedding graph corresponding to the multi-modal signal is identified and labeled to determine the node area with structural abnormalities; for each node, a feature parameter is extracted, and an adjustment intensity index of the node at the current time is calculated; Learning from the regulation mechanism of glial cells in the nervous system for synaptic signal transmission, the modal coupling weight between the node and the adjacent node is selectively adjusted according to the adjustment intensity index of the node, and the feedback path structure between the modes is dynamically updated in combination with the topological structure evolution trajectory recorded in the abnormal fluctuation atlas, to generate a feedback adjustment curve of the coupling strength between the modes over time; S5, based on the multi-modal signal adjusted by the weight, a multi-modal causal structure self-construction algorithm is designed, the abnormal event is identified in combination with the topological entropy dynamic field and the abnormal fluctuation atlas, and the cross-modal spatiotemporal causal path is tracked, and the identified abnormal event is transmitted to a data analysis terminal.

2. The multi-modal item data analysis method of claim 1, wherein, The method for outputting the multi-modal signal comprises: According to the modal type of the heterogeneous modal item data, a multi-scale filter bank is constructed, signal trends are extracted, and modal-specific noise is suppressed; abnormal detection is performed in chronological order, missing values, mutation points and pseudo-synchronous segments are identified, and timestamp correction is performed by using an interpolation compensation method; the heterogeneous modal item data with timestamps are input into a cross-modal time alignment algorithm to generate multi-modal signals under a unified time axis.

3. The multi-modal item data analysis method of claim 2, wherein, The method for modeling the joint probability distribution of the multi-modal signal comprises: The multi-modal signal is converted into a unified embedding representation, different embedding strategies are used for different modes, and after standardization processing of all modal embeddings, the unified embedding representation is input into a Boltzmann machine; a multi-level Boltzmann machine network structure is constructed to form an energy network; An energy function of the energy network is designed, an adjustment factor based on the topological entropy dynamic field is introduced to adjust the sensitivity of the connection weight between the modes; a parameter distribution used for joint modeling and reasoning is obtained, and finally a joint probability distribution model of the multi-modal signal in the joint modal space is output.

4. The multi-modal item data analysis method of claim 3, wherein, The method for generating the abnormal fluctuation atlas comprises: Each modal signal is represented as a vibrating string element, energy propagation and resonance behavior in the covariant field are simulated, dominant modal frequencies, standing wave nodes and phase jump regions are extracted based on a multi-dimensional string vibration model, local topological defects showing energy aggregation and mutation propagation in the covariant field are identified, and an abnormal fluctuation atlas containing abnormal response paths and disturbance area identifiers is constructed.

5. The multi-modal item data analysis method of claim 4, wherein, The method for identifying the abnormal event comprises: Feature variables of each mode in the multi-modal signal are represented as graph nodes, and a causal relationship candidate edge set is constructed in combination with the change sequence, triggering frequency and signal gain between the feature variables; the candidate edges are selected and screened, and the edge relationships located in a low complexity region with a structural complexity less than a preset structural complexity threshold are removed to form an effective causal candidate network under structural modulation; A causal relationship scoring function is constructed, and a reliability weight of the multi-modal causal edge is obtained in combination with the local gradient dynamic weighting of the topological entropy field; based on the reliability weight, the causal candidate edge is structurally constructed, and a multi-modal causal structure atlas is generated by using a maximum weighted directed acyclic graph extraction algorithm; Identify the node cluster in the fluctuation propagation path set in the multi-modal causal structure graph, the local area where the propagation energy aggregation degree is greater than the preset propagation energy aggregation degree threshold, and determine whether the area constitutes an abnormal event in combination with the amplitude transition and phase disturbance of the corresponding position in the abnormal fluctuation graph.

6. The multi-modal item data analysis method of claim 5, wherein, The method for tracking the cross-modal spatiotemporal causal path comprises: Taking the corresponding node of the identified abnormal event in the multi-modal causal structure graph as the tracking starting point, for the concurrent occurrence of different abnormal events, constructing an abnormal source set containing different starting nodes through topological adjacency analysis in the abnormal fluctuation graph; For each abnormal starting node, design a cross-modal causal path scoring function to dynamically score the causal path starting from the node, and recursively search the cross-modal spatiotemporal causal path in time sequence from the starting node according to the scoring result of the path scoring function; During path expansion, preferentially select edges with path scores higher than a preset path score threshold, and guide by referring to the modal coupling strength and abnormal propagation gradient when modal switching occurs; if the path score is lower than the preset path score threshold or a loop structure appears, terminate the current path tracking and perform path pruning operation; integrate all cross-modal spatiotemporal causal paths meeting the conditions into a cross-modal spatiotemporal causal path graph.

Images