Abnormal process detection method based on multi-modal fusion

Abstract

The invention discloses an abnormal process detection method based on multi-modal fusion, and relates to the technical field of data processing and intelligent monitoring, and the method comprises the following steps: extracting a time sequence track of multi-source data under a unified event time baseline, constructing a cross-modal phase difference slope fingerprint, and quantifying an instantaneous drift gradient; based on the phase difference slope fingerprint, obtaining an anti-fact playback evidence chain, replaying the dynamic fragment frame by frame, positioning a response dislocation trigger node, and generating a time sequence offset spectrum; and based on the time sequence offset spectrum, introducing a phase lock consistency index and quantile manifold embedding, identifying a decoherence interval, and solidifying the time sequence coordinates of the anchor points. According to the method, the cross-modal phase difference slope fingerprint and the causal topology path are constructed, fine-grained positioning and trend node identification of abnormal response are realized, time reversal control and conjugate phase intervention are combined, a reverse energy channel is constructed, closed-loop regulation and control are completed, and the anomaly detection precision and robustness in a multi-source heterogeneous environment are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing and intelligent monitoring technology, specifically to an abnormal process detection method based on multimodal fusion. Background Technology

[0002] "Multimodal fusion-based anomaly detection" refers to the use of data from multiple sources (such as sensor signals, images and videos, text logs, and voice information) in complex business or industrial processes. By fusing and analyzing the complementary features of these multimodal data, a unified representation and correlation model is established to enable real-time identification and early warning of potential anomalies during process operation. This method overcomes the limitations of single-data source detection, comprehensively considering temporal, spatial, and semantic features to more fully and accurately reveal hidden anomaly patterns and potential risks in the process, thereby improving the robustness and intelligence of process monitoring.

[0003] Existing technologies have the following shortcomings: In the face of rapidly changing and complex operating scenarios, anomalous signals from multimodal data often fail to maintain strict temporal consistency, easily leading to abrupt phase difference changes between sources. Different modal data should exhibit mutual corroboration and amplification effects when anomalies occur, such as increased sensor signal strength, abrupt changes in image features, and sudden enhancements in acoustic signals. However, due to the lack of dynamic phase alignment and drift compensation mechanisms in existing technologies, when the response speed of one modality lags or advances, a decoherence effect occurs during the fusion process. This decoherence effect not only weakens the overall strength of the anomalous signal but may also mask true anomalous characteristics, making the system's output judgment result appear normal, thus directly causing detection delays. Especially in high-time-sensitive fields such as industrial production, energy dispatching, and financial transactions, such delays can cause anomalous risks to go undetected during critical handling windows, potentially leading to equipment damage, production stoppages, or even systemic accidents.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide an abnormal process detection method based on multimodal fusion to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an anomaly detection method based on multimodal fusion, comprising the following steps: Temporal trajectories of multi-source data are extracted under a unified event time baseline, cross-modal phase difference slope fingerprints are constructed, and instantaneous drift gradients are quantified. Based on the phase difference slope fingerprint, the counterfactual playback evidence chain is obtained, the dynamic segment is replayed frame by frame, the response misalignment trigger node is located, and the temporal offset spectrum is generated. Based on the temporal offset spectrum, phase-locked consistency index and quantile manifold embedding are introduced to identify decoherence intervals and solidify the temporal coordinates of anchor points; Based on the anchor point time series coordinates, a phase traction probe is performed to inject micro-amplitude phase perturbations into the multimodal response, measure the residual gradient field, invert the causal topology, and locate the energy convergence path. Based on the energy convergence path, an adaptive evidence chain is constructed to cross-validate the energy trajectory, anomaly spectrum and trigger source region, generate a set of credible trend nodes, and quantify the intervention amplitude; Based on the trend node set, a time-reversal phase-gated instruction is generated to drive the injection of a phase conjugate micro-envelope into the programmable polarization metasurface. A reverse energy channel is established in the trigger source region to perform a pulse-level quenching operation. The calibration matrix is ​​then written back to the event time baseline to complete the closed-loop control of the anomaly.

[0007] Preferably, the phase difference slope fingerprint construction process is as follows: A global timeline with a unified time baseline is established, and the data sampling time series of the image acquisition device, acoustic receiving device, inertial measurement unit, environmental sensor and electronic probe are extracted. All data frames are linearly normalized and interpolated to achieve time alignment. Perform dynamic change detection of multi-source response, identify the dominant change nodes of each modality data and complete the annotation on a unified time baseline; Calculate the slope of the phase difference increment between mode pairs and construct a cross-modal phase difference slope fingerprint matrix; Instantaneous drift gradients are quantized in slope fingerprints to generate drift gradient heatmaps. Modal shift behaviors with early or delayed responses are extracted and used as a unified alignment reference structure for subsequent anomaly detection.

[0008] Preferably, the time-series offset spectrum generation steps are as follows: Based on the constructed cross-modal phase difference slope fingerprint matrix, time periods with drastic phase difference slope changes are identified as extreme dynamic segments. Extract the original data frames corresponding to extreme dynamic segments and reconstruct the basic replay sequence under a unified event time baseline; Using the image modality as a reference, the dominant response time of other modalities is perturbed within the constructed time window to generate counterfactual scenarios, and response matching metrics are calculated for each scenario. The system identifies response misalignment trigger nodes, constructs multimodal feature vectors, expands the behavior analysis interval, and finally generates a complete temporal offset spectrum for subsequent decoherence interval identification.

[0009] Preferably, the anchor point time-series coordinate solidification step is as follows: Based on the constructed time-series offset spectrum, phase-locked loop consistency index analysis is performed, and time windows with low consistency indices are extracted as preliminary decoherence candidate segments. Based on the decoherent candidate segment, perform quantized manifold embedding operation to extract the trajectory offset behavior of the modal response in the embedding space, and mark the time points when the average curvature and the maximum offset exceed the set threshold. The low-consistency segments and trajectory offset points are intersected, and time points that simultaneously meet both criteria are selected. Their time coordinates are extracted and fixed to form a decoherent anchor point sequence, which is used as a time reference for subsequent causal topological inversion.

[0010] Preferably, the energy convergence path location process is as follows: Based on the fixed anchor point time series coordinates, a perturbation observation window is constructed on a unified time baseline to inject micro-perturbations into a single mode while keeping the original response states of other modes unchanged. A residual gradient field is constructed for the multimodal response sequences before and after the perturbation. A directed edge is established along the path direction of the maximum residual amplitude, pointing from the perturbation source to the response mode, and a causal topological graph is generated. Nodes with more than a threshold number of incoming edges and concentrated residual distribution are extracted from the topological graph and identified as trend nodes in the energy convergence path. These trend nodes are then aggregated to form a set of trend nodes, which serves as input for subsequent dynamic regulation.

[0011] Preferably, the steps for generating a set of reliable trend nodes are as follows: Energy trajectories are extracted based on causal topology and the modal types, response amplitudes, and propagation sequences are labeled. By combining historical anomaly spectrum information to compare response trends, matching segments are extracted and trend confirmation candidate nodes are screened. Based on the initial response source in the energy path, causal path consistency analysis is performed to generate a trend response consistency index and identify credible trend nodes. The response intensity, path concentration, and response stability are calculated sequentially for trusted trend nodes, generating an intervention instruction set that includes time location, modality category, and amplitude settings, which serves as the input for dynamic control.

[0012] Preferably, the intervention amplitude in the intervention instruction set is set based on trend nodes where the response intensity exceeds twice the average response amplitude, the path concentration is higher than the intersection of three propagation paths, and the response direction does not reverse during the duration.

[0013] Preferably, the steps for generating time-reversal gating commands based on the trend node set, driving the polarization metasurface to inject conjugate phase micro-envelopes, constructing an inverse energy channel in the trigger source region, implementing pulse quenching, and writing the calibration results back to the event time baseline are as follows: Construct a time-reversal mapping chain and generate phase-gated instructions that include time, mode, orientation, and priority; Activate the programmable polarization metasurface and complete phase modulation in the intervention region; Inject a phase-conjugate micro-envelope that is opposite to the trend node response trajectory; Construct a reverse energy channel between the trend node and the source trigger that satisfies the conditions of phase change, amplitude suppression and energy density decrease, and perform a pulse-level extinguishing operation; Extract the response differences before and after each modality intervention to construct a calibration matrix and write the results back to a unified event time baseline to achieve closed-loop dynamic control.

[0014] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention constructs a cross-modal phase difference slope fingerprint to accurately characterize the temporal offset and instantaneous drift features between modes. Combined with counterfactual replay, it achieves fine-grained localization of anomalous response paths. Then, it utilizes phase-locked consistency index and quantile manifold embedding to identify decoherence intervals and solidify anchor point coordinates. Furthermore, through phase-driven perturbation and residual gradient inversion, it constructs causal topology and energy convergence paths, ensuring that trend node identification has physical meaning and causal basis. Based on this, a time-reversal control strategy drives the precise injection of conjugate phase perturbations into a programmable polarization metasurface, realizing the construction of reverse energy channels and local extinguishing operations for anomalous response paths. Finally, it completes state calibration and writes back to a unified time baseline, forming a self-closing dynamic control mechanism. Compared to existing static fusion or rule-matching-based detection methods, this scheme not only possesses higher robustness and response accuracy but also can dynamically adapt to sudden anomalous patterns in multi-source heterogeneous data environments. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0016] Figure 1 This is a flowchart of the abnormal process detection method based on multimodal fusion of the present invention. Detailed Implementation

[0017] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0018] This invention provides, for example Figure 1 The abnormal process detection method based on multimodal fusion shown includes the following steps: Under a unified event time baseline, time-series trajectories of multi-source data are extracted, cross-modal phase difference slope fingerprints are constructed, and instantaneous drift gradients are quantized in the phase difference slope fingerprints to provide a unified reference for dynamic anomaly localization. To achieve high-precision alignment of multi-source response data under a unified time baseline and to provide a continuous and comparable reference for dynamic anomaly detection, the following processing flow based on time baseline and phase difference slope fingerprint is proposed, with the specific steps as follows: For multi-source heterogeneous data from image acquisition devices, acoustic receiving devices, inertial measurement units, environmental sensors, and high-frequency sampling electronic probes, the original sampling timestamp sequence of each data source is extracted, and a corresponding data buffer time axis is established. By counting the number of data frames generated by each device per unit second, the actual sampling frequency of that source is calculated, and the theoretical generation time of each frame of data is derived using this frequency. Subsequently, based on a selected unified event time baseline (such as a global time axis with millisecond granularity and system startup time as the starting point), the original timestamps of all data sources are linearly normalized, and interpolation reconstruction technology is used to synchronize the key data frames of each data stream under a unified time coordinate. The interpolation reconstruction uses a linear interpolation algorithm based on the interval between the two most recent frames, and performs frame interpolation operation on data segments with sampling intervals exceeding a preset threshold (such as 20ms) to ensure that there are no discontinuous or overlapping frames on the unified time axis. After the above processing, a complete multi-source data time alignment structure is formed, with all data frames arranged on a unified time baseline with millisecond-level precision, laying the foundation for time series comparison and interaction analysis between multimodal signals.

[0019] For the aligned data structure described above, dynamic change detection of multi-source responses is performed to identify key temporal feature points that can be used to construct cross-modal phase difference slope fingerprints. In image data, a mutation point extraction method based on inter-frame pixel grayscale difference is used to calculate the brightness change rate of the central and edge regions of the image frame by frame, marking time points where the brightness suddenly increases or decreases by more than three standard deviations from the original baseline. In acoustic data, envelope curves are extracted and energy transition points are located. Three consecutive sampling points with opposite directions of envelope change rate are defined as local extrema, and the extremum timestamps are used as acoustic transition nodes. In inertial measurement data, points where the angular velocity change rate changes by more than a set threshold within 1 second are selected as significant response points. In environmental data, such as temperature, humidity, and electromagnetic interference signals, inflection points in the slope curves with long-period trend changes exceeding 5% are identified. Through these methods, the set of dominant temporal change nodes for each modality of data is obtained and labeled on a unified time baseline. Next, based on the time difference of the dominant change points between modal pairs within the same event trigger window, a phase difference sequence is constructed, and the average incremental slope of this phase difference over time is calculated within a time sliding window, serving as a preliminary representation of the slope fingerprint. To enhance the multidimensional expressive power of the fingerprint structure, the slope sequences of all modal combination pairs are superimposed into a two-dimensional time-modality matrix, called the cross-modal phase difference slope fingerprint matrix, which is used to capture the temporal synchronicity evolution trend between data.

[0020] After constructing the cross-modal phase difference slope fingerprint matrix, the instantaneous drift gradient is further quantified within this fingerprint structure to identify dynamic misalignment behavior between modal responses within a short time window. Specifically, a time sliding window approach is used. Within each window, the change in the first-order temporal derivative of the slope fingerprint is calculated, and gradient abrupt change points are extracted. The threshold for judgment is the mean of the slope change in the previous time window plus twice the standard deviation. After identifying a slope abrupt change, the specific time coordinates of its occurrence are recorded, and the original data frame of the corresponding modality is retrieved to analyze whether it is a response delay (i.e., the current modality's abrupt change time is later than other modalities) or a response advance (i.e., the abrupt change time is earlier than other modalities). Based on this, a drift gradient heatmap is constructed with time as the horizontal axis, drift direction as the vertical axis, and drift amplitude as a color map. This heatmap displays the drift trend, offset direction, and intensity of change of each modality within different time periods, serving as an important feature structure for characterizing local misalignment behavior between modalities, and also as an important reference for subsequent identification of decoherence segments.

[0021] The slope fingerprint matrix and drift gradient heatmap are combined as a unified alignment reference structure for dynamic anomaly detection. Based on this reference structure, time-segment clustering analysis is performed to identify a set of time windows with highly consistent drift patterns, forming a candidate set of anomaly precursor segments. Furthermore, the drift direction consistency scalar of each modality within the candidate segments is extracted. If three or more modalities exhibit abrupt changes with consistent drift directions and similar amplitudes within the same time period, this time period is marked as a possible synchronization failure window. Simultaneously, the corresponding dominant drift direction and initial response time are fixed within this window and used as reference anchors in subsequent counterfactual replay and causal structure inversion to support accurate time alignment and causal inference of cross-modal response behavior under non-static data conditions.

[0022] Based on the constructed cross-modal phase difference slope fingerprint, the counterfactual playback evidence chain is obtained, and frame-by-frame replay of extreme dynamic segments is performed. During the replay process, the response misalignment trigger node is located, and a complete temporal offset spectrum is generated to provide continuous support for the identification of decoherence intervals. To identify modal response misalignments in abnormal processes and support accurate localization of subsequent decoherence intervals, a counterfactual replay method based on phase difference slope fingerprinting is proposed. This method combines frame-by-frame playback of extreme dynamic segments with temporal error analysis to construct a complete temporal offset spectrum. The method specifically includes the following steps: Based on the constructed cross-modal phase difference slope fingerprint matrix, time periods exhibiting drastic phase difference slope changes are identified as initial candidate regions for extreme dynamic segments. Specifically, the phase difference slope fingerprint matrix is ​​first derived along the time axis to extract the slope change amplitude of each modal pair between adjacent time frames, and the average change value of all modal pairs is calculated. Time segments with an average change value greater than three times the standard deviation of the total sequence change value are marked as significant change segments. Subsequently, within these segments, brightness abrupt changes in image data, energy transition points in acoustic data, extreme points of angular velocity in inertial data, and voltage and current abrupt change nodes in environmental data are compared one by one to confirm whether there are significant misalignments in the peak response times between multiple modes. If, within the same time interval, the time difference between the dominant responses of two or more modes exceeds 30 milliseconds, this time period is identified as an extreme dynamic segment. Based on this, all original data frames for each mode within the corresponding time period are extracted, and timestamps are re-aligned according to a unified event time baseline to construct a high-precision multimodal replay sequence, providing a complete and traceable data trajectory for subsequent analysis.

[0023] In the basic replay sequence, for each extreme dynamic segment, a counterfactual replay process based on the dominant modality reference is implemented to simulate the impact of multiple potential response timing combinations. Using the image modality as the initial reference source, its main response peak time point is identified as the baseline, and a symmetrical time window is constructed, with a default window length of 200 milliseconds before and after it. Within this window, the response peak times of other modalities are artificially perturbed to create multiple time-shifted scenarios. For example, the main transition time point of the acoustic mode is shifted forward by 20 milliseconds, and the angular acceleration change point of the inertial mode is delayed backward by 15 milliseconds, constructing counterfactual states of "acoustic early triggering" and "inertial lag response," respectively. For each counterfactual scenario, the phase difference, response amplitude difference, and synchronicity index between modalities are recalculated, and it is recorded whether the image dominant event still forms a co-intensifying effect with other modalities. This guided temporal perturbation operation can identify which modal combinations are most likely to form anomaly reinforcement under different temporal misalignment configurations, and which combinations will lead to the masking of anomalies, providing a solid behavioral comparison basis for subsequent identification of decoherence mechanisms.

[0024] Based on the response matching results of various modal combinations in the counterfactual playback scenario, key response misalignment triggering nodes causing intermodal interference or cooperative failure are identified and marked. Specifically, local extremum point analysis is performed on the phase difference time curve extracted for each playback scenario to identify the location with the largest response offset. Then, focusing on these time points, the brightness slope change rate of the image modality, the amplitude abrupt change of the acoustic modality, the angular velocity fluctuation value of the inertial modality, and the energy parameter jump value of the environmental modality are extracted to construct a multimodal feature vector for analyzing the behavior patterns of these features at the misalignment triggering nodes. If the feature combination of a node exhibits inconsistent cross-modal abrupt change directions within 10 milliseconds and shows a significant decrease in anomaly detection capability in the counterfactual scenario (e.g., reduced prediction score, reduced signal coverage), then this node is identified as a key response misalignment triggering point causing the decoherence effect. Furthermore, taking this node as the center, the time window is extended by ±100 milliseconds to analyze the behavioral changes of the data before and after the mutation of each mode, and to obtain the changes in response inertia, hysteresis trend and precursor signal characteristics, so as to form a high-resolution, multi-angle misalignment response feature set.

[0025] All misalignment triggering nodes and their extended feature sets are aggregated to construct a temporal migration spectrum covering the entire anomaly evolution process, used to continuously represent the evolution of response synchronicity between modes. The temporal migration spectrum uses a unified event time baseline as the horizontal axis and the time offset between each mode pair as the vertical axis, encoding the response difference at corresponding time points as weights, and introducing color gradients to visualize the migration direction and amplitude. For example, blue represents an acoustic mode lagging behind the image mode by more than 50 milliseconds, and red represents an inertial mode leading the image mode by more than 30 milliseconds. Curve fitting is performed on the entire spectrum to identify time regions with continuous high-amplitude fluctuations, and the trajectory of inconsistent modal responses is extracted centered on these regions. Furthermore, temporal clustering is performed on overlapping high-fluctuation segments in the migration spectrum to form a representative set of decoherent segments. Each set includes multi-dimensional features such as typical response migration patterns, central migration nodes, the degree of participation of the dominant mode, and the fluctuation range of the synchronicity index. This offset spectrum can not only be used to directly identify distortion windows caused by response misalignment in process anomalies, but also provide a high-precision, structured, and dynamically evolving reference for the subsequent construction of causal structures, extraction of trend nodes, and design of phase intervention paths.

[0026] Based on the generated temporal offset spectrum, the phase-locked consistency index and quantile manifold embedding method are introduced to perform multidimensional comparison of multi-source responses, identify potential decoherence intervals, and solidify the temporal coordinates of decoherence anchor points for temporal reference in causal topological inversion. To achieve in-depth identification of potential decoherence behavior in multi-source responses, it is necessary to sequentially introduce phase-locked consistency index analysis and quantile manifold embedding methods based on existing time-series offset spectra. This process completes the identification of decoherence segments, spatial analysis of response trajectories, and solidification of anchor point time-series coordinates, providing accurate time references for causal topology inversion. The specific implementation includes the following steps: In the constructed time-series offset spectrum, phase-locked loop consistency index analysis is performed using each pair of multi-source mode combinations as the basic unit to identify the dynamic evolution trend of the synchronization level between modal responses. Specifically, the offset values ​​of all possible mode combination pairs (image mode and acoustic mode, image mode and inertial mode, image mode and environmental mode, acoustic mode and inertial mode, acoustic mode and environmental mode, inertial mode and environmental mode) at each time point are extracted and organized into a complete set of time difference sequences. Each time difference sequence is sliced ​​using a fixed-length sliding time window, with each window length set to 200 milliseconds and a step size set to 50 milliseconds. Within each time window, the phase difference change rate of the mode pair is calculated, i.e., the increment of the offset value between adjacent time points. Based on this change rate, the fluctuation direction consistency index is calculated, which is the ratio of the number of points with the same offset change direction within the window to the total number of points in the window, defined as the phase-locked loop consistency index. If the index is below a preset threshold (e.g., 0.3), it indicates a high risk of asynchrony between modal responses within that time window. All time windows meeting the low consistency condition are extracted and compared with the time-axis overlap of these windows with the slope abrupt change segments already marked in the time-series offset spectrum. Time periods deemed anomalous in both are retained and marked as high-confidence decoherence candidate segments, providing preliminary screening results for subsequent spatial feature verification operations.

[0027] A quantile manifold embedding operation is performed on high-confidence decoherence candidate regions to deeply analyze the deviation trajectories of modal response features in multidimensional space, thereby identifying structurally inconsistent modal behaviors. This process first extracts the dominant response feature value sequence for each mode within each candidate region: for image modes, the grayscale value change amplitude at the point of maximum inter-frame brightness gradient is extracted; for acoustic modes, the short-time average energy of the waveform is extracted; for inertial modes, the sum of the three-axis angular velocity changes is extracted; and for environmental modes, the first derivative of temperature change and current fluctuation rate are extracted. These four types of modal features are concatenated in chronological order into a multidimensional feature vector sequence, forming a three-dimensional data structure with dimensions of time × mode × feature. Next, this data structure is input into a manifold mapping space constructed based on quantile density. Embedding paths between feature spaces are constructed using the local density sorting of adjacent feature points, and the response trajectory of each mode in the embedding space is extracted. Curvature calculation is performed on all trajectories, and the average curvature and maximum spatial offset of each modal trajectory within the corresponding time segment are statistically analyzed. If the average curvature of a certain mode exceeds 1.8 times the average curvature of all mode trajectories, and its maximum spatial offset exceeds 15% of the trajectory convergence center, then the mode is determined to have a structural response offset within that time period. This offset behavior is typically manifested as a time lag in the response start point, an opposite response trend direction, or a reversed amplitude response path curvature direction, significantly different from the concentrated response paths of other modes. The time points corresponding to the above modal offset behavior are used as abnormal trajectory behavior markers, and their original timestamps, corresponding mode types, and corresponding 3D embedded coordinate values ​​are recorded to assist in subsequent time anchor point extraction.

[0028] After completing the identification of decoherent segments and modeling of structural response shift behavior, the intersection of the abnormal windows and trajectory shift points identified in the first two steps is processed to extract a set of high-confidence decoherent anchor points, and their time-series coordinates are fixed as a time reference for subsequent causal topology inversion. Specifically, the time coordinates of all low consistency index segments and abnormal embedded trajectory shift points are traversed, and window overlap operations are performed. If a time point exists simultaneously in a low phase-locked index segment, and its corresponding modal response shows significant structural shift in the embedding space, then this time point is marked as an anchor point to be fixed. Subsequently, modal response features of this anchor point are integrated, extracting its feature vectors for the corresponding time frames in each mode, and calculating the cross-modal response direction consistency value. If this consistency value is less than a threshold (e.g., 0.5), it indicates that there is directional divergence in the response, and it should be prioritized as a decoherent anchor point. Finally, all time points that meet the conditions are organized into an anchor point sequence in chronological order. Each anchor point includes time coordinates, dominant response mode, response direction feature value, spatial shift description, phase-locked index, and its evolution trend information. This anchor sequence serves as a highly reliable temporal reference framework under the temporal baseline. It is used to deduce the causal chain of response, invert the propagation path of anomalies, and construct a directional causal topology graph structure. This ensures that each causal edge is anchored to a real time point with structural anomaly characteristics, significantly improving the accuracy and reliability of causal relationship modeling.

[0029] Based on the fixed anchor point time series coordinates, a phase traction probe operation is performed to inject a small phase perturbation into the cross-modal response, measure the residual gradient field, invert the causal topology, locate the energy convergence path, and use it to generate the trend node set. To accurately identify the energy propagation mechanism and reliably extract trend nodes in abnormal processes, phase-driven probing operations need to be conducted under a unified time baseline based on the aforementioned fixed anchor point time-series coordinates. This involves three stages: perturbation injection, residual gradient measurement, and topology inversion, to construct a structured path suitable for trend judgment and causal deduction. The process includes the following steps: Based on the established decoherence anchor point time-series coordinates, an independent dynamic perturbation observation window is constructed for each anchor point under a unified event time baseline, and a cross-modal phase-traction probe operation is performed. Specifically, the operation involves extending the time window forward and backward by 200 milliseconds, centered on the anchor point's time position, to form a closed time window with a total length of 400 milliseconds. Within this window, the grayscale gradient change sequence of the central region of the image mode, the short-time energy envelope curve of the acoustic mode, the angular velocity triaxial change curve of the inertial measurement mode, and the current transient transition data of the environmental mode are extracted. These multi-source sequences are then standardized. After standardization, a small perturbation is injected into the main response band of each mode. The perturbation amplitude is set to 5% of the maximum response value of that mode. The perturbation forms include phase forward shift, phase backward shift, amplitude boosting, and amplitude attenuation. The perturbation only affects one mode, while the other modes remain unchanged. Taking image modalities as an example, the grayscale jump point can be advanced by one sampling unit in the frame sequence, or its brightness gradient can be increased by 10%, to simulate an early response or enhanced reaction scenario. This operation is used to detect whether a small change in one modality produces a cascade response to other modalities, thereby inferring the potential causal linkage between modalities.

[0030] After the perturbation is completed, the multimodal response sequences before and after the perturbation are compared time-by-time to calculate residual values ​​and construct a residual gradient field, which is used to characterize the propagation trajectory of the perturbation response between different modes. Specifically, the response sequences of each mode before and after the perturbation are compared at each time point to construct a perturbation response residual sequence. Then, the first derivative of the residual sequence of each mode is calculated to obtain the response change rate at each time point. The rate data is then aggregated by mode to generate a two-dimensional residual gradient map with time as the horizontal axis and mode as the vertical axis. In this map, the residual amplitude change curves of each mode after the perturbation injection point are observed to identify mode combinations with concentrated response peaks and consistent response directions. If, within 50 milliseconds after the perturbation injection, two or more modes show synchronous response increases and have the same residual direction, it indicates the existence of an energy conduction channel between these modes. Furthermore, the points with the maximum residual amplitude are connected along the time axis to construct directed edges from the perturbation source mode to the response mode, with the edge weight being the magnitude of the residual peak value. By summarizing the residual propagation paths of all disturbance sources, a causal topological graph containing nodes (mode-time pairs) and edges (energy influence paths) can be generated. The edges in the graph have clear directions, and the weights accurately reflect the degree of disturbance influence, which is the core basis for subsequent identification of trend nodes.

[0031] Based on the causal topological map, nodes with significant energy convergence behavior are identified and incorporated into the trend node set as the structural core of the anomaly chain. The specific analysis process is as follows: For each node in the map, the number of incoming edges is calculated. Nodes with more than two incoming edges are multi-source input nodes, indicating that they are locations where multiple modes of energy converge. Then, the residual response amplitude, duration of response time, and dominant parameter characteristics of the mode in which these nodes are located are quantitatively evaluated. For example, the brightness change rate of convergence nodes in the image mode is more than three times the average value of the same period; the energy envelope fluctuation value of convergence nodes in the acoustic mode continuously increases for more than 100 milliseconds; the angular velocity variation direction of inertial mode nodes is consistent with the disturbance direction and the change rate exceeds 5 degrees per second; and non-periodic peaks appear more than three times continuously in the energy supply signal of the environmental mode. Nodes that meet the above characteristics are judged as trend nodes with strong response and stable structure. All trend nodes that meet the conditions are organized according to their response mode, time sequence, and influence path number to form a trend node set. This node set has clear time coordinates, modal sources, energy influence directions, and dynamic evolution trajectories, which can be used as input conditions for subsequent dynamic regulation schemes.

[0032] Based on the energy convergence path, an adaptive evidence chain is constructed to cross-validate the energy trajectory, suspected anomaly spectrum and trigger source region, generate a set of credible trend nodes, and quantify the intervention amplitude for dynamic control of input. To achieve accurate verification of key trend nodes in anomaly processes and scientific setting of intervention amplitudes, a complete adaptive evidence chain needs to be constructed based on the aforementioned energy convergence path and causal topology. This chain should gradually integrate energy trajectories, anomaly spectrum response characteristics, and source triggering paths, and complete the quantification of intervention inputs. The specific implementation steps are as follows: Based on the identified energy convergence paths in the constructed causal topology, all response nodes and propagation sequences in the paths are extracted to construct a cross-modal, cross-time-time energy trajectory sequence. This trajectory sequence must contain the actual response information of each mode in its corresponding path and be mapped to a unified event time baseline. In specific operations, for the image mode, the image frame grayscale change value, gradient growth slope, and frame number at each response node are recorded; for the acoustic mode, the maximum value of the continuous sound pressure envelope, the energy distribution within the frequency band, and the trend of the dominant frequency change are extracted; for the inertial mode, the amplitude peaks and duration of the abrupt changes in the angular velocity along the three axes are extracted; for the environmental mode, the maximum value, fluctuation range length, and duration of the anomaly in the voltage response segment are quantified. The above data are used to construct a complete energy trajectory structure in chronological order. Each response node in the structure records the mode type, time position, response amplitude, and logical relationship before and after path propagation. This energy trajectory structure not only supports subsequent response consistency analysis but also serves as a quantitative basis for screening the influence level of trend nodes.

[0033] By combining the anomaly spectrum information extracted from historical anomaly detection results, multi-level matching analysis is performed on the aforementioned energy trajectories to determine whether the response segments in the path are strongly correlated with the actual anomaly modes. In specific implementation, the start time, end time, anomaly response mode, characteristic abrupt change index, and amplitude change rate corresponding to each anomaly segment in the anomaly spectrum are extracted. For example, in the image mode anomaly spectrum, the average grayscale value increases by more than 30 units within 5 frames; in the acoustic mode anomaly segment, the energy density doubles within 1 second; in the inertial mode anomaly, the angular velocity abruptly reverses direction and remains for more than 0.2 seconds; and in the environmental mode, the voltage or temperature change value reaches more than twice the standard deviation of the original curve. These anomaly spectrum segments are compared one by one with the corresponding modal time periods in the energy trajectory. Response segments that simultaneously satisfy similarity in time point, modal type, and response trend are marked as anomaly matching segments. All matching segments are summarized in chronological order to form a high-confidence set of anomaly trajectories. Further determine whether the trend node falls within any response segment in the set. If it appears in two or more abnormal trajectory segments and is the point with the largest response amplitude in that segment, then mark the node as a candidate node for trend confirmation and proceed to the subsequent verification process.

[0034] After completing the cross-validation of the anomaly spectrum, the origin of the initial response of the trend node in the energy path is further traced to determine whether it originates from a common triggering source, and the consistency strength in its causal link is evaluated. Specifically, the energy path to which each trend node belongs is identified, and its starting node time coordinates and modal origin in the topology graph are traced. If multiple trend nodes originate from the same starting time period and have the same modal type (e.g., image modal originates from adjacent frames within the same frame region), a common triggering source is considered possible. Based on this, the path chain from the common source node to the trend node is extracted, and the link propagation time, response amplitude change factor, inter-node phase difference change value, and path energy density distribution are calculated. All indicators are standardized and combined into a trend response consistency index to quantify the consistency degree of the trend node in the response propagation process. If the consistency index exceeds a set threshold (e.g., 0.75), the node is listed as a credible trend node, and all response enhancement behaviors along its path are recorded to provide input for subsequent intervention settings. If a trend node only appears in isolated paths, or exhibits unstable behaviors such as response backpropagation or energy weakening in the propagation path, it is not included in the credible trend node set but is retained as a structural reference point.

[0035] Based on the established set of reliable trend nodes, the intervention amplitude is quantified for each node to provide parameterized control input for subsequent anomaly suppression operations. The intervention amplitude is set based on three aspects: first, response intensity, i.e., the multiple of the response amplitude at the node relative to the average value of the same mode; second, path concentration, i.e., how many propagation paths the node appears at in the topology; and third, response stability, i.e., whether the response direction and amplitude trends remain consistent within consecutive time periods before and after the node. Specific settings include: in the image mode, if the node response intensity reaches 2.5 times the average grayscale change value of the region, the path concentration exceeds 3 paths, and the response direction continues to rise for more than 150 milliseconds, then the intervention amplitude is set to 1.8 times the baseline perturbation level; in the acoustic mode, if the response is concentrated in the high-frequency region and the energy value exceeds the threshold by 3 times, it is set to 2 times the baseline pulse intensity; in the inertial mode, if the node experiences a continuous sudden increase in angular velocity, the angular braking feedback is set to 1.3 times the average deceleration; in the environmental mode, if it is a voltage jump region, the voltage feedback limit is set to 1.5 times the average fluctuation value, and a time lag control factor is added. All intervention amplitudes will form a set of intervention instructions precisely labeled by time, mode, and amplitude, serving as the main control input in the dynamic control system to ensure accurate, efficient, and sustainable suppression of reliable trend nodes.

[0036] Based on a set of reliable trend nodes, a time-reversal phase-gated instruction is generated to drive the injection of a phase conjugate micro-envelope into a programmable polarization metasurface. This establishes a reverse energy channel in the trigger source region, performs a pulse-level quenching operation, and writes the calibration matrix back to a unified event time baseline, thus realizing a closed-loop dynamic control process to achieve rapid suppression and precise handling of anomalies. To achieve dynamic closed-loop control of the entire abnormal process, a set of reliable trend nodes is needed as the control starting point. This involves generating time-reversal phase-gated commands, precisely driving a programmable polarization metasurface, injecting conjugate phase micro-envelopes, constructing an inverse energy channel, and executing pulse-level quenching, ultimately completing the state calibration write-back. The specific implementation steps are as follows: Based on the confirmed set of credible trend nodes, all trend nodes are traversed one by one under a unified event time baseline to extract their response peak time point, modality classification, response direction, and amplitude level. Combined with the energy propagation path structure, possible source trigger time points are deduced in reverse, forming a time-inversion mapping chain. By performing intersection operations on the backpropagation paths of multiple trend nodes, the intervention initiation time window is determined. This time window consists of the intervention start time and intervention duration. Furthermore, the time window is combined with modality correspondence information to construct a time-inversion phase-gated instruction. This instruction contains five elements: intervention start time, intervention duration, target modality type, intervention direction (suppression or weakening), and intervention priority. After the instruction structure is determined, it is projected onto the unified event time baseline for time alignment, forming a time-driven control reference framework, which serves as the starting point for all subsequent physical interventions and responses.

[0037] Based on the target mode type and spatial location parameters defined in the time-reversal phase gating command, the programmable polarization metasurfaces in the corresponding regions are activated and enter the state reconstruction stage. Taking the image mode as an example, if several pre-target regions are located in the dynamic edge region of the lower left quadrant of the image, the polarization unit array in this region needs to adjust its voltage excitation value in real time so that its polarization direction is opposite to the direction of the intervention target, thereby achieving phase control of the image brightness response. For the acoustic mode, if the target is the energy enhancement response of mid-frequency sound waves in a specific propagation path, the microstructure acoustic diaphragm in the corresponding region needs to reconstruct its vibration direction and frequency so that the sound wave undergoes the expected reverse deflection after incident. In the inertial response mode, if the angular velocity response direction needs to be suppressed, the mechanical rotational inertia of the inertial feedback channel needs to be adjusted so that it forms a torque cancellation in a specific phase segment. In the environmental parameter mode, if several pre-targets are temperature gradient reversal regions, the thermally sensitive materials arranged on the microscale thermal control surface excite cooling feedback, causing the heat flux density in this region to reverse direction. All of the above operations are completed synchronously based on the preset time and space coordinates in the gating command, ensuring that a structured phase offset capability is formed in each modal response path.

[0038] After the polarization metasurface completes state modulation, phase-conjugate micro-envelopes are injected into the target response path according to the timing control points defined in the gating instructions to achieve a reverse intervention effect symmetrical to the anomalous response behavior. The injection process uses the response trajectory of the trend node as a mirror reference to construct a perturbation envelope with completely opposite phase, amplitude, and frequency. For the image mode, a brightness decrease trajectory is injected into the pixel-level image, making its gradient change direction consistent with but opposite to the direction of anomalous brightness enhancement. For the acoustic mode, a waveform energy envelope with the same frequency as the original response but with the opposite phase is injected into the time-frequency domain, weakening the original acoustic response within the target frequency band. In the inertial mode, a torque-driven signal opposite to the original angular velocity direction is injected to suppress the original rotation trend and form an anti-rotation dynamic. In the environmental mode, a perturbation sequence with reverse temperature or voltage variations is constructed, and high-frequency, low-amplitude perturbations are injected through the microscale input terminal to form a reverse adjustment. The injection of all micro-envelopes is strictly synchronized with the response time point corresponding to the trend node and continues until the end of the intervention target segment, ensuring a complete phase-conjugate intervention process.

[0039] As the phase-conjugate micro-envelope continues to act on the response path, a structurally closed reverse energy channel is gradually established between the trend node and the trigger source. Channel construction requires three conditions: first, at least three consecutive nodes in the response path exhibit a decreasing trend in response amplitude after intervention, and the phase change direction shifts from positive to negative; second, the residual change rate in the decreasing response amplitude section of the energy propagation path converges to below the initial perturbation level; and third, the energy flow density of the path between the trend node and the source trigger region is less than 20% of the minimum transmission value under abnormal conditions. When these three conditions are met, the reverse energy channel is considered successfully established. Immediately, a pulse-level extinguishing operation is executed, consisting of two parts: first, simultaneously injecting a unit-duration, bipolar short-time pulse into all response nodes in the path to clear residual energy; and second, applying a phase-delayed reverse perturbation to key relay nodes, causing their path connection to break, thereby severing the abnormal feedback loop in the response chain. The entire extinguishing operation must be completed within 50 milliseconds after the maximum response point of the response path to ensure that abnormal behavior is quickly suppressed and the propagation trend is blocked.

[0040] After the reverse energy channel completes its extinguishing operation, the final stage of the abnormal closed-loop control begins. This involves calculating the response matrix differences for each stage before, during, and after the intervention, and then writing the calibration results back to the unified event time baseline. Specifically, this is implemented as follows: The response frame sequence change map of the image modality is extracted; the abnormal peak frames before intervention are compared pixel-level with the stable frames after intervention, and the brightness recovery rate and spatial gradient convergence value are calculated. In the acoustic modality, the spectral energy difference before and after intervention is extracted, and the frequency band suppression ratio and sound pressure stability index are calculated. In the inertial modality, the ratio of angular velocity decrease before and after the maximum change point in the angular velocity time series is extracted. In the environmental modality, the change in the number of peaks and the maximum peak reduction rate of the temperature or voltage time curves are extracted. The response correction data from these four dimensions are summarized to form a complete intervention calibration matrix. The time coordinates, modal types, and response correction magnitudes in this matrix are written into the corresponding positions on the unified event time baseline, and the status is updated to "Intervention Completed." Thus, the entire anomaly response process, from identification and trend node confirmation to intervention execution and state recovery, forms a closed loop. The structure is written back in a time-aligned manner, providing complete time-series data support for subsequent intelligent process optimization.

[0041] This invention constructs a cross-modal phase difference slope fingerprint to accurately characterize the temporal offset and instantaneous drift features between modes. Combined with counterfactual replay, it achieves fine-grained localization of anomalous response paths. Then, it utilizes phase-locked consistency index and quantile manifold embedding to identify decoherence intervals and solidify anchor point coordinates. Furthermore, through phase-driven perturbation and residual gradient inversion, it constructs causal topology and energy convergence paths, ensuring that trend node identification has physical meaning and causal basis. Based on this, a time-reversal control strategy drives the precise injection of conjugate phase perturbations into a programmable polarization metasurface, realizing the construction of reverse energy channels and local extinguishing operations for anomalous response paths. Finally, it completes state calibration and writes back to a unified time baseline, forming a self-closing dynamic control mechanism. Compared to existing static fusion or rule-matching-based detection methods, this scheme not only possesses higher robustness and response accuracy but also can dynamically adapt to sudden anomalous patterns in multi-source heterogeneous data environments.

[0042] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. An anomaly detection method based on multimodal fusion, characterized in that, Includes the following steps: Temporal trajectories of multi-source data are extracted under a unified event time baseline, cross-modal phase difference slope fingerprints are constructed, and instantaneous drift gradients are quantified. Based on the phase difference slope fingerprint, the counterfactual playback evidence chain is obtained, the dynamic segment is replayed frame by frame, the response misalignment trigger node is located, and the temporal offset spectrum is generated. Based on the temporal offset spectrum, phase-locked consistency index and quantile manifold embedding are introduced to identify decoherence intervals and solidify the temporal coordinates of anchor points; Based on the anchor point time series coordinates, a phase traction probe is performed to inject micro-amplitude phase perturbations into the multimodal response, measure the residual gradient field, invert the causal topology, and locate the energy convergence path. Based on the energy convergence path, an adaptive evidence chain is constructed to cross-validate the energy trajectory, anomaly spectrum and trigger source region, generate a set of credible trend nodes, and quantify the intervention amplitude; Based on the trend node set, a time-reversal phase-gated instruction is generated to drive the injection of a phase conjugate micro-envelope into the programmable polarization metasurface, establish a reverse energy channel in the trigger source region, perform a pulse-level quenching operation, and write the calibration matrix back to the event time baseline.

2. The abnormal process detection method based on multimodal fusion according to claim 1, characterized in that, The phase difference slope fingerprint construction process is as follows: A global timeline with a unified time baseline is established, and the data sampling time series of the image acquisition device, acoustic receiving device, inertial measurement unit, environmental sensor and electronic probe are extracted. All data frames are linearly normalized and interpolated to achieve time alignment. Perform dynamic change detection of multi-source response, identify the dominant change nodes of each modality data and complete the annotation on a unified time baseline; Calculate the slope of the phase difference increment between mode pairs and construct a cross-modal phase difference slope fingerprint matrix; Instantaneous drift gradients are quantized in slope fingerprints to generate drift gradient heatmaps. Modal shift behaviors with early or delayed responses are extracted and used as a unified alignment reference structure for subsequent anomaly detection.

3. The abnormal process detection method based on multimodal fusion according to claim 2, characterized in that, The steps for generating the time-series migration spectrum are as follows: Based on the constructed cross-modal phase difference slope fingerprint matrix, time periods with drastic phase difference slope changes are identified as extreme dynamic segments. Extract the original data frames corresponding to extreme dynamic segments and reconstruct the basic replay sequence under a unified event time baseline; Using the image modality as a reference, the dominant response time of other modalities is perturbed within the constructed time window to generate counterfactual scenarios, and response matching metrics are calculated for each scenario. The system identifies response misalignment trigger nodes, constructs multimodal feature vectors, expands the behavior analysis interval, and finally generates a complete temporal offset spectrum for subsequent decoherence interval identification.

4. The abnormal process detection method based on multimodal fusion according to claim 3, characterized in that, The steps for fixing anchor point time coordinates are as follows: Based on the constructed time-series offset spectrum, phase-locked loop consistency index analysis is performed, and time windows with low consistency indices are extracted as preliminary decoherence candidate segments. Based on the decoherent candidate segment, perform quantized manifold embedding operation to extract the trajectory offset behavior of the modal response in the embedding space, and mark the time points when the average curvature and the maximum offset exceed the set threshold. The low-consistency segments and trajectory offset points are intersected, and time points that simultaneously meet both criteria are selected. Their time coordinates are extracted and fixed to form a decoherent anchor point sequence, which is used as a time reference for subsequent causal topological inversion.

5. The abnormal process detection method based on multimodal fusion according to claim 4, characterized in that, The energy convergence path localization process is as follows: Based on the fixed anchor point time series coordinates, a perturbation observation window is constructed on a unified time baseline to inject micro-perturbations into a single mode while keeping the original response states of other modes unchanged. A residual gradient field is constructed for the multimodal response sequences before and after the perturbation. A directed edge is established along the path direction of the maximum residual amplitude, pointing from the perturbation source to the response mode, and a causal topological graph is generated. Nodes with more than a threshold number of incoming edges and concentrated residual distribution are extracted from the topological graph and identified as trend nodes in the energy convergence path. These trend nodes are then aggregated to form a set of trend nodes, which serves as input for subsequent dynamic regulation.

6. The abnormal process detection method based on multimodal fusion according to claim 5, characterized in that, The steps for generating a set of reliable trend nodes are as follows: Energy trajectories are extracted based on causal topology and the modal types, response amplitudes, and propagation sequences are labeled. By combining historical anomaly spectrum information to compare response trends, matching segments are extracted and trend confirmation candidate nodes are screened. Based on the initial response source in the energy path, causal path consistency analysis is performed to generate a trend response consistency index and identify credible trend nodes. The response strength, path concentration, and response stability are calculated sequentially for trusted trend nodes to generate an intervention instruction set, which serves as the input for dynamic regulation.

7. The abnormal process detection method based on multimodal fusion according to claim 6, characterized in that, The intervention amplitude in the intervention instruction set is set based on trend nodes where the response intensity exceeds twice the average response amplitude, the path concentration is higher than the intersection of three propagation paths, and the response direction does not reverse during the duration.

8. The abnormal process detection method based on multimodal fusion according to claim 6, characterized in that, The steps are as follows: Based on the trend node set, time-reversal gating commands are generated to drive the polarization metasurface to inject conjugate phase microenvelopes, construct an inverse energy channel in the trigger source region, implement pulse quenching, and write the calibration results back to the event time baseline. Construct a time-reversal mapping chain and generate phase-gated instructions that include time, mode, orientation, and priority; Activate the programmable polarization metasurface and complete phase modulation in the intervention region; Inject a phase-conjugate micro-envelope that is opposite to the trend node response trajectory; Construct a reverse energy channel between the trend node and the source trigger that satisfies the conditions of phase change, amplitude suppression and energy density decrease, and perform a pulse-level extinguishing operation; Extract the response differences before and after each modality intervention to construct a calibration matrix and write the results back to a unified event time baseline to achieve closed-loop dynamic control.

Images