A multi-modal anomaly detection method and system based on time sequence dislocation analysis

Abstract

The present application belongs to the field of information security and signal processing technology, and particularly relates to a multi-modal anomaly detection method and system based on time sequence misplacement analysis. The method synchronously collects multi-modal time sequence signals and standardizes time stamps, detects or injects time sequence anchor point events. Multi-time scale windows are set, time sequence misplacement feature vectors between modes are calculated based on the anchor points, multi-scale feature sets are aggregated to generate comprehensive consistency scores. Stability evaluation scores are calculated based on historical feature vector sequences, and a time sequence misplacement manifold model is constructed through manifold learning using normal samples. Multi-scale feature vectors are mapped to model embedding space and the distance from the statistical boundary is calculated, multi-scale distances, comprehensive consistency scores and stability scores are fused, and anomalies are determined according to preset rules. The normal time sequence fingerprint of each device and scene combination can be learned, so that inherent normal deviation and abnormal deviation can be distinguished, and content camouflage is not easy to mislead.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of information security and signal processing, and particularly relates to a multi-modal anomaly detection method and system based on time sequence dislocation analysis. BACKGROUND

[0002] With the integration of Internet of Things and artificial intelligence technology, the collaborative processing technology of multi-modal data (such as video, audio, IMU inertial measurement unit, acceleration, and environmental sensor) is widely used in security monitoring, intelligent vehicle system, wearable devices, intelligent manufacturing, and remote medical monitoring. These applications usually rely on the consistency of different modal signals in the time dimension to ensure the reliability and authenticity of the perceived information.

[0003] However, in the anomaly detection and security anti-counterfeiting scenarios, attackers often apply tampering, synthesis, or replay attacks on a single modality (such as video stream or audio stream). Such attacks may maintain the continuity of the single modality content, but inevitably destroy the inherent time consistency between cross-modal signals. Specifically, there are slight asynchronies between speech and lip movements, mismatches between mechanical vibration signals and picture movements, and time sequence dislocation phenomena such as sensor triggering events preceding or lagging behind picture abnormalities. Therefore, the detection of cross-modal time sequence consistency has become a key link to ensure system security.

[0004] Currently, the mainstream cross-modal anomaly detection methods mainly include: (1) simply concatenating or early fusing the feature vectors of different modalities, and then inputting the classifier for anomaly judgment. The focus of this method is on the semantic alignment and fusion at the content level, rather than the synchronization analysis at the time level, and it is not sensitive to minor time sequence dislocations. (2) The synchronization score of speech and lip shape is calculated to determine the authenticity. This method involves time sequence, but is usually limited to audio and video modalities, and is easily misled by content similarity or appearance disguise, lacking effective identification mechanism. (3) Single-modal time sequence consistency verification: for example, by checking the differences between video key frames to determine whether they are tampered with. This method completely ignores the correlation information between cross-modalities, and cannot cope with attacks using time sequence differences between multi-modalities. SUMMARY

[0005] To solve the above problems, the application provides a multi-modal anomaly detection method and system based on time sequence dislocation analysis.

[0006] In a first aspect, the application provides a multi-modal anomaly detection method based on time sequence dislocation analysis, comprising the following steps:

[0007] S1, synchronously collecting multi-modal time sequence signals, and performing timestamp standardization processing on each modality signal;

[0008] S2, detecting natural timing anchor events in the multi-modal timing signals, and / or injecting artificial timing anchor events into the acquisition environment by controlling external devices; the timing anchor events are transient events that produce detectable responses in signals of at least two different modalities;

[0009] S3, setting a plurality of time windows of different time scales; for each time scale, within the corresponding time window, calculating a timing misalignment feature vector between two different modalities of timing signals based on the timing anchor events;

[0010] S4, aggregating the timing misalignment feature vectors obtained at all time scales to form a multi-scale timing misalignment feature set, and generating a comprehensive consistency score using a weighted fusion method;

[0011] S5, calculating a stability evaluation score representing short-term fluctuations and long-term drift trends based on a historical sequence of timing misalignment feature vectors; the historical sequence of timing misalignment feature vectors refers to an ordered set of all timing misalignment feature vectors generated by S3 before the current calculation period;

[0012] S6, using historical multi-modal signal samples collected under normal conditions, constructing a timing misalignment manifold model through a manifold learning algorithm to establish an embedding space and statistical boundary representing the inherent normal timing relationship of the device and the scene;

[0013] S7, inputting each timing misalignment feature vector in the multi-scale timing misalignment feature set into the timing misalignment manifold model, mapping it to a low-dimensional embedding space and calculating its distance from the statistical boundary;

[0014] S8, fusing the distance calculation results at all scales, the comprehensive consistency score and the stability evaluation score, and making a final abnormality determination according to a preset rule.

[0015] As a further limitation of the technical scheme of the present application, S2 specifically includes:

[0016] For each modality of the multi-modal timing signals, a corresponding transient event detection algorithm is run respectively to obtain candidate anchor time stamps for each modality;

[0017] Correlating and matching candidate anchor time stamps of different modalities with a time difference less than a set threshold to confirm that they are the same natural timing anchor event; outputting the confirmed time stamps of the same natural timing anchor event in each modality signal.

[0018] The natural anchor is automatically verified and confirmed through cross-modality correlation and matching, effectively avoiding reference errors caused by single-modality false detection, and providing a highly reliable time reference for subsequent timing analysis.

[0019] As a further limitation of the technical solution of the application, in S2, the injection of artificial time anchor events specifically includes:

[0020] Generate a control instruction, synchronize the injection of the command timestamp, and send the control instruction to an external device to trigger it to generate a transient event in the form of sound, light, or vibration as an artificial time anchor point;

[0021] In the collected multi-modal time sequence signal, the artificial time anchor point is detected, and its actual occurrence timestamp in each modal signal stream is determined; the actual occurrence timestamp, i.e., the artificial time anchor point timestamp, is output.

[0022] The artificial anchor injection mechanism provides an active calibration means, which ensures that in an environment where natural anchor points are scarce and signal quality is poor, the system can still obtain a high-precision time sequence reference, greatly enhancing the applicability and reliability of the method. By comparing the instruction timestamp with the actual occurrence timestamp, the inherent end-to-end delay of the system can be quantified and compensated for, improving the time accuracy of the entire method.

[0023] As a further limitation of the technical solution of the application, in S3, for any two modal time sequence signals X(t) and Y(t), the following steps are performed to calculate the time sequence misalignment feature vector:

[0024] S31, calculate the generalized cross-correlation-phase transform function of X(t) and Y(t), extract the global peak position of the function as the time delay estimation value τ, and calculate the peak value ratio, full width at half maximum, and sidelobe ratio of the peak value;

[0025] S32, calculate the mutual information value of X(t) and Y(t) at different time offset amounts, generate a mutual information-offset curve, and extract the peak position and distribution entropy of the curve;

[0026] S33, calculate the dynamic time warping alignment path between X(t) and Y(t), and obtain the total cumulative cost of the path;

[0027] S34, in a short-time sliding window, calculate the sequence of the time delay estimation value τ, and calculate the variance of the sequence;

[0028] S35, combine all or part of the feature indicators extracted in steps S31 to S34 to form a time sequence misalignment feature vector; the feature indicators include the time delay estimation value τ, the peak value ratio, the full width at half maximum, the sidelobe ratio, the peak position and the distribution entropy, the total cumulative cost, and the variance.

[0029] By fusing multiple features such as cross-correlation, mutual information, and dynamic time warping, a feature vector with extremely rich information is formed, which can comprehensively and multi-angulary describe complex and diverse time sequence misalignment patterns, laying a solid foundation for subsequent determination.

[0030] As a further limitation of the technical solution of the application, the step of S4 comprises:

[0031] For each time scale, input the time series misalignment feature vector in the time window of the scale into a pre-trained scale-specific score model, and output a scale consistency score representing the degree of synchronization consistency of the time scale ;

[0032] Scale consistency scores of all Q time scales are weighted and fused to generate a comprehensive consistency score C ;

[0033]

[0034] Wherein, Q is the number of time scales, is the weight corresponding to the i th time scale, and the weight allocation is negatively related to the length of the time scale, and the shorter the length of the time window of the time scale, the higher the weight .

[0035] As a further limitation of the technical solution of the application, the step of S5 comprises:

[0036] S51, from the historical time series misalignment feature vector sequence, a time delay sequence composed of time delay estimation values τ is extracted in time sequence τ 1, τ 2,..., τm}, wherein m is the length of the historical sequence;

[0037] S52, in a set statistical analysis window, the variance or standard deviation of the time delay sequence is calculated as a short-term fluctuation indicator ;

[0038] S53, in a set trend analysis window, linear fitting is performed on the time delay sequence to obtain the slope of the fitting straight line , which is the long-term drift rate;

[0039] S54, substitute the short-term fluctuation indicator and the long-term drift rate into the following formula to calculate the stability evaluation score S :

[0040]

[0041] In the formula, and are weighting coefficients for adjusting the importance of short-term fluctuation and long-term drift, and ε is a very small positive number to prevent the denominator from being zero.

[0042] By separating short-term fluctuations and long-term drift trends and merging them into a comprehensive score, the stability changes of the system synchronization state can be clearly represented.

[0043] As a further limitation of the technical solution of the application, the step S6 comprises:

[0044] S61, extracting the time sequence misplacement feature vector corresponding to the historical multi-modal signal sample collected in the normal state to form a high-dimensional training data set;

[0045] S62, using a nonlinear manifold learning algorithm to reduce the dimension of the high-dimensional training data set, mapping each high-dimensional time sequence misplacement feature vector to a low-dimensional embedding space, thereby learning the mapping relationship from the high-dimensional feature space to the low-dimensional embedding space, and forming the embedding space of the time sequence misplacement manifold;

[0046] S63, in the low-dimensional embedding space, based on the distribution of all training data points, using an unsupervised boundary learning algorithm to fit a statistical boundary representing the distribution range of normal samples.

[0047] By manifold learning to find the internal distribution structure of high-dimensional time sequence misplacement features in a low-dimensional space, and fitting a statistical boundary, the system can learn the normal standard without pre-defining a fixed threshold, and without training a large number of abnormal samples, thereby achieving efficient, accurate and adaptive anomaly perception.

[0048] As a further limitation of the technical solution of the application, the step S7 comprises:

[0049] S71, inputting each time sequence misplacement feature vector in the multi-scale time sequence misplacement feature set into the time sequence misplacement manifold model in sequence, mapping each high-dimensional feature vector to a low-dimensional embedding space to obtain its corresponding low-dimensional embedding point .

[0050] S72, for each low-dimensional embedding point , using the time sequence misplacement manifold model to calculate the distance between the point and the statistical boundary; the distance is the Euclidean distance; obtaining a scale distance set corresponding to the multi-scale time sequence misplacement feature set , wherein Q is the number of time scales.

[0051] By calculating the distance of the real-time data point to the normal manifold boundary, an intuitive and interpretable abnormality degree quantitative index is obtained. The larger the distance, the higher the abnormality possibility, which provides a clear numerical basis for judgment, rather than just a binary classification result.

[0052] As a further limitation of the technical solution of the application, the step S8 comprises:

[0053] S81, the obtained scale distance set Overall consistency score C and stability evaluation score S ;

[0054] S82. Compare the acquired information with a preset threshold, and make a final determination based on the following rules:

[0055] If the overall consistency score C Below the scoring threshold or stability evaluation score S Below the evaluation threshold or scale distance set In, there is any distance Exceeding its corresponding scale threshold If so, it is considered abnormal;

[0056] If the overall consistency score C Greater than or equal to the scoring threshold And the stability evaluation score S Greater than or equal to the evaluation threshold and scale distance set In the middle, all distances All are lower than or equal to their corresponding scale thresholds If so, it is considered normal;

[0057] S83. Output the final anomaly determination result; if the anomaly is determined, output one or more rule identifiers that triggered the determination; the rule identifiers are used to indicate the specific cause category that led to the anomaly determination, including: comprehensive consistency violation, stability violation or scale distance violation.

[0058] By combining multi-scale distance, consistency score, and stability score for fusion judgment, all available information at the instantaneous, short-term, and long-term levels is integrated, significantly improving the confidence and robustness of the decision. Simultaneously, the output rule identifier enables preliminary root cause analysis, indicating the specific direction of anomaly occurrence and providing clues for subsequent investigation and handling.

[0059] Secondly, the present invention also provides a multimodal anomaly detection system based on temporal misalignment analysis, comprising:

[0060] The signal acquisition module is used to synchronously acquire multimodal time-series signals and perform timestamp standardization processing on each modal signal;

[0061] The timing anchor detection and injection module is used to detect natural timing anchor events in the multimodal timing signals and / or inject artificial timing anchor events into the acquisition environment by controlling external devices; the timing anchor event is an instantaneous event that generates a detectable response in at least two different modal signals;

[0062] The timing misalignment feature calculation module is used to set multiple time windows with different time scales; for each time scale, within the corresponding time window, the timing misalignment feature vector between pairs of different modal timing signals is calculated based on the timing anchor event;

[0063] The multi-scale feature aggregation module is used to aggregate the temporal misalignment feature vectors obtained at all time scales to form a multi-scale temporal misalignment feature set, and to generate a comprehensive consistency score by using a weighted fusion method.

[0064] The stability evaluation module is used to calculate the stability evaluation score representing short-term fluctuations and long-term drift trends based on the historical time-series misalignment feature vector sequence; the historical time-series misalignment feature vector sequence refers to the ordered set of all time-series misalignment feature vectors generated by the time-series misalignment feature calculation module before the current calculation period;

[0065] The temporal misalignment manifold model construction module is used to construct a temporal misalignment manifold model by using historical multimodal signal samples collected under normal conditions and manifold learning algorithms, so as to establish an embedding space and statistical boundary representing the inherent normal temporal relationship between the device and the scene;

[0066] The anomaly detection module is used to input each temporal misalignment feature vector in the multi-scale temporal misalignment feature set into the temporal misalignment manifold model, map it to a low-dimensional embedding space and calculate its distance to the statistical boundary; integrate the distance calculation results at all scales, the comprehensive consistency score and the stability evaluation score, and make the final anomaly detection according to the preset rules.

[0067] As can be seen from the above technical solutions, this application has the following advantages: it systematically and multi-scale extracts and analyzes cross-modal latency, jitter, and drift as core features, overcoming the shortcomings of focusing on content fusion while neglecting temporal consistency. Through multi-scale analysis and stability evaluation, this method has high sensitivity to millisecond-level micro-misalignments, slow long-term drift, and abnormally low jitter (over-smoothing), and can detect subtle anomalies that traditional methods miss. Through a temporal misalignment manifold model, this method can learn the normal temporal fingerprint of each device and scene combination, thereby distinguishing between inherent normal offsets and abnormal offsets, and is less susceptible to being misled by content spoofing.

[0068] The method relies on timing features rather than semantic content features, can run on encrypted or blurred data, protects privacy, reduces dependence on complex computing models, and is easier to deploy on the edge. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0070] Figure 1 The flowchart of the method provided by the embodiment of the present application is shown.

[0071] Figure 2 The block diagram of the system provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0072] In order to make the application purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions protected by the present application will be described clearly and completely by using specific embodiments and drawings. Obviously, the following described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0073] Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application.

[0074] As shown in Figure 1 The embodiment of the present application provides a multi-modal anomaly detection method based on timing misplacement analysis, which comprises the following steps:

[0075] S1, synchronously collecting multi-modal timing signals, and performing timestamp standardization processing on each modal signal;

[0076] Synchronization of multi-modal time-series signals is achieved by starting all sensors simultaneously through a hardware trigger signal. Specifically, all sensors are started simultaneously to begin acquisition by a unified hardware trigger signal (such as a pulse generated by an FPGA) to achieve precise synchronization at the hardware level. Alternatively, at the software level, the acquisition of each sensor is started as close to simultaneously as possible by calling the driving API of each sensor through the system clock. Each sensor or acquisition thread immediately timestamps each data packet (such as a video frame, an audio frame, or a sensor data packet) with a timestamp from a local clock source as soon as the data packet is generated. For example, a camera timestamps each frame of image with its internal clock time t_cam, and a microphone timestamps each audio block with t_mic.

[0077] The embodiment of the present application needs to convert the local timestamps of all modalities into the same reference time coordinate system. The timestamp standardization process for each modality signal is as follows:

[0078] The offset between the clock of each sensor and the reference time reference (such as the system host clock or the clock of a designated main sensor) is calculated, and the local timestamps of each modality are converted into unified standard timestamps according to the offset. The offset is calculated through the Precision Time Protocol (PTP) or the Network Time Protocol (NTP).

[0079] The embodiment of the present application also includes a timestamp alignment sub-step:

[0080] For each modality signal with different sampling rates, a resampling or interpolation algorithm is used to unify them to the same time axis for subsequent calculation of cross-modality time-series misalignment features.

[0081] S2, detecting natural time-series anchor point events in the multi-modal time-series signals, and / or injecting artificial time-series anchor point events into the acquisition environment by controlling external equipment; the time-series anchor point event is a transient event that produces a detectable response in at least two different modalities and has a duration of no more than 100 ms;

[0082] S3, setting a plurality of time windows of different time scales; for each time scale, calculating a time-series misalignment feature vector between two different modalities within the corresponding time window based on the time-series anchor point event;

[0083] S4, aggregating the time-series misalignment feature vectors obtained at all time scales to form a multi-scale time-series misalignment feature set, and generating a comprehensive consistency score using a weighted fusion method;

[0084] S5, calculating a stability evaluation score representing short-term fluctuations and long-term drift trends based on a historical sequence of time-series misalignment feature vectors; the historical sequence of time-series misalignment feature vectors refers to an ordered set of all time-series misalignment feature vectors generated by S3 before the current calculation period.

[0085] S6, constructing a time series dislocation manifold model by a manifold learning algorithm using historical multi-modal signal samples collected in a normal state, to establish an embedding space and a statistical boundary representing inherent normal time series relationship of the device and the scene;

[0086] S7, inputting each time series dislocation feature vector in the multi-scale time series dislocation feature set into the time series dislocation manifold model, mapping it to a low-dimensional embedding space and calculating its distance to the statistical boundary;

[0087] S8, fusing distance calculation results at all scales, comprehensive consistency scores and stability evaluation scores, and making a final abnormality determination according to a preset rule.

[0088] In some embodiments, the detecting natural time series anchor point events in S2 specifically comprises:

[0089] respectively running a corresponding instantaneous event detection algorithm on each of the multi-modal time series signals to obtain candidate anchor point timestamps of each modality;

[0090] associating and matching candidate anchor point timestamps of different modalities with a time difference less than a set threshold to confirm that they are the same natural time series anchor point event; and outputting the confirmed timestamps of the same natural time series anchor point event in each modality signal.

[0091] The detecting natural time series anchor point events specifically comprises:

[0092] If the multi-modal time series signal contains first and second modality signals, running a first detection algorithm on the first modality signal to obtain first candidate timestamps, running a second detection algorithm on the second modality signal to obtain second candidate timestamps, and associating and matching the first candidate timestamps and the second candidate timestamps with a time difference less than a threshold.

[0093] When the multi-modal time series signal contains a third modality signal, the associating and matching is associating and matching candidate anchor point timestamps in the first, second and third modalities with a time difference less than a set threshold.

[0094] The complete flow and output of the natural anchor point detection path should be:

[0095] Input: original signals of each modality.

[0096] Processing: independently running detection algorithms on each modality to obtain a list of candidate anchor point timestamps of each modality (List<t_candidate_video>, List<t_candidate_audio>,...).

[0097] Correlation matching: traverse the lists to find candidate points that satisfy |t_candidate_i-t_candidate_j|<Δt_threshold.

[0098] Match the candidate points that satisfy the condition into a group that originates from the same natural time-series anchor event.

[0099] Output: for each matched group, output the specific time-stamp for each modality within the group.

[0100] For example: given a matched clapping event, output [t_hand_close_video, t_clap_sound_audio]. If the IMU also detected a vibration, output [t_hand_close_video, t_clap_sound_audio, t_vibration_imu].

[0101] The output time-stamps are the same as the time-stamps output by the artificial anchor path, and are fed into S3 for time-delay computation.

[0102] Injecting artificial time-series anchor events specifically includes:

[0103] Generating a control command, recording an injection command time-stamp, and sending the control command to an external device to trigger a transient event in the form of sound, light, or vibration as an artificial time-series anchor;

[0104] Detecting the artificial time-series anchor in the collected multi-modal time-series signals, and determining its actual occurrence time-stamp in each modality signal stream; outputting the actual occurrence time-stamp as the artificial time-series anchor time-stamp.

[0105] Computing the delay difference (Δt=t_actual-t_command) between the actual occurrence time-stamp and the injection command time-stamp;

[0106] In subsequent time-series misalignment computation, the delay difference is compensated as a system inherent delay.

[0107] The external device includes an LED light, a buzzer, or a vibration motor. The transient event includes a flash, a short beep, or a vibration pulse.

[0108] At the same time of injecting an artificial time-series anchor event, the system records an injection command time-stamp, and detects the actual occurrence time-stamp of the artificial event in the subsequently collected signals, and the difference between the two is used for system delay calibration.

[0109] In some embodiments, for any two modal time-series signals X(t) and Y(t) in S3, the following steps are performed to compute the time-series misalignment feature vector:

[0110] S31, calculate the generalized cross-correlation-phase transform function of X(t) and Y(t), extract the global peak position of the function as the time delay estimation value τ, and calculate the peak ratio, full width at half maximum (FWHM), and sidelobe ratio of the peak;

[0111] Pre-emphasize and frame window the signals X(t) and Y(t).

[0112] Perform short-time Fourier transform (STFT) on X(t) and Y(t) to obtain frequency domain representations X(f) and Y(f).

[0113] Calculate the cross-power spectrum: (conj represents the conjugate complex number).

[0114] Calculate the PHAT weighting function: Φ(f) = 1 / |G_xy(f)|.

[0115] Weighted cross-power spectrum: This step is equivalent to only retaining the phase information and being insensitive to the amplitude, which improves the robustness in real environment. Calculate the generalized cross-correlation function:

[0116] Perform inverse Fourier transform (IFFT) on G_xy_phat(f) to obtain the time-domain generalized cross-correlation function R_xy_phat(τ).

[0117] Find the time delay τ corresponding to the global maximum value of R_xy_phat(τ): τ = argmax(R_xy_phat(τ)).

[0118] Find the global maximum value Max1 and the second maximum value Max2, and calculate the peak ratio PeakRatio = Max1 / Max2. The higher the ratio, the higher the confidence of the delay estimation.

[0119] Find the distance between the two points where the horizontal line at half the peak height intersects the peak curve to obtain the full width at half maximum (FWHM). The narrower the width, the more accurate the delay estimation.

[0120] Calculate the ratio of the main peak energy to the average energy of all sidelobes (other peaks except the main peak) to obtain the sidelobe ratio. A low sidelobe ratio indicates the presence of multipath interference or noise.

[0121] S32, calculate the mutual information value of X(t) and Y(t) at different time offsets, generate a mutual information-offset curve, and extract the peak position and distribution entropy of the curve;

[0122] Discretize the continuous signals X(t) and Y(t) into finite amplitude bins, i.e. into multiple bins, to facilitate the calculation of probability distributions.

[0123] For a range of time offsets d, calculate:

[0124] The histogram of signal X(t), i.e. the marginal distribution P(X).

[0125] The histogram of signal Y(t+d), i.e. the marginal distribution P(Y).

[0126] The joint histogram of X(t) and Y(t+d), i.e. the joint distribution P(X,Y).

[0127] For each offset d, calculate the mutual information:

[0128] .

[0129] Traverse the range of d values to obtain the mutual information-offset curve MI(d).

[0130] Find the offset d_{max} that corresponds to the global maximum of the MI(d) curve to obtain the peak position. This provides another estimate of the time delay.

[0131] Calculate the entropy of the MI(d) curve, i.e. the distribution entropy: . A high entropy value corresponds to a flat curve and poor synchronization, while a low entropy value corresponds to a sharp curve and good synchronization.

[0132] S33、Calculate the dynamic time warping alignment path between X(t) and Y(t), and obtain the total cumulative cost of the path;

[0133] Construct a cumulative cost matrix:

[0134] Assume that the length of sequence X is M and the length of sequence Y is N.

[0135] Calculate a local distance matrix of size M N. In this application, the Euclidean distance is used.

[0136] Construct a cumulative cost matrix D of size M N, where the element D(i,j) in the cost matrix D represents the minimum cumulative path cost from (1,1) to (i,j).

[0137] Use a dynamic programming algorithm (such as the Bellman equation) to backtrack from D(M,N) to find an alignment path that minimizes the total cumulative cost, i.e. the optimal path.

[0138] Directly take the value of D(M,N) to obtain the total cumulative cost. The larger this value, the greater the overall shape difference between the two sequences, the more stretching or compression operations required for synchronization, and the poorer the synchronization.

[0139] S34, calculate a sequence of jitter estimation values τ within the short-time sliding window, and calculate the variance of the sequence;

[0140] S31 is performed continuously for a set time period (e.g. the past 5 seconds), resulting in a series of jitter estimation values τ1, τ2,..., τn, i.e. a jitter sequence;

[0141] A short-time statistical window is defined (e.g. containing the most recent 20 τ values).

[0142] The variance of the jitter sequence within the short-time statistical window is calculated: Jitter Variance = Σ(τi-μ)^2 / (K-1), where μ is the mean of the jitter within the window, and K is the window size. The variance is the final feature. The larger the variance, the more jitter, the less stable the synchronization.

[0143] S35, combine all or part of the feature indicators extracted in steps S31 to S34 to form a time sequence misalignment feature vector; the feature indicators include the jitter estimation value τ, the peak ratio, the full width at half maximum, the side lobe ratio, the peak position and distribution entropy, the total cumulative cost, and the variance.

[0144] In some embodiments, the step S4 comprises:

[0145] For each time scale, input the time sequence misalignment feature vector within the time window at the scale into a pre-trained scale-specific score model, and output a scale consistency score representing the degree of synchronization consistency at the time scale .

[0146] Only normal data is used to train a model for each scale, and the distribution of normal misalignment features at the scale is learned. The model is a type of SVM, and for a time sequence misalignment feature vector at a scale, the model outputs a probability or distance that it belongs to the normal distribution. This probability or distance, after conversion, can be used as the consistency score of scale i .

[0147] The scale consistency scores of all Q time scales are fused by weighting to generate a comprehensive consistency score C ; wherein the weights are inversely related to the length of the time scale, and the shorter the length of the time window, the higher the weight of the time scale. Q is the number of time scales, is the weight corresponding to the i-th time scale. i

[0148] In some embodiments, the step S5 comprises: ​

[0149] S51, extracting a delay sequence composed of delay estimation values τ from the historical time sequence misalignment feature vector sequence in chronological order; τ 1, τ 2,..., τm} where m is the length of the historical sequence;

[0150] S52, calculating the variance or standard deviation of the delay sequence as a short-term fluctuation indicator within a set statistical analysis window ;

[0151] S53, performing linear fitting on the delay sequence within a set trend analysis window to obtain the slope of the fitted straight line , which is the long-term drift rate;

[0152] S54, substituting the short-term fluctuation indicator and the long-term drift rate into the following formula to calculate the stability evaluation score S :

[0153]

[0154] wherein and are weighting coefficients for adjusting the importance of short-term fluctuations and long-term drifts, and ε is a very small positive number to prevent the denominator from being zero. ε = 1e-9.

[0155] In some embodiments, the step S6 comprises:

[0156] S61, extracting time sequence misalignment feature vectors corresponding to historical multi-modal signal samples collected under normal conditions to form a high-dimensional training data set;

[0157] S62, using a nonlinear manifold learning algorithm to reduce the dimension of the high-dimensional training data set, mapping each high-dimensional time sequence misalignment feature vector to a low-dimensional embedding space, thereby learning the mapping relationship from the high-dimensional feature space to the low-dimensional embedding space, forming an embedding space of the time sequence misalignment manifold;

[0158] S63, in the low-dimensional embedding space, based on the distribution of all training data points, using an unsupervised boundary learning algorithm to fit a statistical boundary representing the distribution range of normal samples.

[0159] The nonlinear manifold learning algorithm includes UMAP (Uniform Manifold Approximation and Projection) or isometric mapping, and the unsupervised boundary learning algorithm includes a class of support vector machines or Gaussian mixture model GMM.

[0160] In some embodiments, the step S7 comprises:

[0161] S71, inputting each of the time warp feature vectors in the multi-scale time warp feature set into the time warp manifold model in sequence, mapping each of the high-dimensional feature vectors into a low-dimensional embedding space to obtain a corresponding low-dimensional embedding point ;

[0162] S72, for each of the low-dimensional embedding points , calculating the distance between the point and the statistical boundary using the time warp manifold model ; the distance is the Euclidean distance; obtaining a scale distance set corresponding to the multi-scale time warp feature set , wherein Q is the number of time scales.

[0163] In some embodiments, the step S8 comprises:

[0164] S81, obtaining the scale distance set , the consistency score C , and the stability evaluation score S ;

[0165] S82, comparing the obtained information with the preset threshold value, and making a final determination according to the following rules:

[0166] if the consistency score C is lower than the score threshold value , or the stability evaluation score S is lower than the evaluation threshold value , or any of the distances in the scale distance set exceeds the corresponding scale threshold value , it is determined to be abnormal;

[0167] if the consistency score C is greater than or equal to the score threshold value , and the stability evaluation score S is greater than or equal to the evaluation threshold value , and all distances in the scale distance set are lower than or equal to the corresponding scale threshold value , it is determined to be normal;

[0168] S83, outputting the final abnormality determination result; if it is determined to be abnormal, outputting one or more rule identifiers triggering the determination simultaneously; the rule identifier is used to indicate the specific reason category leading to the abnormality determination, including: consistency violation, stability violation, or scale distance violation. ​​

[0169] The following will be described in detail with a multi-modal time series signal as an example. It should be noted that the multi-modal described in the present application is not limited to the signal form, and can also be applied to other multi-source data with time sequence relationship.

[0170] This embodiment takes the detection of abnormal behavior and signal tampering in the security monitoring scene as the background, and elaborates the complete execution process of the abnormal detection method based on time sequence dislocation analysis for three types of multi-modal time series signals: video (V(t)), audio (A(t)), and IMU inertial measurement signal (S(t) containing acceleration and angular velocity data).

[0171] I. Multi-modal signal acquisition and timestamp standardization

[0172] 1.1 Signal acquisition scheme

[0173] A synchronous acquisition strategy of hardware triggering and software completion is adopted to adapt to the real-time requirements of the security monitoring scene:

[0174] A network camera with a hardware synchronous trigger interface (supporting 1080P / 30fps video output), an industrial-grade microphone (48kHz sampling rate, single-channel), and a 6-axis IMU sensor (sampling rate 200Hz, range ±16g acceleration, ±2000° / s angular velocity) are selected. The three are started by a 1ms pulse signal output by the FPGA module to avoid initial asynchronization caused by software call delay.

[0175] The video is stored in H.264 encoding, and each frame carries the camera internal clock stamp tv (precision 1μs); the audio is stored in PCM format, and each frame (2048 samples) carries the microphone clock stamp ta (precision 1μs); the IMU data is output every 5ms, and each packet contains 6-axis data and IMU clock stamp ts (precision 1μs).

[0176] 1.2 Timestamp standardization processing

[0177] The following steps are taken to unify the time base of the three types of signals:

[0178] The system clock of the FPGA (100MHz, precision 10ns) is used as the global reference clock, and the clock offset of the three types of devices is calculated through the Precision Time Protocol (PTP):

[0179] Camera clock offset: Δtv = reference clock time - camera local clock tv;

[0180] Microphone clock offset: Δta = reference clock time - microphone local clock ta;

[0181] IMU clock offset: Ats = reference clock time - IMU local clock ts.

[0182] Convert original timestamps to global reference time, formula as follows:

[0183] Normalized video frame timestamp: Tv = tv + Atv;

[0184] Normalized audio frame timestamp: Ta = ta + Ata;

[0185] Normalized IMU data packet timestamp: Ts = ts + Ats.

[0186] Due to the difference in sampling rate of the three types of signals (video 30fps ≈ 33ms / frame, audio 48kHz ≈ 21μs / sample point, IMU 200Hz = 5ms / packet), linear interpolation and resampling are used to align the audio and IMU data to the 33ms time axis of the video;

[0187] Resample the audio sample points in each 33ms window to generate a 33ms audio segment, which matches the corresponding video frame;

[0188] Mean filter the 6 IMU data packets (5ms / packet) in each 33ms window to generate a 33ms IMU feature value, ensuring consistency with the video frame time granularity.

[0189] II. Time sequence anchor detection and injection

[0190] The time sequence anchor is the time reference for subsequent calculation of cross-modal time delay. In this embodiment, a mixed strategy of natural anchor and artificial anchor is adopted, and anchor detection schemes are designed for video, audio and IMU signals:

[0191] 2.1 Natural time sequence anchor detection

[0192] Natural anchor is a transient event that triggers the response of two or more modalities in the scene at the same time. The specific detection process is as follows:

[0193] Extraction of candidate anchor points for each modality:

[0194] Video modality V(t): Use optical flow mutation and edge detection algorithm to detect hand transient motion (such as waving hands, making fists), object falling and other events: calculate the optical flow field for 3 consecutive video frames, when the modulus of the optical flow vector exceeds the threshold (such as 20 pixels / frame) and the edge contour change rate > 30%, record the frame timestamp T_v_candidate as the candidate anchor point;

[0195] Audio modality A(t): Using short-time energy and zero-crossing rate algorithm to detect transient sounds such as clapping and shouting: Calculate the short-time energy of the audio frame (window length 256 sample points), when the energy value exceeds 3 times the average and the zero-crossing rate is > 100 times / s, record the audio frame timestamp Ta_candidate as the candidate anchor point;

[0196] IMU modality S(t): Using acceleration peak detection algorithm to detect acceleration mutations caused by human body actions (such as walking and turning): When the acceleration value of any axis exceeds ±2g and the duration is <100ms, record the IMU data packet timestamp Ts_candidate as the candidate anchor point.

[0197] Anchor point association matching: Set the time difference threshold ΔT=50ms (covering the inherent delay of the device), traverse the candidate anchor point list of the three modalities, if there is an anchor point group that satisfies |Tv_candidate-Ta_candidate|<ΔT and |Tv_candidate-Ts_candidate|<ΔT, then confirm that the group is a natural time sequence anchor point, output its standardized timestamp group (Tv_anchor, Ta_anchor, Ts_anchor). For example: a clapping event is detected, and the anchor point group (1620000123456μs, 1620000123458μs, 1620000123460μs) is output, representing the trigger time of the event in video, audio, and IMU.

[0198] 2.2 Artificial time sequence anchor injection

[0199] When the natural anchor points in the scene are sparse (such as an empty monitoring area), controllable anchors can be injected through external settings, and the specific process is as follows:

[0200] Injection device and event design: Deploy LED lights (light signals) and micro vibration motors (vibration signals) in the monitoring area, LED lights can produce a 100ms white flash, and vibration motors can produce a 50ms vibration pulse (frequency 200Hz), both of which are triggered synchronously through system instructions;

[0201] System generates artificial anchor injection instructions, records instruction sending timestamp T_command (global reference time);

[0202] The LED light produces a flash after receiving the instruction, which is captured by the camera, and the standardized timestamp Tv_artificial of the flash frame is detected;

[0203] The vibration motor produces vibration after receiving the instruction, which is captured by the Z-axis acceleration sensor of the IMU, and the standardized timestamp Ts_artificial corresponding to the vibration peak is detected;

[0204] If the audio modality needs to be associated, the buzzer (1 kHz, 50 ms) can be triggered synchronously to detect the audio peak timestamp Ta_artificial;

[0205] System inherent delay:

[0206] Δtv_art=Tv_artificial-T_command, Δts_art=Ts_artificial-T_command, which needs to be deducted in subsequent delay calculation to eliminate the system error of artificial anchor injection.

[0207] Three, multi-scale time sequence misalignment feature vector calculation

[0208] Four time scales (32 ms, 128 ms, 1 s, 5 s) are set, and under each scale, the time sequence anchor point is taken as the reference to calculate the misalignment features of three pairs of modalities of video-audio V-A, video-IMU V-S, and audio-IMU A-S, and finally a feature vector with a dimension of 28 is formed (9 features for each pair of modalities, 27 for three pairs, and another 1 scale identifier). The following takes the 128 ms time scale as an example to explain the feature calculation process in detail:

[0209] 3.1 Video-audio modality misalignment feature calculation

[0210] 1. GCC-PHAT delay estimation:

[0211] Take the video frames (4 frames, 30 fps) and audio segments (48 kHz x 0.128 s = 6144 sampling points) within the 128 ms time window, pre-emphasize (high-pass filter, cutoff frequency 80 Hz) and Hanning window the audio;

[0212] Do short-time Fourier transform (STFT, window length 512, overlap rate 50%) on the audio segment to get the frequency domain representation A(f); extract the gray value sequence (take the interested area such as human contour, calculate the average gray value per frame) of the video frame as the video time domain signal V(t), and also do STFT to get V(f);

[0213] Calculate the cross-power spectrum, PHAT weighting function, and weighted cross-power spectrum;

[0214] Do inverse Fourier transform (IFFT) on the weighted cross-power spectrum to get the time-domain generalized cross-correlation function R_xy_phat(τ), and find the time delay τ corresponding to the global maximum value of R_xy_phat(τ): τ = argmax(R_xy_phat(τ)). (unit: ms), τ is the time delay estimation value of V-A modality.

[0215] Build two 4-length sequences with V(t) (4 gray value data points in 128ms window) and A(t) (re-sample 6144 sampling points to 4 data points);

[0216] Calculate Euclidean distance matrix, build cumulative cost matrix Dva with dynamic programming algorithm;

[0217] Extract the last element Dva(M, N) (M=4, N=4) of cumulative cost matrix as DTW cost Cost_va, cost > 100 indicates significant time sequence misalignment of V-A modal.

[0218] Slide 5 32ms sub-windows in 128ms window, calculate τ value in each sub-window, get time delay sequence {τ_va1, τ_va2, τ_va3, τ_va4, τ_va5};

[0219] Calculate the variance of the sequence Jitter Variance = Σ(τ_vai-μ)^2 / (5-1) (μ is the mean of the sequence), variance > 10ms² indicates abnormal jitter of V-A modal.

[0220] 3.2 Video-IMU and audio-IMU modal feature calculation

[0221] V-S modal: video signal takes the motion vector of human body contour (such as x-axis displacement), IMU signal takes x-axis acceleration data, repeat the GCC-PHAT, mutual information, DTW, jitter calculation steps in 3.1, get τ_vs, PeakRatio_vs, Entropy_vs, Cost_vs, Jitter_vs, etc.

[0222] A-S modal: audio signal takes short-time energy sequence, IMU signal takes z-axis acceleration data (corresponding to chest vibration when human body speaks), similarly get τ_as, PeakRatio_as, Entropy_as, Cost_as, Jitter_as, etc.

[0223] 3.3 Feature vector construction

[0224] V-A, V-S, A-S three pairs of modal in 128ms time scale extract 27 features in total, add scale identifier (128ms corresponds to identifier "2"), form feature vector F128=[τ, PeakRatio,..., Jitter_as, 2], similarly generate 32ms, 1s, 5s scale feature vectors F_32, F_1, F_5.

[0225] Four, multi-scale feature aggregation and comprehensive consistency score

[0226] 4.1 Scale-specific score model training

[0227] Training data collection: Under normal monitoring scenarios without abnormalities and tampering, 24 hours of video, audio, and IMU data are continuously collected, and 100,000 normal feature vectors are generated at scales of 32 ms, 128 ms, 1 s, and 5 s to form a training set.

[0228] Model selection and training: A type of SVM is used as a specificity score model for each scale, with a radial basis function as the kernel function, and the training target is to learn the distribution boundary of normal features.

[0229] 32 ms scale model M_32: Focuses on capturing millisecond-level microscopic errors, with a penalty parameter CC=10 and a kernel function parameter γ=0.1 during training.

[0230] 128 ms scale model M_128: Balances microscopic and macroscopic errors, with CC=5 and γ=0.2.

[0231] 1 s scale model M_1: Focuses on second-level mesoscopic errors, with CC=3 and γ=0.5.

[0232] 5 s scale model M_5: Focuses on long-term macroscopic errors, with CC=1 and γ=1.0.

[0233] The model output is the distance D_scale of the feature vector to the normal boundary, which is mapped to the scale consistency score in the [0, 1] interval through the Sigmoid function. The closer to 1, the more consistent the time sequence is at this scale.

[0234] 4.2 Weighted fusion calculation of comprehensive consistency score

[0235] Set the weights of the four scales:

[0236] 32 ms scale weight w_32=0.4 (millisecond-level microscopic errors are sensitive to abnormalities, with the highest weight);

[0237] 128 ms scale weight w_128=0.3;

[0238] 1 s scale weight w_1=0.2;

[0239] 5 s scale weight w_5=0.1 (long-term scale is easily disturbed by the environment, with the lowest weight);

[0240] The comprehensive consistency score formula is C = w_32*S_32 + w_128*S_128 + w_1*S_1 + w_5*S_5, wherein S_32, S_128, S_1 and S_5 are specificity scores of four scales, respectively. For example, S_32 = 0.8, S_128 = 0.7, S_1 = 0.6 and S_5 = 0.9 at a certain moment, then C = 0.4*0.8 + 0.3*0.7 + 0.2*0.6 + 0.1*0.9 = 0.74.

[0241] Five, stability evaluation score calculation

[0242] The stability evaluation aims to quantify the short-term fluctuations and long-term drifts of the time sequence dislocation, and avoid the contingency of the single moment score. The specific calculation is as follows:

[0243] 5.1 Extraction of historical time sequence dislocation feature sequence

[0244] From the historical data before the current calculation period (such as t = 100s), the time sequence of time delay estimation value τ under the scale of 32ms is extracted in time sequence:

[0245] Extraction object: τ_va sequence of V-A mode (since V-A mode is most sensitive to tampering);

[0246] Sequence length: take the τ_va values of the last 100 32ms windows to form the historical sequence {τ_va1, τ_va2,..., τ_va100} (covering 3.2s of historical data, balancing real-time performance and statistical reliability).

[0247] 5.2 Calculation of short-term fluctuations and long-term drifts

[0248] Short-term fluctuation index : Set the statistical analysis window to 20 data points (covering 0.64s), and calculate the standard deviation of the τ_va sequence in the window: =√[Σ(τ_vai-μ_short)² / (20-1)], wherein μ_short is the mean value in the window;

[0249] Long-term drift rate : Set the trend analysis window to 100 data points (covering 3.2s), and do linear fitting y = at + b (t is the time index, 1~100) for the τ_va sequence, and the fitting slope is the long-term drift rate (unit: ms / window), The absolute value of >0.5ms / window indicates that there is obvious drift.

[0250] 5.3 Stability evaluation score S calculation

[0251] ​The formula of weighted fusion of short-term and long-term indicators is set, the weighted coefficients are α=0.6 (short-term fluctuation weight is higher), β=0.4, and the minimum positive number ε=1e-9 (to avoid zero denominator):

[0252]

[0253] S is normalized to the interval [0, 1].

[0254] Six, time sequence dislocation manifold (TMM) model construction

[0255] The device-scene inherent time delay distribution fingerprint is constructed by using normal state data, and the specific process is as follows:

[0256] 6.1 Construction of high-dimensional training data set

[0257] Normal data acquisition: under the normal working condition of the security monitoring scene (no tampering, no abnormal behavior), 72 hours of video, audio and IMU data are continuously collected, and 10,000 time sequence dislocation feature vectors are generated for each scale (such as 32ms scale feature vector containing V-A, V-S, A-S three pairs of modal GCC-PHAT time delay, mutual information entropy, DTW cost, etc. 27 features), and four scale features are combined to form a high-dimensional training data set Dtrain (the dimension is 4x27=108, and the sample size is 400,000).

[0258] Data preprocessing: the Dtrain is standardized to eliminate dimensional differences, and the formula is: xnorm = σx − μ , wherein μ , σ are the mean and standard deviation of the feature in the training set, respectively.

[0259] 6.2 Manifold learning and low-dimensional embedding

[0260] The UMAP (Uniform Manifold Approximation and Projection) nonlinear manifold learning algorithm is used to map the 108-dimensional high-dimensional features to a 2-dimensional embedding space.

[0261] 6.3 Statistical boundary fitting

[0262] A one-class SVM unsupervised boundary learning algorithm is used to fit the distribution boundary of normal samples in the low-dimensional embedding space:

[0263] Model training: taking the low-dimensional embedding coordinates as input, setting the kernel function as radial basis function RBF, and the penalty parameter nu=0.01, a one-class SVM model is trained.

[0264] Boundary generation: the decision function outputted by the model can divide the low-dimensional space into "normal region" and "abnormal region", the average distance from the decision boundary to the cluster center is recorded as Dbound , as the reference benchmark for subsequent anomaly determination (e.g. Dbound =5.2, points with a distance exceeding this value are determined to deviate from the normal distribution).

[0265] Seven, multi-scale feature vector mapping and distance calculation

[0266] Input the multi-scale time sequence misplacement feature vector collected in real time into the TMM model, calculate its distance to the normal statistical boundary, the specific steps are as follows:

[0267] Real-time feature extraction: for the video, audio and IMU data at the current time (e.g. t=100s), generate four-scale feature vectors of 32ms, 128ms, 1s and 5s according to step three 、 、 、 ;

[0268] Low-dimensional mapping: input the four-scale real-time feature vectors into the UMAP model trained in step six in turn, to obtain the corresponding 2-dimensional embedded points 、 、 、 .

[0269] Calculate the distance of each embedded point to the TMM statistical boundary using the Euclidean distance, the formula is: , where , ) is the point on the boundary closest to the embedded point. For example:

[0270] P 32 The distance of (2.1, 3.5) to the boundary d 32 =3.8;

[0271] P 128 The distance of (1.8, 4.2) to the boundary d 128 =2.5;

[0272] P The distance of (0.9, 5.1) to the boundary d 1=4.1;

[0273] P The distance of (1.5, 3.9) to the boundary d 5=3.2;

[0274] Final scale distance set D scale ={3.8,2.5,4.1,3.2}。

[0275] Eight, multi-index fusion and final abnormality judgment

[0276] 8.1 Threshold setting

[0277] Based on statistical analysis of historical normal and abnormal data, set three types of judgment threshold (can be dynamically adjusted according to the scene):

[0278] Comprehensive consistency score threshold =0.6: below this value indicates poor multi-modal time series consistency;

[0279] Stability evaluation score threshold =0.5: below this value indicates time series misplacement fluctuation or drift anomaly;

[0280] Scale distance threshold: set according to scale, 32ms scale, =4.0, 128ms scale =3.5, 1s scale, =4.5, 5s scale, =3.8, exceeding the corresponding threshold indicates that the feature deviates from the normal distribution at this scale.

[0281] 8.2 Judgment rule execution

[0282] According to the results of steps four to seven, the following logic is used for judgment:

[0283] Abnormality judgment: if any of the following conditions is met, it is judged as abnormal:

[0284] Comprehensive consistency score C Cth (such as C=0.5<0.6);

[0285] Stability evaluation score Snorm Sth (such as Snorm =0.4<0.5);

[0286] Any distance in the scale distance set exceeds the corresponding threshold (such as d 1=4.6>4.5).

[0287] Normal judgment: if all the following conditions are met, it is judged as normal:

[0288] C Greater than or equal to ;

[0289] ​​S greater than or equal to ;

[0290] All scale distances are less than or equal to the corresponding threshold.

[0291] 8.3 Result output

[0292] Determination result output: if determined as abnormal, output abnormal label and rule identifier, for example: d 1=4.6>4.5 triggers an exception, and outputs an exception-scale distance violation (1s scale); if there is also C =0.5<0.6, then output an exception-comprehensive consistency violation+scale distance violation (1s scale);

[0293] It should be noted that when the device clock offset exceeds 10ms, the relative time window is used instead of the absolute time stamp, and the relative time delay of each modality signal in the 500ms window before and after the anchor point is calculated based on the timing anchor point, so as to weaken the influence of absolute clock error;

[0294] Sliding linear drift compensation: linear fitting is performed on the time delay sequence of 10 consecutive anchors, and if the drift rate exceeds 0.5ms / anchor, the compensation value of subsequent time delay calculation is adjusted in real time (for example, if the drift rate=0.6ms / anchor, then the time delay estimation value is reduced by 0.6ms every 1 anchor interval).

[0295] As shown in Figure 2 , the embodiment of the application provides a multi-modal anomaly detection system based on timing dislocation analysis, comprising:

[0296] A signal acquisition module is configured to synchronously acquire multi-modal timing signals and perform timestamp standardization processing on each modality signal.

[0297] A timing anchor point detection and injection module is configured to detect natural timing anchor point events in the multi-modal timing signals and / or inject artificial timing anchor point events into the acquisition environment by controlling external equipment; the timing anchor point event is a transient event that produces a detectable response in signals of at least two different modalities.

[0298] A timing dislocation feature calculation module is configured to set a plurality of time windows of different time scales; for each time scale, a timing dislocation feature vector between two different modalities is calculated based on the timing anchor point event within the corresponding time window.

[0299] A multi-scale feature aggregation module is configured to aggregate the timing dislocation feature vectors obtained under all time scales to form a multi-scale timing dislocation feature set, and generate a comprehensive consistency score using a weighted fusion method.

[0300] a stability evaluation module configured to calculate a stability evaluation score representing short-term fluctuations and long-term drift trends based on a historical time-series misalignment feature vector sequence, the historical time-series misalignment feature vector sequence being an ordered set of all time-series misalignment feature vectors generated by the time-series misalignment feature calculation module before the current calculation period;

[0301] a time-series misalignment manifold model construction module configured to construct a time-series misalignment manifold model by a manifold learning algorithm using historical multi-modal signal samples collected in a normal state to establish an embedding space and a statistical boundary representing inherent normal time-series relationships of the device and the scene;

[0302] an anomaly determination module configured to input each time-series misalignment feature vector in the multi-scale time-series misalignment feature set into the time-series misalignment manifold model, map it to a low-dimensional embedding space and calculate its distance to the statistical boundary, fuse distance calculation results at all scales, comprehensive consistency scores and stability evaluation scores, and make a final anomaly determination according to a preset rule.

[0303] In some embodiments, the time-series anchor point detection and injection module comprises:

[0304] a candidate anchor point detection unit configured to run a corresponding transient event detection algorithm on each modality of the multi-modal time-series signal to obtain a candidate anchor point timestamp of each modality;

[0305] an anchor point association matching unit configured to associate and match candidate anchor point timestamps of different modalities with a time difference less than a set threshold to confirm them as the same natural time-series anchor point event, and output a confirmed timestamp of the same natural time-series anchor point event in each modality signal.

[0306] In some embodiments, the time-series anchor point detection and injection module further comprises:

[0307] a control instruction generation and sending unit configured to generate a control instruction, record an injection command timestamp synchronously, and send the control instruction to an external device to trigger it to generate a transient event in the form of sound, light or vibration as an artificial time-series anchor point;

[0308] an artificial anchor point detection unit configured to detect the artificial time-series anchor point in the collected multi-modal time-series signal and determine its actual occurrence timestamp in each modality signal stream, and output the artificial time-series anchor point timestamp.

[0309] In some embodiments, the time-series misalignment feature calculation module calculates the time-series misalignment feature vector for any two modalities of time-series signals X(t) and Y(t) by performing the following steps:

[0310] a time delay estimation unit, which calculates the generalized cross-correlation phase transform function of X(t) and Y(t), extracts the global peak position of the function as the time delay estimation value τ, and calculates the peak ratio, full width at half maximum, and sidelobe ratio of the peak;

[0311] a mutual information calculation unit, which calculates the mutual information value of X(t) and Y(t) at different time offsets, generates a mutual information-offset curve, and extracts the peak position and distribution entropy of the curve;

[0312] a dynamic time warping unit, which calculates the dynamic time warping alignment path between X(t) and Y(t), and obtains the total cumulative cost of the path;

[0313] a jitter statistics unit, which calculates the sequence of the time delay estimation value τ in a short sliding window, and calculates the variance of the sequence;

[0314] a feature vector construction unit, which combines all or part of the feature indicators extracted by the above units to form a time series misalignment feature vector; the feature indicators include the time delay estimation value τ, the peak ratio, the full width at half maximum, the sidelobe ratio, the peak position and the distribution entropy, the total cumulative cost, and the variance.

[0315] In some embodiments, the multi-scale feature aggregation module includes:

[0316] a scale-specific score unit, which, for each time scale, inputs the time series misalignment feature vector in the time window at the scale into a pre-trained scale-specific score model, and outputs a scale consistency score representing the degree of synchronization consistency at the time scale ;

[0317] a weighted fusion unit, which performs weighted fusion on the scale consistency scores of the Q time scales to generate a comprehensive consistency score ; C ;

[0318]

[0319] wherein Q is the number of time scales, is the weight corresponding to the i th time scale, the weight allocation is negatively related to the length of the time scale, and the shorter the length of the time window of the time scale, the higher the weight .

[0320] In some embodiments, the stability evaluation module includes:

[0321] a time delay sequence extraction unit, which extracts a time delay sequence composed of the time delay estimation value τ in time sequence from the sequence of historical time series misalignment feature vectors τ 1, τ2,..., τm}, wherein m is the length of the history sequence;

[0322] a short-term fluctuation calculating unit, configured to calculate the variance or the standard deviation of the time delay sequence as a short-term fluctuation indicator within a set statistical analysis window; ;

[0323] a long-term drift calculating unit, configured to perform linear fitting on the time delay sequence within a set trend analysis window to obtain the slope of the fitted straight line, which is the long-term drift rate; ;

[0324] a stability score calculating unit, configured to substitute the short-term fluctuation indicator and the long-term drift rate into the following formula to calculate the stability evaluation score S :

[0325]

[0326] wherein and are weighting coefficients for adjusting the importance of the short-term fluctuation and the long-term drift, and ε is a very small positive number for preventing the denominator from being zero.

[0327] In some embodiments, the time series misalignment manifold model constructing module comprises:

[0328] a high-dimensional data set constructing unit, configured to extract time series misalignment feature vectors corresponding to historical multi-modal signal samples collected in a normal state to form a high-dimensional training data set;

[0329] a manifold learning unit, configured to perform dimension reduction on the high-dimensional training data set by using a nonlinear manifold learning algorithm, map each high-dimensional time series misalignment feature vector to a low-dimensional embedding space, and thus learn the mapping relationship from the high-dimensional feature space to the low-dimensional embedding space to form an embedding space of the time series misalignment manifold;

[0330] a statistical boundary fitting unit, configured to fit a statistical boundary surrounding or representing the distribution range of the normal samples in the low-dimensional embedding space by using an unsupervised boundary learning algorithm based on the distribution of all training data points.

[0331] In some embodiments, the anomaly determination module comprises:

[0332] a feature vector mapping and distance calculating unit, configured to input each time series misalignment feature vector in the multi-scale time series misalignment feature set into the time series misalignment manifold model in sequence, map each high-dimensional feature vector to the low-dimensional embedding space to obtain its corresponding low-dimensional embedding point ; and calculate the distance between each low-dimensional embedding point , the distance between the point and the statistical boundary is calculated using the time warp manifold model ; the distance is the Euclidean distance; a scale distance set corresponding to the multi-scale time warp feature set is obtained , where Q is the number of time scales.

[0333] In some embodiments, the anomaly determination module further comprises:

[0334] an anomaly determination unit, which obtains the scale distance set , a comprehensive consistency score C , and a stability evaluation score S ; compares the obtained information with preset threshold values, and makes a final determination according to the following rules:

[0335] if the comprehensive consistency score C is lower than a score threshold value , or the stability evaluation score S is lower than an evaluation threshold value , or any distance in the scale distance set exceeds the corresponding scale threshold value , it is determined to be abnormal;

[0336] if the comprehensive consistency score C is greater than or equal to the score threshold value , and the stability evaluation score S is greater than or equal to the evaluation threshold value , and all distances in the scale distance set are lower than or equal to the corresponding scale threshold value , it is determined to be normal;

[0337] a result output unit, which outputs the final anomaly determination result; if it is determined to be abnormal, one or more rule identifiers triggering the determination are also outputted; the rule identifiers are used to indicate the specific reason category leading to the anomaly determination, including a comprehensive consistency violation, a stability violation, or a scale distance violation.

[0338] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those of ordinary skill in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.​​

Claims

1. A multi-modal anomaly detection method based on timing misalignment analysis, the multi-modal data comprising video data, audio data, IMU inertial measurement unit data, acceleration data, environmental sensor data, characterized in that, The method comprises the following steps: S1, synchronously collecting multi-modal time series signals, and performing timestamp standardization processing on each modal signal; S2, detecting natural time sequence anchor point events in the multi-modal time series signals, and / or injecting artificial time sequence anchor point events into the acquisition environment by controlling external equipment; the time sequence anchor point event is a transient event that produces a detectable response in signals of at least two different modalities; S3, setting a plurality of time windows of different time scales; for each time scale, in the corresponding time window, taking the time sequence anchor point event as the reference, the time sequence misalignment feature vector between different modal time series signals is calculated; wherein, for any two modal time series signals X(t) and Y(t), the following steps are performed to calculate the time sequence misalignment feature vector: S31, calculating the generalized cross-correlation-phase transform function of X(t) and Y(t), extracting the global peak position of the function as the time delay estimation value τ, and calculating the peak value ratio, full width at half maximum and sidelobe ratio of the peak value; S32, calculating the mutual information value of X(t) and Y(t) under different time offset, generating a mutual information-offset curve, and extracting the peak position and distribution entropy of the curve; S33, calculating the dynamic time warping alignment path between X(t) and Y(t), and obtaining the total cumulative cost of the path; S34, in a short-time sliding window, a sequence of time delay estimation values τ is calculated, and the variance of the sequence is calculated; S35, combining all or part of the feature indicators extracted in steps S31 to S34 to form a time sequence misalignment feature vector; the feature indicators include time delay estimation value τ, peak value ratio, full width at half maximum, sidelobe ratio, peak position and distribution entropy, total cumulative cost, and variance; S4, aggregating the time sequence misalignment feature vectors obtained at all time scales to form a multi-scale time sequence misalignment feature set, and generating a comprehensive consistency score using a weighted fusion method; S5, based on a sequence of historical time sequence misalignment feature vectors, a stability evaluation score representing short-term fluctuations and long-term drift trends is calculated; the sequence of historical time sequence misalignment feature vectors refers to an ordered set of all time sequence misalignment feature vectors generated by S3 before the current calculation period; specifically including: S51, from the historical time sequence misalignment feature vector sequence, a time delay sequence composed of time delay estimation value τ is extracted in time sequence τ 1, τ 2,..., τm} wherein m is the historical sequence length; S52、in a set statistical analysis window, calculate the variance or standard deviation of the delay sequence as a short-term fluctuation indicator ; S53、in a set trend analysis window, linear fitting is performed on the time delay sequence to obtain a slope of a fitting straight line The slope is a long-term drift rate. S54、the short-term fluctuation index and the long-term drift rate Substituting into the following formula, calculate the stability evaluation score S : wherein and are weighting factors for the importance of short-term fluctuations and long-term drifts, and ε is a preset small positive number to prevent the denominator from being zero. S6, using historical multi-modal signal samples collected under normal conditions, a time sequence misalignment manifold model is constructed by a manifold learning algorithm to establish an embedding space and a statistical boundary representing the inherent normal time sequence relationship of the device and the scene; S7, inputting each time sequence misalignment feature vector in the multi-scale time sequence misalignment feature set into the time sequence misalignment manifold model, mapping it to a low-dimensional embedding space and calculating its distance from the statistical boundary; S8, fusing the distance calculation results at all scales, the comprehensive consistency score and the stability evaluation score, and making a final abnormality judgment according to a preset rule.

2. The method of claim 1, wherein, The detection of natural time sequence anchor point events in S2 specifically includes: respectively running a corresponding transient event detection algorithm on each modal time sequence signal in the multi-modal time series signals to obtain candidate anchor point timestamps for each modality; Correlate and match different modalities at candidate anchor time stamps with a time difference less than a set threshold, confirm that they are the same natural time sequence anchor event, and output the confirmed time stamps of the same natural time sequence anchor event in each modality signal.

3. The method of claim 2, wherein, In S2, injecting an artificial time sequence anchor event specifically includes: Generating a control instruction, synchronously recording an injection command time stamp, and sending the control instruction to an external device to trigger it to generate a transient event in the form of sound, light, or vibration as an artificial time sequence anchor point; In the collected multi-modal time sequence signal, detecting the artificial time sequence anchor point and determining its actual occurrence time stamp in each modality signal stream; outputting the actual occurrence time stamp, i.e., the artificial time sequence anchor point time stamp.

4. The method of claim 3, wherein, The steps of S4 include: For each time scale, input the time series misalignment feature vector within the time window at the scale into a pre-trained scale-specific score model, and output a scale consistency score representing the degree of synchronization consistency at the time scale ; scale-consistency score for all Q time scales performing a weighted fusion to generate a comprehensive consistency score C ; wherein Q is the number of time scales, is the weight corresponding to the i th time scale, the allocation of the weight is negatively related to the length of the time scale, the shorter the length of the time window of the time scale, the higher the weight of the time scale.

5. The method of claim 4, wherein, The steps of S6 include: S61, extracting the time sequence misalignment feature vector corresponding to the historical multi-modal signal sample collected under normal conditions to form a high-dimensional training data set; S62, using a nonlinear manifold learning algorithm to reduce the dimension of the high-dimensional training data set, mapping each high-dimensional time sequence misalignment feature vector to a low-dimensional embedding space, and thus learning the mapping relationship from the high-dimensional feature space to the low-dimensional embedding space to form the embedding space of the time sequence misalignment manifold; S63, in the low-dimensional embedding space, based on the distribution of all training data points, an unsupervised boundary learning algorithm is used to fit a statistical boundary representing the distribution range of normal samples.

6. The method of claim 5, wherein, The steps of S7 include: S71. Input each temporal misalignment feature vector from the multi-scale temporal misalignment feature set into the temporal misalignment manifold model in sequence, and map each high-dimensional feature vector to a low-dimensional embedding space to obtain its corresponding low-dimensional embedding point. ; S72, for each low-dimensional embedding point , calculate the distance between the point and the statistical boundary using the time warped manifold model ; the distance is the Euclidean distance; obtain a scale distance set corresponding to the multi-scale time warped feature set , where Q is the number of time scales.

7. The method of claim 6, wherein, The steps of S8 include: S81, the acquired scale distance set , the comprehensive consistency score C , and the stability evaluation score S ; S82, comparing the obtained information with a preset threshold, and making a final determination according to the following rules: if the integrated consistency score C is below a score threshold or the stability evaluation score S is below an evaluation threshold or the set of scale distances has any one distance exceeding its corresponding scale threshold then the data is determined to be anomalous; if the integrated consistency score C is greater than or equal to a score threshold , and the stability evaluation score S is greater than or equal to an evaluation threshold , and all distances in the set of scale distances are less than or equal to their corresponding scale thresholds , then the sample is determined to be normal; S83, outputting the final abnormality determination result; if it is determined to be abnormal, outputting one or more rule identifiers that trigger the determination; the rule identifier is used to indicate the specific reason category that leads to the abnormality determination, including comprehensive consistency violation, stability violation, or scale distance violation.

8. A multi-modal anomaly detection system based on timing skew analysis for implementing the method of any one of claims 1-7, characterized in that, It includes: A signal acquisition module for synchronously acquiring multi-modal time sequence signals and performing time stamp standardization processing on each modality signal; A time sequence anchor point detection and injection module for detecting natural time sequence anchor events in the multi-modal time sequence signals, and / or injecting artificial time sequence anchor events into the acquisition environment by controlling external devices; the time sequence anchor event is a transient event that produces a detectable response in signals of at least two different modalities; A time sequence misalignment feature calculation module for setting a plurality of time windows of different time scales; for each time scale, within the corresponding time window, taking the time sequence anchor event as a reference, calculating the time sequence misalignment feature vector between different modalities of time sequence signals; A multi-scale feature aggregation module for aggregating the time sequence misalignment feature vectors obtained under all time scales to form a multi-scale time sequence misalignment feature set, and generating a comprehensive consistency score using a weighted fusion method; A stability evaluation module for calculating a stability evaluation score representing short-term fluctuations and long-term drift trends based on a sequence of historical time sequence misalignment feature vectors; the sequence of historical time sequence misalignment feature vectors refers to an ordered set of all time sequence misalignment feature vectors generated by the time sequence misalignment feature calculation module before the current calculation period; The time sequence dislocation manifold model construction module is configured to utilize historical multi-modal signal samples collected in a normal state, and construct a time sequence dislocation manifold model by a manifold learning algorithm, so as to establish an embedding space and a statistical boundary representing inherent normal time sequence relationships of the device and the scene. The anomaly determination module is configured to input each time sequence dislocation feature vector in the multi-scale time sequence dislocation feature set into the time sequence dislocation manifold model, map the time sequence dislocation feature vector to the low-dimensional embedding space, and calculate a distance between the time sequence dislocation feature vector and the statistical boundary. The anomaly determination module is further configured to fuse distance calculation results at all scales, a comprehensive consistency score, and a stability evaluation score, and perform final anomaly determination according to a preset rule.

Images