Multimodal anomaly detection method and system for audio and video streams based on feature tracking

Abstract

The present invention belongs to the field of audio and video processing technology, and discloses a multimodal real-time anomaly detection method and system for audio and video streams based on intelligent feature tracking, including: accurately matching the current frame with the historical frame through grayscale image conversion, feature point extraction and global alignment, and judging abnormal feature points in combination with feature point displacement and local grayscale difference, which can effectively capture subtle changes in the video. Based on the dynamic threshold, the video anomaly is judged in combination with the number of abnormal, disappeared and newly added feature points, thereby enhancing the adaptability and accuracy of the detection. At the same time, the sampling parameters are set and frame division and STFT processing are performed, and the logarithmic Mel spectrum is extracted using the Mel filter group. The audio anomaly is comprehensively analyzed through the comparison of multiple indicators such as energy ratio, spectrum center difference and Frobenius norm difference. The multimodal combination of audio and video and comprehensive judgment from different dimensions improves the comprehensiveness and reliability of anomaly detection, and is suitable for scenarios such as real-time monitoring.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio and video processing, and relates to a method and system for detecting multimodal anomalies in audio and video streams based on feature tracking. Background Art

[0002] With the increasing demand for social security and the accelerated pace of urbanization, video surveillance systems have been widely used in fields such as public safety, intelligent transportation, and industrial monitoring. However, existing surveillance systems primarily rely on manual monitoring or simple motion detection algorithms, such as frame difference and background subtraction. These systems exhibit significant shortcomings in practical applications: Limited by manpower and availability, manual monitoring cannot achieve all-weather, all-time monitoring. Furthermore, simple motion detection algorithms are slow to respond to emergencies in complex environments and struggle to capture early abnormal changes in the image. Simple difference-based methods are susceptible to changes in lighting and background noise, resulting in high false alarm and false negative rates. Furthermore, single-modality detection (video or audio only) has limitations and cannot fully reflect the on-site situation. Furthermore, existing systems lack efficient video data caching and storage mechanisms, making it difficult to quickly locate critical time periods and image data after an abnormal event occurs, hindering subsequent analysis and evidence collection. Therefore, there is an urgent need for an advanced monitoring technology solution that integrates visual and auditory multimodal information and has intelligent feature tracking, global motion compensation, and real-time data backtracking. This can accurately capture subtle changes and provide timely warnings in the embryonic stage of abnormal events, while ensuring the complete preservation of post-event data, providing reliable support for safe disposal and subsequent investigations. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems in the existing technology of insufficient real-time performance of manual monitoring, low detection accuracy, inability to fully reflect on-site conditions and difficulty in data management, and to provide a multimodal anomaly detection method and system for audio and video streams based on feature tracking.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The multimodal anomaly detection method for audio and video streams based on feature tracking includes:

[0006] Collect and store audio and video stream data, and select the video stream data of the current frame and any stored historical frame video stream data for conversion to obtain a grayscale image;

[0007] Feature points are extracted from the grayscale images corresponding to the current frame video stream data and the historical frame video stream data, and the grayscale images and feature points corresponding to the historical frame video stream data are globally aligned to match the current frame with the globally aligned historical frame image.

[0008] Extract local areas from the current frame and the aligned historical frame images, and determine whether each feature point is an abnormal feature point based on the displacement of the feature points at different times and the average grayscale difference of the local area;

[0009] Determine whether the selected video stream data is abnormal based on the relationship between the number of abnormal feature points, the sum of disappeared feature points and newly added feature points and a preset dynamic threshold;

[0010] Setting audio sampling parameters and extracting audio data from the selected audio and video stream data;

[0011] The audio data is framed and processed by STFT to obtain the audio spectrum, which is then extracted and converted into a logarithmic Mel spectrum graph using a Mel filter bank;

[0012] Based on the set time interval m, obtain the energy ratio index and spectrum center difference between the current audio segment and the previous m seconds audio segment; and based on the logarithmic Mel-spectrogram, obtain the Frobenius norm difference of the logarithmic Mel-spectrogram;

[0013] The energy ratio index, spectrum center difference and Frobenius norm difference are compared with the preset energy ratio threshold, spectrum center difference threshold and Frobenius norm difference threshold respectively to determine whether the extracted audio is an abnormal sound.

[0014] A further improvement of the present invention is:

[0015] Furthermore, feature points are extracted from the grayscale images corresponding to the current frame video stream data and the historical frame video stream data, and the grayscale images and feature points corresponding to the historical frame video stream data are globally aligned, thereby matching the current frame with the globally aligned historical frame image. Specifically,

[0016] The current frame f t Video stream data and historical frames Convert the video stream data into a grayscale image;

[0017] Based on the corner detection algorithm in the current frame f t Extract feature point set from ;

[0018]

[0019] in, is the i-th feature point of the current frame, and r is the total number of feature points;

[0020] Historical Frame and its corresponding feature points are globally aligned so that the current frame f t With historical frame Matching is performed in the same coordinate system;

[0021]

[0022] in, is the i-th feature point of the historical frame; is the alignment transformation matrix; f t-k is the aligned historical frame, is the i-th feature point of the aligned historical frame.

[0023] Furthermore, local regions are extracted from the current frame and the aligned historical frame images. Based on the displacement of the feature points at different times and the average grayscale difference of the local region, whether each feature point is an abnormal feature point is determined as follows:

[0024] The displacement of the feature point at different times is:

[0025]

[0026] d i =||Δp i ||

[0027] Where Δp i is the displacement value of the feature point at different times, d i Δp i The absolute value of

[0028] In order to eliminate the influence of different device fields of view and image sizes, the displacement of feature points at different times is normalized as follows:

[0029]

[0030] Among them, D max Generally, the diagonal length of the image or the upper limit set according to the field of view of the device is taken; is the normalized value of the feature point;

[0031] For each feature point, a local patch of size V×V is extracted from the current frame and the aligned historical frame, and the grayscale value is normalized to [0,1] and then calculated:

[0032]

[0033] Among them, D i Represents the average grayscale difference of the local area; represents the neighborhood image frame centered on the i-th feature point of the current frame, Represents the neighborhood image frame centered on the i-th feature point of the historical frame;

[0034] For each feature point p iWhether it is an abnormal feature point is judged. When the normalized displacement value of the feature point is greater than the displacement abnormality threshold, or the average grayscale difference of the local area is greater than the local difference threshold, the current feature point is an abnormal feature point. Specifically:

[0035]

[0036] Among them, θ d is the displacement anomaly threshold, θ p is the local difference threshold; A i Indicates the abnormal feature point flag, A i When it is 1, it is an abnormal point, A i When it is 0, it is a non-abnormal point.

[0037] Furthermore, based on the relationship between the number of abnormal feature points, the sum of disappeared feature points and newly added feature points and a preset dynamic threshold, it is determined whether the selected video stream data is abnormal, specifically:

[0038] Count the number of abnormal feature points, specifically:

[0039]

[0040] in, is the number of abnormal feature points;

[0041] The total number of anomalies is as follows:

[0042]

[0043] Among them, N abn is the total number of anomalies, N disappeared is the number of disappeared feature points, that is, the number of feature points that existed in the historical frame but were not matched in the current frame; N new is the number of newly added feature points, that is, the number of feature points that appear in the current frame but do not exist in the historical frames;

[0044] Compare the obtained total abnormality number with the preset dynamic threshold, and when the total abnormality number is greater than the preset dynamic threshold, determine whether the selected video stream data is abnormal;

[0045] The preset dynamic threshold is

[0046] θ n (t) = α2·M + β2

[0047] Among them, M is the total number of feature points in the current frame; α2 represents the probability that a single feature point in the current scene is misjudged as an abnormality; β2 is the compensation coefficient.

[0048] Further, set the audio sampling parameters, and extract the audio data from the selected audio-visual stream data, specifically:

[0049] The audio sampling parameters include: sampling rate, quantization bit depth, number of channels, and data format;

[0050] Divide the collected audio data to obtain several data segments;

[0051] The length of the data segment is processed with a 1-second time window, and each segment contains 16,000 sampling data;

[0052] The data per second constitutes a time window,

[0053]

[0054] where x t represents the audio sampling data vector at the t-th second, represents the value of the i-th sampling point in the t-th second, is an integer set.

[0055] Further, perform frame division and STFT processing on the audio data to obtain an audio spectrum, and extract and convert it into a logarithmic mel spectrogram with the help of a mel filter bank, specifically:

[0056] Divide the audio data into frames:

[0057] x n (u) = x(u + nS), 0 ≤ u < L, 0 ≤ n < N

[0058] where u is the sample index in each frame, indicating the number of the sampling point within the frame, with the range 0 ≤ u < L, where L is the length of each frame;

[0059] n is the frame number, indicating which frame it is currently, with the range 0 ≤ n < N, where N is the total number of frames;

[0060] S is the frame shift, indicating the interval between the starting points of adjacent frames, in the unit of the number of sampling points;

[0061] Weighted Hamming window:

[0062]

[0063] where π is a mathematical symbol;

[0064] [[ID=5�]]Calculate the STFT of each frame:

[0065]

[0066] where j is the imaginary unit;

[0067] Use the Mel filter bank to map the STFT to the Mel scale:

[0068]

[0069] Among them, H l (k) represents the weight of the lth Mel filter corresponding to the kth frequency point; P represents the number of Mel filters;

[0070] Then perform logarithmic transformation, specifically:

[0071] logS n (l) = log(S n (l)+δ)

[0072] Here, δ is a positive constant used to avoid the calculation problem of log(0) in logarithmic operations.

[0073] Furthermore, based on the set time interval m, the energy ratio index and spectrum center difference between the current audio segment and the previous m seconds audio segment are obtained, specifically:

[0074] The energy ratio index of the current audio segment and the previous m seconds audio segment is specifically:

[0075] The root mean square energy of the current audio segment is:

[0076]

[0077] Where N is the number of sampled data;

[0078] The root mean square energy E of the current segment t and the previous RMS energy E t-m The ratio is

[0079]

[0080] Where ε is a positive constant used to avoid the denominator being zero;

[0081] Perform FFT on the audio to get the spectrum S t (f) Calculate the spectrum center, specifically:

[0082]

[0083] The difference between the center of the spectrum of the current audio segment and the previous m seconds of audio segment is:

[0084] ΔC=|C t -C t-m |

[0085] The method of obtaining the Frobenius norm difference of the logarithmic Mel-spectrogram based on the logarithmic Mel-spectrogram is specifically as follows:

[0086] The difference in the Frobenius norm of the logarithmic Mel spectrogram between the current time t and the previous m seconds is D S (t,tm), expressed as the logarithmic Mel spectrum Frobenius norm difference is

[0087] D S (t,tm)=||logS t -logS t-m || F

[0088] Among them, logS t is the logarithmic Mel spectrum at the current time t, logS t-m The logarithmic Mel-spectrogram of the first m seconds.

[0089] The multimodal anomaly detection system for audio and video streams based on feature tracking includes:

[0090] An acquisition module, which acquires and stores audio and video stream data, and selects the video stream data of the current frame and any stored historical frame video stream data for conversion to obtain a grayscale image;

[0091] A global alignment module, which extracts feature points from the grayscale images corresponding to the current frame video stream data and the historical frame video stream data, and globally aligns the grayscale images and feature points corresponding to the historical frame video stream data, thereby matching the current frame with the globally aligned historical frame image;

[0092] a first judgment module, which extracts a local area from the current frame and the aligned historical frame images, and judges whether each feature point is an abnormal feature point based on the displacement of the feature point at different times and the average grayscale difference of the local area;

[0093] a second judgment module, which judges whether the selected video stream data is abnormal based on the relationship between the number of abnormal feature points, the sum of disappeared feature points and newly added feature points and a preset dynamic threshold;

[0094] An extraction module, wherein the extraction module sets audio sampling parameters and extracts audio data from the selected audio and video stream data;

[0095] A conversion module, which performs framing and STFT processing on the audio data to obtain an audio spectrum, and extracts and converts it into a logarithmic Mel spectrum graph using a Mel filter bank;

[0096] an acquisition module, wherein the acquisition module acquires an energy ratio index and a spectrum center difference between a current audio segment and an audio segment m seconds before the current audio segment based on a set time interval m; and acquires a Frobenius norm difference of the logarithmic Mel-spectrogram based on the logarithmic Mel-spectrogram;

[0097] The third judgment module compares the energy ratio index, the spectrum center difference and the Frobenius norm difference with the preset energy ratio threshold, the spectrum center difference threshold and the Frobenius norm difference threshold respectively to determine whether the extracted audio is an abnormal sound.

[0098] Compared with the prior art, the present invention has the following beneficial effects:

[0099] The present invention accurately matches the current frame with the historical frame through grayscale conversion, feature point extraction and global alignment, and judges abnormal feature points by combining feature point displacement and local grayscale difference, which can effectively capture subtle changes in the video. Based on the dynamic threshold, the video anomaly is judged by combining the number of abnormal, disappeared and newly added feature points, which enhances the adaptability and accuracy of detection. At the same time, the sampling parameters are set and frame division and STFT processing are performed. The logarithmic Mel spectrum is extracted using the Mel filter group. The audio anomaly is comprehensively analyzed through the comparison of multiple indicators such as energy ratio, spectrum center difference and Frobenius norm difference. The multi-modal combination of audio and video and comprehensive judgment from different dimensions improves the comprehensiveness and reliability of anomaly detection, and is suitable for scenarios such as real-time monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0101] Figure 1 Schematic diagram of the flow of the multimodal anomaly detection method for audio and video streams based on feature tracking of the present invention;

[0102] Figure 2 This is a structural diagram of the multimodal anomaly detection system for audio and video streams based on feature tracking of the present invention. DETAILED DESCRIPTION

[0103] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0104] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0105] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition or explanation in subsequent drawings.

[0106] In the description of the embodiments of the present invention, it should be noted that if the terms "upper," "lower," "horizontal," "inner," etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the inventive product is typically placed when in use. These terms are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0107] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0108] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0109] The present invention is described in further detail below with reference to the accompanying drawings:

[0110] See also Figure 1 The present invention discloses a multimodal anomaly detection method for audio and video streams based on feature tracking, comprising:

[0111] S101, collecting and storing audio and video stream data, and selecting the video stream data of the current frame and any stored historical frame video stream data for conversion to obtain a grayscale image;

[0112] S102, extracting feature points from the grayscale images corresponding to the current frame video stream data and the historical frame video stream data, and globally aligning the grayscale images and feature points corresponding to the historical frame video stream data, thereby matching the current frame with the globally aligned historical frame image;

[0113] The current frame f t Video stream data and historical frames Convert the video stream data into a grayscale image;

[0114] Based on the corner detection algorithm in the current frame f t Extract feature point set from ;

[0115]

[0116] in, is the i-th feature point of the current frame, and r is the total number of feature points;

[0117] Historical Frame and its corresponding feature points are globally aligned so that the current frame f t With historical frame Matching is performed in the same coordinate system;

[0118]

[0119] in, is the i-th feature point of the historical frame; is the alignment transformation matrix; f t-k is the aligned historical frame, is the i-th feature point of the aligned historical frame.

[0120] S103, extracting local regions from the current frame and the aligned historical frame images, and determining whether each feature point is an abnormal feature point based on the displacement of the feature points at different times and the average grayscale difference of the local region;

[0121] The displacement of the feature point at different times is:

[0122]

[0123] d i =||Δp i ||

[0124] Where Δp i is the displacement value of the feature point at different times, d i Δp i The absolute value of

[0125] In order to eliminate the influence of different device fields of view and image sizes, the displacement of feature points at different times is normalized as follows:

[0126]

[0127] Among them, D max Generally, the diagonal length of the image or the upper limit set according to the field of view of the device is taken; is the normalized value of the feature point;

[0128] For each feature point, a local patch of size V×V is extracted from the current frame and the aligned historical frame, and the grayscale value is normalized to [0,1] and then calculated:

[0129]

[0130] Among them, D i Represents the average grayscale difference of the local area; represents the neighborhood image frame centered on the i-th feature point of the current frame, Represents the neighborhood image frame centered on the i-th feature point of the historical frame;

[0131] For each feature point p i Whether it is an abnormal feature point is judged. When the normalized displacement value of the feature point is greater than the displacement abnormality threshold, or the average grayscale difference of the local area is greater than the local difference threshold, the current feature point is an abnormal feature point. Specifically:

[0132]

[0133] Among them, θ d is the displacement anomaly threshold, θ p is the local difference threshold; A i Indicates the abnormal feature point flag, A i When it is 1, it is an abnormal point, A i When it is 0, it is a non-abnormal point.

[0134] Displacement abnormal threshold θ d The local difference threshold θ is set to 0.05~0.15 in the normalized range. If the threshold is exceeded, it is considered as abnormal motion. p Take a small value within the normalized grayscale range, the specific value depends on the noise level and scene complexity.

[0135] S104, determining whether the selected video stream data is abnormal based on the relationship between the number of abnormal feature points, the sum of disappeared feature points and newly added feature points and a preset dynamic threshold;

[0136] Count the number of abnormal feature points, specifically:

[0137]

[0138] in, is the number of abnormal feature points;

[0139] The total number of anomalies is as follows:

[0140]

[0141] Among them, N abn is the total number of anomalies, N disappeared is the number of disappeared feature points, that is, the number of feature points that existed in the historical frame but were not matched in the current frame; N new is the number of newly added feature points, that is, the number of feature points that appear in the current frame but do not exist in the historical frames;

[0142] Compare the obtained total abnormality number with the preset dynamic threshold, and when the total abnormality number is greater than the preset dynamic threshold, determine whether the selected video stream data is abnormal;

[0143] The preset dynamic threshold is

[0144] θ n (t) = α2·M + β2

[0145] Among them, M is the total number of feature points in the current frame; α2 represents the probability that a single feature point in the current scene is misjudged as an abnormality; β2 is the compensation coefficient.

[0146] α2 is determined by collecting a large number of labeled video samples, including both normal and abnormal video. Statistical analysis is then performed on the feature points extracted from each frame. The percentage of misclassified feature points due to factors such as lighting variations, noise, and camera shake is calculated, even when no abnormalities are present. For example, if, after multiple experiments, statistics show that the average percentage of misclassified feature points in a frame is between 2% and 5%, the α2 value is 0.02 and 0.05.

[0147] The recommended range is generally between 0.01 and 0.05. If the α2 value is too low, it may underestimate false positives due to noise, resulting in a higher false alarm rate in noisy environments. If the value is too high, anomaly detection may be overly sensitive, misclassifying normal fluctuations as anomalies. Therefore, it is generally recommended to determine an intermediate value through offline experiments and then adaptively adjust it based on actual scenarios.

[0148] The reference value range of β2 is usually recommended to be between 5 and 15. The specific value depends on the resolution of the video collected by the system, the number and distribution of feature points, and experimental statistical results.

[0149] S105, setting audio sampling parameters and extracting audio data from the selected audio and video stream data;

[0150] The audio sampling parameters include: sampling rate, quantization bit depth, number of channels, and data format;

[0151] Divide the collected audio data to obtain several data segments;

[0152] The length of the data segment is processed with a 1-second time window, and each segment contains 16,000 sampling data; The data per second forms a time window, specifically:

[0153]

[0154] where x t represents the audio sampling data vector at the t-th second, represents the value of the i-th sampling point in the t-th second, is an integer set, and the value range is within [-32768, 32767].

[0155] S106, perform frame division and STFT processing on the audio data to obtain an audio spectrum, and extract and convert it into a logarithmic mel spectrogram with the help of a mel filter bank;

[0156] Divide the audio data into frames, specifically:

[0157] x n (u) = x(u + nS), 0 ≤ u < L, 0 ≤ n < N

[0158] where u is the sample index in each frame, representing the number of the sampling point within the frame, with the range 0 ≤ u < L, where L is the length of each frame;

[0159] n is the frame number, representing which frame it is currently, with the range 0 ≤ n < N, where N is the total number of frames;

[0160] S is the frame shift, representing the interval between the starting points of adjacent frames, in the unit of the number of sampling points;

[0161] Weighted Hamming window:

[0162]

[0163] where π is a mathematical symbol;

[0164] Calculate the STFT for each frame:

[0165]

[0166] where j is the imaginary unit;

[0167] Use the mel filter bank to map the STFT to the mel scale:

[0168]

[0169] Among them, H l (k) represents the weight of the lth Mel filter corresponding to the kth frequency point; P represents the number of Mel filters;

[0170] Then perform logarithmic transformation, specifically:

[0171] logS n (l) = log(S n (l)+δ)

[0172] Here, δ is a positive constant used to avoid the calculation problem of log(0) in logarithmic operations.

[0173] S107, based on a set time interval m, obtaining an energy ratio index and a spectrum center difference between the current audio segment and the previous m seconds of the audio segment; and based on the logarithmic Mel-spectrogram, obtaining a Frobenius norm difference of the logarithmic Mel-spectrogram;

[0174] Based on the set time interval m, obtain the energy ratio index and spectrum center difference between the current audio segment and the previous m seconds audio segment, specifically:

[0175] The energy ratio index of the current audio segment and the previous m seconds audio segment is specifically:

[0176] The root mean square energy of the current audio segment is:

[0177]

[0178] Where N is the number of sampled data;

[0179] The root mean square energy E of the current segment t and the previous RMS energy E t-m The ratio is

[0180]

[0181] Where ε is a positive constant used to avoid the denominator being zero;

[0182] Perform FFT on the audio to get the spectrum S t (f) Calculate the spectrum center, specifically:

[0183]

[0184] The difference between the center of the spectrum of the current audio segment and the previous m seconds of audio segment is:

[0185] ΔC=|C t -C t-m |

[0186] The method of obtaining the Frobenius norm difference of the logarithmic Mel-spectrogram based on the logarithmic Mel-spectrogram is specifically as follows:

[0187] The difference in the Frobenius norm of the logarithmic Mel spectrogram between the current time t and the previous m seconds is D S (t,tm), expressed as

[0188] D S (t,tm)=||logS t -logS t-m || F

[0189] Among them, logS t is the logarithmic Mel spectrum at the current time t, logS t-m is the logarithmic Mel spectrogram of the first m seconds; the Frobenius norm difference D between the logarithmic Mel spectrograms S (t, tk) is used to measure the change in the spectrum structure between the current window and the historical window.

[0190] S108 , respectively comparing the energy ratio index, the spectrum center difference, and the Frobenius norm difference with a preset energy ratio threshold, a spectrum center difference threshold, and a Frobenius norm difference threshold to determine whether the extracted audio is an abnormal sound.

[0191] If R E When ΔC>1.5, it means that the energy of the current sound segment is significantly improved; if ΔC>300Hz, it is considered that the spectrum distribution has changed significantly; when any indicator meets the conditions, such as R E >1.5 or ΔC>300Hz, and D S When (t, tm) is greater than the preset threshold and the state lasts longer than the preset minimum abnormal duration MIN_ABNORMAL_DURATION, such as 1 to 2 seconds, the system determines it as an abnormal sound and triggers an early warning.

[0192] See also Figure 2 The present invention discloses a multimodal anomaly detection system for audio and video streams based on feature tracking, comprising:

[0193] A conversion module, wherein the conversion module collects and stores audio and video stream data, and selects the video stream data of the current frame and any stored historical frame video stream data for conversion to obtain a grayscale image;

[0194] A global alignment module, which extracts feature points from the grayscale images corresponding to the current frame video stream data and the historical frame video stream data, and globally aligns the grayscale images and feature points corresponding to the historical frame video stream data, thereby matching the current frame with the globally aligned historical frame image;

[0195] a first judgment module, which extracts a local area from the current frame and the aligned historical frame images, and judges whether each feature point is an abnormal feature point based on the displacement of the feature point at different times and the average grayscale difference of the local area;

[0196] a second judgment module, which judges whether the selected video stream data is abnormal based on the relationship between the number of abnormal feature points, the sum of disappeared feature points and newly added feature points and a preset dynamic threshold;

[0197] An extraction module, wherein the extraction module sets audio sampling parameters and extracts audio data from the selected audio and video stream data;

[0198] A conversion module, which performs framing and STFT processing on the audio data to obtain an audio spectrum, and extracts and converts it into a logarithmic Mel spectrum graph using a Mel filter bank;

[0199] an acquisition module, wherein the acquisition module acquires an energy ratio index and a spectrum center difference between a current audio segment and an audio segment m seconds before the current audio segment based on a set time interval m; and acquires a Frobenius norm difference of the logarithmic Mel-spectrogram based on the logarithmic Mel-spectrogram;

[0200] The third judgment module compares the energy ratio index, the spectrum center difference and the Frobenius norm difference with the preset energy ratio threshold, the spectrum center difference threshold and the Frobenius norm difference threshold respectively to determine whether the extracted audio is an abnormal sound.

[0201] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A multimodal anomaly detection method for audio and video streams based on feature tracking, characterized in that: include: Collect and store audio and video stream data, and select the video stream data of the current frame and any stored historical frame video stream data for conversion to obtain a grayscale image; Feature points are extracted from the grayscale images corresponding to the current frame video stream data and the historical frame video stream data, and the grayscale images and feature points corresponding to the historical frame video stream data are globally aligned to match the current frame with the globally aligned historical frame image. Extract local areas from the current frame and the aligned historical frame images, and determine whether each feature point is an abnormal feature point based on the displacement of the feature points at different times and the average grayscale difference of the local area; The displacement of the feature point at different times is: d i =||Δp i ||; Where Δp i is the displacement value of the feature point at different times, d i Δp i The absolute value of Indicates the position of the i-th feature point at the current time t; Indicates the position of the i-th feature point at the current time tk; In order to eliminate the influence of different device fields of view and image sizes, the displacement of feature points at different times is normalized as follows: Among them, D max Take the diagonal length of the image or the upper bound set according to the field of view of the device; is the normalized value of the feature point; For each feature point, a local patch of size V×V is extracted from the current frame and the aligned historical frame, and the grayscale value is normalized to [0,1] and then calculated: Among them, D i Represents the average grayscale difference of the local area; represents the neighborhood image frame centered on the i-th feature point of the current frame t, Represents the neighborhood image frame centered on the i-th feature point of the historical frame tk; Represents the norm of the difference vector between two image frames; For each feature point p i Whether it is an abnormal feature point is judged. When the normalized displacement value of the feature point is greater than the displacement abnormality threshold, or the average grayscale difference of the local area is greater than the local difference threshold, the current feature point is an abnormal feature point. Specifically: Among them, θ d is the displacement anomaly threshold, θ p is the local difference threshold; A i Indicates the abnormal feature point flag, A i When it is 1, it is an abnormal point, A i When it is 0, it is a non-abnormal point; Determine whether the selected video stream data is abnormal based on the relationship between the number of abnormal feature points, the sum of disappeared feature points and newly added feature points and a preset dynamic threshold; Count the number of abnormal feature points, specifically: in, is the number of abnormal feature points; r is the total number of feature sampling points; The total number of anomalies is as follows: Among them, N abn is the total number of anomalies, N disappeared is the number of disappeared feature points, that is, the number of feature points that existed in the historical frame but were not matched in the current frame; N new is the number of newly added feature points, that is, the number of feature points that appear in the current frame but do not exist in the historical frames; Compare the obtained total abnormality number with the preset dynamic threshold, and when the total abnormality number is greater than the preset dynamic threshold, determine whether the selected video stream data is abnormal; The preset dynamic threshold is θ n (t) = α2·M+β2; Where M is the total number of feature points in the current frame; α2 represents the probability that a single feature point in the current scene is misjudged as an abnormality; β2 is the compensation coefficient; Setting audio sampling parameters and extracting audio data from the selected audio and video stream data; The audio data is framed and processed by STFT to obtain the audio spectrum, which is then extracted and converted into a logarithmic Mel spectrum graph using a Mel filter bank; Based on the set time interval m, obtain the energy ratio index and spectrum center difference between the current audio segment and the previous m seconds audio segment; and based on the logarithmic Mel-spectrogram, obtain the Frobenius norm difference of the logarithmic Mel-spectrogram; The energy ratio index of the current audio segment and the previous m seconds audio segment is specifically: The root mean square energy of the current audio segment is: Where N is the number of sampled data; is the value of the cth sound data sampling point in the tth second; The root mean square energy E of the current segment t and the previous RMS energy E t-m The ratio is Where ε is a positive constant used to avoid the denominator being zero; Perform FFT on the audio to get the spectrum S t (f) Calculate the spectrum center, specifically: Among them, S t (f) represents the spectrum value obtained by FFT of the audio signal at time t; f is the frequency variable, and F is the frequency set; The difference between the center of the spectrum of the current audio segment and the previous m seconds of audio segment is: ΔC=|C t -C t-m |; Among them, C t is the spectrum center at time t, C t-m is the center of the spectrum at time tm, where tm is m seconds before time t; The method of obtaining the Frobenius norm difference of the logarithmic Mel-spectrogram based on the logarithmic Mel-spectrogram is specifically as follows: The difference in the Frobenius norm of the logarithmic Mel spectrogram between the current time t and the previous m seconds is D S (t,tm), expressed as D S (t,t-m)=||logS t -logS t-m || F ; Among them, logS t is the logarithmic Mel spectrum at the current time t, logS t-m is the logarithmic Mel spectrum of the first m seconds; The energy ratio index, spectrum center difference and Frobenius norm difference are compared with the preset energy ratio threshold, spectrum center difference threshold and Frobenius norm difference threshold respectively to determine whether the extracted audio is an abnormal sound.

2. The method for detecting multimodal anomalies of audio and video streams based on feature tracking according to claim 1 is characterized in that: The feature points of the grayscale images corresponding to the current frame video stream data and the historical frame video stream data are extracted respectively, and the grayscale images and feature points corresponding to the historical frame video stream data are globally aligned, and then the current frame and the globally aligned historical frame images are matched. Specifically, The current frame f t Video stream data and historical frames Convert the video stream data into a grayscale image; Based on the corner detection algorithm in the current frame f t Extract feature point set from ; in, is the i-th feature point of the current frame, and r is the total number of feature points; Historical Frame and its corresponding feature points are globally aligned so that the current frame f t With historical frame Matching is performed in the same coordinate system; in, is the i-th feature point of the historical frame; is the alignment transformation matrix; f t-k is the aligned historical frame, is the i-th feature point of the aligned historical frame; is the current frame f t The history frames after k frames.

3. The method for detecting multimodal anomalies of audio and video streams based on feature tracking according to claim 2, wherein: The audio sampling parameters are set to extract the audio data from the selected audio and video stream data, specifically: The audio sampling parameters include: sampling rate, quantization bit depth, number of channels, and data format; Divide the collected audio data to obtain a number of data segments; The length of the data segment is processed with a 1 - second time window, and each segment contains 16,000 sampling data; The data per second constitutes a time window. Among them, x t Represents the audio sample data vector at the tth second, Represents the value of the p-th sampling point in the t-th second, is a set of integers.

4. The method for detecting multimodal anomalies of audio and video streams based on feature tracking according to claim 3 is characterized in that: Perform frame segmentation and STFT processing on the audio data to obtain an audio spectrum, and extract and convert it into a logarithmic mel - spectrogram with the help of a mel - filter bank. Specifically: Segment the audio data by frames: x n (u)=x(u+nS),0≤u<L,0≤n<N; Here, u is the sample index in each frame, representing the nth sampling point within the frame, where the range is 0 ≤ u < L, and L is the length of each frame; n is the frame number, representing which frame it is currently, where the range is 0 ≤ n < N, and N is the total number of frames; S is the frame shift, representing the interval between the starting points of adjacent frames, in the unit of the number of sampling points; Weighted Hamming window: Here, π is a mathematical symbol; Calculate the STFT for each frame: Here, j is the imaginary unit; Use the mel - filter bank to map the STFT to the mel scale: Among them, H l (q) represents the weight of the lth Mel filter corresponding to the qth frequency point; P represents the number of Mel filters; Then perform a logarithmic transformation, specifically: logS n (l)=log(S n (l)+d); Here, δ is a positive constant, used to avoid the calculation problem of log(0) in logarithmic operations.

5. A multimodal anomaly detection system for audio and video streams based on feature tracking, characterized by: Adopt the audio - video stream multi - modal anomaly detection method based on feature tracking as described in claim 1, including: An acquisition module, which acquires and stores audio - video stream data, and selects the video - stream data of the current frame and any historical - frame video - stream data stored for conversion to obtain a grayscale image; A global alignment module, which extracts feature points from the grayscale images corresponding to the current - frame video - stream data and the historical - frame video - stream data respectively, and globally aligns the grayscale image and feature points corresponding to the historical - frame video - stream data, and then matches the current frame and the globally - aligned historical - frame image; A first judgment module, which extracts local regions from the current frame and the aligned historical - frame image, and judges whether each feature point is an abnormal feature point based on the displacement of feature points and the average grayscale difference of local regions at different times; A second judgment module, which judges whether the selected video - stream data is abnormal based on the relationship between the sum of the number of abnormal feature points, disappeared feature points and newly - added feature points and a preset dynamic threshold; An extraction module, which sets audio sampling parameters and extracts the audio data from the selected audio - video stream data; A conversion module, which performs frame segmentation and STFT processing on the audio data to obtain an audio spectrum, and extracts and converts it into a logarithmic mel - spectrogram with the help of a mel - filter bank; An acquisition module, which obtains the energy - ratio index and the spectral - center difference between the current audio segment and the previous m - second audio segment based on the set time interval m; and obtains the Frobenius - norm difference of the logarithmic mel - spectrogram based on the logarithmic mel - spectrogram; A third judgment module, which respectively compares the energy - ratio index, the spectral - center difference and the Frobenius - norm difference with preset energy - ratio thresholds, spectral - center - difference thresholds and Frobenius - norm - difference thresholds to judge whether the extracted audio is abnormal sound.

Images