Audio and video monitoring and early warning method and system based on multi-modal model driving

Abstract

The invention relates to the technical field of big data, and discloses an audio and video monitoring and early warning method and system based on multi-modal model driving, and the method comprises the steps: S1, collecting audio and video signals in a monitoring scene, and extracting the energy distribution characteristics of the audio and video signals, wherein the energy distribution characteristics at least comprise the frequency energy distribution of the audio signal and the brightness and color change rate of the video signal; s2, multi-modal feature extraction of energy distribution features is carried out based on audio and video signals, and environment background feature vectors are constructed in combination with environment background information; s3, dynamically adjusting anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector; s4, comparing the audio and video signal energy distribution characteristics extracted in real time with an anomaly detection threshold value, and when the audio and video signal energy distribution characteristics exceed the range of the anomaly detection threshold value, determining an abnormal event and obtaining early warning information; and S5, sending the early warning information from the edge equipment to a monitoring center through a low-bandwidth communication protocol.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and particularly to an audio - video monitoring and early warning method and system driven by a multi - modal model. Background Art

[0002] With the rapid development of intelligent monitoring technology, audio - video monitoring systems have been widely used in fields such as public security, traffic management, and industrial production. Traditional audio - video monitoring methods mainly rely on single - modal signal processing or deep - learning models based on content recognition, and achieve anomaly detection through semantic analysis of audio content (such as speech recognition) or video content (such as object detection, behavior analysis). However, the existing technologies have the following deficiencies in practical applications, which limit their popularization and application in complex scenarios.

[0003] First of all, the monitoring methods based on content recognition have high requirements for computing resources and training data. Existing technologies mostly use complex deep - learning models (such as convolutional neural network CNN, recurrent neural network RNN) for audio - video content analysis, which require powerful computing power support and high - quality labeled datasets. This makes it difficult to deploy such methods on edge devices with limited computing power, especially in remote areas or scenarios with limited resources, and the system operation efficiency and real - time performance are significantly affected.

[0004] Secondly, most of the existing anomaly detection mechanisms use fixed thresholds or static rules for judgment, lacking the ability to adapt to environmental changes. In scenarios where conditions such as light intensity and environmental noise level change frequently, fixed thresholds are likely to cause false alarms or missed alarms. For example, when the light is insufficient at night, the brightness change of the video signal may be misjudged as an anomaly; in a high - noise environment, normal audio fluctuations may trigger false warnings, reducing the reliability of the system.

[0005] Thirdly, there is a problem of low resource utilization efficiency in the transmission of early warning information in existing audio - video monitoring systems. Traditional early warning information transmission usually uploads the complete audio - video data to the cloud or the monitoring center, occupying a large amount of network bandwidth. Especially in scenarios with poor network conditions, the latency of information transmission becomes a key factor restricting the real - time performance of the system.

[0006] In recent years, with the development of edge computing and multi-modal signal processing technologies, some research has attempted to offload signal processing tasks to edge devices and perform anomaly detection by combining multi-modal features (such as audio energy distribution and video motion trajectories). For example, an anomaly detection method based on audio energy features realizes the identification of emergencies by analyzing the frequency distribution of audio signals; another example is that a literature explores the application of the brightness change features of video signals in a low-computing-power environment. However, these methods still do not fully consider the impact of the environmental background on the detection threshold, have limited dynamic adaptation capabilities, and lack optimized designs for the transmission efficiency of early warning information and the long-term stability of the system.

[0007] To address the above problems, there is an urgent need for an audio-visual monitoring and early warning method that can operate efficiently on edge devices, has environmental adaptability, and enables low-bandwidth transmission of early warning information, so as to overcome the deficiencies of the existing technologies and ensure the safe and efficient operation of audio-visual monitoring and early warning. Summary of the Invention

[0008] The present invention provides an audio-visual monitoring and early warning method and system based on a multi-modal model drive, aiming to solve the technical problems that existing audio-visual monitoring methods mostly rely on high-computing-power and complex content recognition models and are difficult to meet the deployment requirements of the edge side; moreover, the fixed threshold strategy is prone to false alarms and missed alarms under dynamic environmental changes and lacks environmental adaptability; at the same time, the high-bandwidth transmission of early warning information brings communication bottlenecks and insufficient real-time performance.

[0009] The present invention provides an audio-visual monitoring and early warning method based on a multi-modal model drive, including:

[0010] Collect audio-visual signals in the monitoring scene and extract the energy distribution features of the audio-visual signals, where the energy distribution features at least include the frequency energy distribution of the audio signal and the change rates of the brightness and color of the video signal;

[0011] Perform multi-modal feature extraction of the energy distribution features based on the audio-visual signals and construct an environmental background feature vector in combination with the environmental background information;

[0012] Dynamically adjust the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector;

[0013] Compare the energy distribution features of the audio-visual signals extracted in real time with the anomaly detection thresholds, and when it exceeds the anomaly detection threshold range, determine it as an abnormal event to obtain early warning information;

[0014] Send the early warning information from the edge device to the monitoring center through a low-bandwidth communication protocol.

[0015] In the above solution, through multi-modal feature extraction and environmental background feature vector construction, combined with a dynamic threshold adjustment mechanism, the system can adapt to various complex and changeable monitoring environments, effectively reducing the false alarm rate and missed alarm rate; the dynamic threshold adjustment can adaptively adjust the detection sensitivity according to the environmental background and historical data, enabling the system to maintain good detection performance at different times and in different scenarios. Through the extraction method of energy distribution features, the dependence on high computing power of traditional content recognition methods is avoided, enabling the system to operate efficiently on edge devices; at the same time, the adoption of a low-bandwidth communication protocol ensures efficient data transmission in an edge computing environment. The combined analysis of audio and video signals significantly improves the system's ability to recognize abnormal events. The complementary characteristics of audio and video enable the system to capture more types of abnormal events, avoiding the information dimension limitation of single-modal monitoring. By performing feature extraction and preliminary judgment on edge devices and only transmitting necessary warning information, the system's requirements for network bandwidth and central server computing resources are greatly reduced, realizing the efficient utilization of monitoring resources.

[0016] Preferably, it further includes: regularly self-calibrating the dynamically adjusted abnormal detection threshold based on historical data on the edge device to form a self-calibration log.

[0017] In the above solution, a dynamic threshold self-calibration mechanism is introduced to improve the long-term stability and adaptive ability of the system; the method of the present invention introduces a dynamic threshold self-calibration mechanism to optimize the threshold parameters regularly based on historical data. It is precisely because it can continuously learn and adjust the threshold parameters according to the actual situation of the system operation that the system can continuously adapt to environmental changes and user needs, avoid performance degradation after long-term operation, thereby significantly improving the long-term stability and adaptive ability of the system, extending the service life of the system, and reducing the maintenance cost. The self-calibration mechanism ensures the long-term effectiveness and intelligent level of the system.

[0018] Preferably, collecting the audio and video signals in the monitoring scene and extracting the energy distribution features of the audio and video signals includes:

[0019] Collecting the audio signal using a high-sensitivity microphone and extracting the energy proportion of the audio signal in three frequency bands of low frequency, medium frequency, and high frequency by using fast Fourier transform as the frequency energy distribution feature of the audio signal;

[0020] Collecting the video signal using a high-definition camera and extracting the brightness change rate of each pixel point in the video signal and the motion trajectory feature based on the optical flow method as the change rate feature of the brightness and color of the video signal.

[0021] In the above solution, using a high-sensitivity microphone can ensure the quality of audio signal acquisition and capture weak sound signals. The fast Fourier transform is used to convert the time-domain audio signal to the frequency domain, and the energy ratio of the low-frequency, medium-frequency, and high-frequency bands is extracted, which can effectively characterize the frequency energy distribution characteristics of the audio signal. For example, abnormal sounds such as sudden screams and glass breaking usually cause energy mutations in specific frequency bands. The Fourier transform algorithm has high computational efficiency and meets the real-time processing requirements of edge devices. Using a high-definition camera ensures the clarity of the video signal and provides high-quality video data for subsequent feature extraction. The frame difference method is used to extract the brightness change rate of each pixel point in the video signal, which can effectively detect brightness mutation events in the picture, such as flashes and rapid light changes. The motion trajectory features based on the optical flow method can effectively detect abnormal movements of objects in the picture, such as rapid movement and abnormal trajectories. The frame difference method and the optical flow method have relatively low computational complexity, are suitable for edge device deployment, and can effectively capture the dynamic change information in the video signal as an effective supplement to the features of brightness and color change rates. The extraction methods of the audio and video energy distribution characteristics are specifically defined, providing an efficient, accurate, and edge-device-deployment-suitable energy distribution feature extraction solution, laying a foundation for subsequent anomaly detection based on energy distribution characteristics.

[0022] Preferably, the multi-modal feature extraction of energy distribution characteristics based on audio-visual signals includes:

[0023] Deploy a lightweight multi-modal deep learning model with the MobileNet architecture on the edge device;

[0024] Extract the energy ratio characteristics of low frequency, medium frequency, and high frequency from the audio signal, and extract the intra-frame brightness gradient characteristics and inter-frame color change rate characteristics from the video signal;

[0025] Use the attention mechanism to perform adaptive weight allocation on the energy ratio characteristics of the audio and the brightness gradient characteristics and color change rate characteristics of the video to form a fused feature representation.

[0026] In the above solution, the MobileNet architecture is a lightweight deep learning model with the characteristics of a small number of parameters and low computational complexity. It can effectively reduce the computing power requirements for model deployment and operation, enabling it to run on edge devices. Using a deep learning model for feature extraction can learn deeper and more abstract feature representations compared to traditional handcrafted features, enhancing the representational ability of the features. The specific feature types extracted by the multimodal model are clarified, and these feature types correspond to the energy distribution features, ensuring the consistency and effectiveness of feature extraction. The intra-frame luminance gradient feature can capture the texture and edge information within a video frame, and the inter-frame color change rate feature can capture the dynamic changes in color between video frames, complementing the luminance change rate and motion trajectory features. Using the attention mechanism to adaptively assign weights to audio and video features can dynamically adjust the importance of different modal features according to different scenarios and events. For example, when an audio anomaly event (such as a scream) occurs, the weight of the audio feature may increase; when a video anomaly event (such as a fight) occurs, the weight of the video feature may increase. The attention mechanism can achieve more effective multimodal feature fusion, enhancing the representational ability and robustness of the fused features. By adopting a lightweight multimodal deep learning model, clarifying the feature types, and introducing the attention mechanism, effective multimodal feature extraction and fusion are achieved under the computing power constraints of edge devices, improving the efficiency and effectiveness of feature extraction.

[0027] Preferably, constructing the environmental background feature vector by combining environmental background information includes:

[0028] Using the built-in light sensor and noise sensor of the edge device to collect the environmental light intensity and environmental noise level in real time;

[0029] Normalize the collected environmental light intensity and environmental noise level with the fused features, and combine historical environmental data to construct an environmental background feature vector including timestamp, light intensity, noise level, and season information.

[0030] In the above solution, the ambient light sensor and the noise sensor can obtain the ambient light intensity and the noise level in real time and accurately. These two environmental factors are important factors affecting the performance of the audio-visual monitoring system. Normalizing the ambient light intensity and the noise level with the obtained fusion features ensures the consistency of different types of features in the numerical range, facilitating subsequent feature fusion and threshold calculation. In addition to real-time environmental information, historical environmental data is also combined, such as historical light intensity, noise level, and season information, etc., which can more comprehensively characterize the environmental background state. For example, season information can reflect the long-term change trend of the environment, and historical light intensity and noise level can reflect the daily change rules of the environment. Constructing an environmental background feature vector containing timestamp, light intensity, noise level, and season information can provide richer environmental context information and a more reliable basis for dynamic threshold adjustment. By clarifying the acquisition method, feature type, and vector construction method of the environmental background information, a comprehensive and accurate environmental background feature vector is constructed, providing high-quality environmental context information for dynamic threshold adjustment and improving the accuracy and effectiveness of dynamic threshold adjustment.

[0031] Preferably, the dynamic adjustment of the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector includes:

[0032] Construct an audio anomaly detection threshold calculation formula:

[0033] ThA(t) = μA(E) + α·σA(E)·f(t)

[0034] where ThA(t) is the audio anomaly detection threshold at time t, μA(E) is the mean of the audio energy distribution under the current environmental background feature vector E, σA(E) is the corresponding standard deviation, α is an adjustable coefficient, and f(t) is a time weighting function used to adjust the sensitivity in different time periods;

[0035] Construct a video anomaly detection threshold calculation formula:

[0036] ThV(t) = μV(E) + β·σV(E)·g(L)

[0037] where ThV(t) is the video anomaly detection threshold at time t, μV(E) is the mean of the video brightness and color change rate under the current environmental background feature vector E, σV(E) is the corresponding standard deviation, β is an adjustable coefficient, and g(L) is a light intensity weighting function used to adjust the threshold sensitivity under different light conditions.

[0038] In the above solution, μA(E), σA(E), μV(E), and σV(E) in the formula are all calculated based on the current environmental background feature vector E, reflecting the dynamic correlation between the threshold and the environmental background. By calculating the mean and standard deviation of the audio energy distribution and the video brightness and color change rates in the current environmental background, the signal feature distribution in the current environment can be effectively reflected, providing a benchmark for dynamic threshold adjustment. α and β are adjustable coefficients that allow users to adjust the sensitivity of the threshold according to actual needs. For example, increasing α or β can increase the threshold and reduce the false alarm rate, but may increase the miss rate; decreasing α or β can decrease the threshold and increase the detection rate, but may increase the false alarm rate. The adjustable coefficients provide users with flexible means of threshold adjustment. f(t) is a time-weighting function that can adjust the threshold sensitivity according to the time period. For example, the audio threshold can be appropriately reduced at night to increase the detection rate of abnormal sounds at night; the audio threshold can be appropriately increased during the day to reduce the interference of environmental noise during the day. g(L) is a light intensity-weighting function that can adjust the video threshold sensitivity according to the light intensity. For example, the video threshold can be appropriately reduced in low light conditions to increase the detection rate of abnormal events under low light; the video threshold can be appropriately increased in strong light conditions to avoid false alarms caused by strong light interference. The introduction of the time-weighting function and the light intensity-weighting function further improves the fineness and environmental adaptability of dynamic threshold adjustment. By providing a specific threshold calculation formula, dynamic threshold adjustment based on the environmental background feature vector is realized, enabling the threshold to be adaptively adjusted according to factors such as environmental changes, time periods, and light intensities, improving the accuracy and robustness of anomaly detection.

[0039] Preferably, comparing the energy distribution characteristics of the real-time extracted audio-visual signals with the anomaly detection threshold, when exceeding the anomaly detection threshold range, it is determined as an abnormal event, and the warning information obtained includes:

[0040] Comparing the energy distribution characteristics of the real-time extracted audio frequency with the audio anomaly detection threshold, and comparing the real-time extracted video brightness change rate characteristics with the video anomaly detection threshold;

[0041] When the energy distribution characteristics of the real-time extracted audio frequency exceed the audio anomaly detection threshold range, or the real-time extracted video brightness change rate characteristics exceed the video anomaly detection threshold range, it is determined as an abnormal event;

[0042] Storing the anomaly detection results in the edge device in the form of an energy distribution map and a change rate curve, and generating warning information including key fields and an energy distribution feature map.

[0043] In the above solution, anomaly detection is performed on the audio and video modalities respectively, realizing independent anomaly detection of multiple modalities, and being able to more precisely identify anomaly events in different modalities. The criteria for determining anomaly events are clarified, with clear logic and easy to implement. The setting of the threshold range determines the sensitivity and specificity of the system. The anomaly detection results are stored in the edge device in visual forms such as energy distribution diagrams and change rate curves, facilitating subsequent offline analysis and problem tracing, and at the same time providing data support for user feedback and system optimization. The warning information includes key fields such as anomaly type and anomaly timestamp, ensuring that the warning information can quickly and accurately convey the key information of the anomaly event. At the same time, the inclusion of the energy distribution feature map can intuitively display the characteristics of the anomaly event, providing a basis for the preliminary judgment and analysis by the personnel in the monitoring center. By clarifying the anomaly event determination logic, warning information generation and storage methods, effective anomaly detection based on dynamic thresholds and the generation of warning information rich in content are realized, providing strong support for the monitoring center to respond to anomaly events in a timely manner.

[0044] Preferably, the sending of the warning information from the edge device to the monitoring center through the low-bandwidth communication protocol includes:

[0045] Adopting the MQTT protocol or the LoRaWAN protocol, and sending the warning information to the monitoring center in a compressed format;

[0046] Prioritizing the transmission of the key fields of the warning information, and the key fields include anomaly type, anomaly timestamp, and anomaly confidence;

[0047] Delaying the transmission of the energy distribution feature map to reduce network bandwidth occupancy.

[0048] In the above solution, both the MQTT and LoRaWAN protocols are communication protocols designed specifically for low-bandwidth and unreliable network environments, with characteristics such as lightweight, low power consumption, and efficient transmission, and are very suitable for the communication between the edge device and the monitoring center. Compressing the warning information can effectively reduce the data transmission volume, further reduce the bandwidth requirement, and improve the transmission efficiency. In the case of limited bandwidth, prioritizing the transmission of the key fields of the warning information, such as anomaly type, timestamp, confidence, etc., ensures that the monitoring center can obtain the key information of the anomaly event in the first time and make a timely response. Delaying the transmission of the energy distribution feature map can effectively reduce network bandwidth occupancy and avoid network congestion. By adopting the low-bandwidth communication protocol, information compression, and priority transmission strategy, the fast, reliable, and efficient transmission of the warning information is realized in the scenario of limited network bandwidth, ensuring that the monitoring center can receive the key warning information in a timely manner and providing a guarantee for the quick response to the anomaly event.

[0049] Preferably, the self-calibration of the dynamically adjusted anomaly detection threshold based on historical data on the edge device regularly to form a self-calibration log includes:

[0050] Edge nodes regularly collect historical anomaly detection data and environmental background data;

[0051] Based on the false positive rate and false negative rate in the historical anomaly detection data, construct a threshold self-calibration optimization objective function:

[0052] minJ(θ)=λ1·FPR(θ)+λ2·FNR(θ)+λ3·||θ-θ0|| 2

[0053] where θ represents the set of parameters in the threshold calculation formula, the set of parameters includes α, β; FPR represents the false positive rate, FNR represents the false negative rate, θ0 represents the initial value of the parameter, and λ1, λ2, λ3 are weight coefficients;

[0054] Perform parameter optimization once every 24 hours, and use the stochastic gradient descent method to solve the optimal parameter θ * ;

[0055] Introduce a decay factor γ(0<γ<1), so that the new parameter θ new =γ·θ * +(1-γ)·θ old , ensuring smooth update of the parameters;

[0056] Record the parameter changes and performance metrics of each self-calibration to form a self-calibration log.

[0057] In the above solution, by collecting historical anomaly detection data and environmental background data, it provides a data basis for threshold self-calibration. Construct an optimization objective function based on the false positive rate and false negative rate, and introduce a regularization term ||θ-θ0|| 2 , which clarifies the optimization objective of self-calibration, that is, while reducing the false positive rate and false negative rate, avoiding excessive deviation of the threshold parameter from the initial value and ensuring the stability of the parameter. The weight coefficients λ1, λ2, λ3 can adjust the degree of emphasis on false positives, false negatives, and parameter stability according to actual needs. Using the stochastic gradient descent method to solve the optimal parameter, the algorithm is simple and efficient, and is suitable for the environment with limited resources of edge devices. Introduce a decay factor γ, so that the new parameter is a weighted average between the optimal parameter θ * and the old parameter θ old , ensuring smooth update of the parameters and avoiding system performance instability caused by parameter mutations. Record the parameter changes and performance metrics of each self-calibration, which is convenient for monitoring personnel to understand the self-calibration process and effect, and provides data support for system maintenance and performance evaluation. Through measures such as constructing an optimization objective function, using the stochastic gradient descent method, introducing a decay factor, and forming a self-calibration log, the regular automatic optimization of dynamic threshold parameters is realized, the false positive rate and false negative rate are reduced, and the long-term operation performance and adaptive ability of the system are improved.

[0058] Preferably, an audio - video monitoring and early warning system based on a multi - modal model drive includes:

[0059] A signal acquisition module, configured to acquire audio - video signals in a monitoring scene and extract the energy distribution characteristics of the audio - video signals, where the energy distribution characteristics at least include the frequency - energy distribution of the audio signal and the change rates of the brightness and color of the video signal;

[0060] A feature extraction module, configured to perform multi - modal feature extraction of the energy distribution characteristics based on the audio - video signals and construct an environmental background feature vector in combination with environmental background information;

[0061] A threshold adjustment module, configured to dynamically adjust the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector;

[0062] An anomaly detection module, configured to compare the energy distribution characteristics of the audio - video signals extracted in real - time with the anomaly detection thresholds. When the range of the anomaly detection thresholds is exceeded, it is determined as an abnormal event, and early warning information is obtained;

[0063] An early warning transmission module, configured to send the early warning information from the edge device to the monitoring center through a low - bandwidth communication protocol;

[0064] A self - calibration module, configured to periodically perform self - calibration of the dynamically adjusted anomaly detection thresholds on the edge device based on historical data, and form a self - calibration log.

[0065] In the above solution, the system acquires audio - video signals in the monitoring scene and extracts the energy distribution characteristics therefrom. These characteristics are not traditional content - recognition characteristics, but energy characteristics at the physical level, specifically including the frequency - energy distribution of the audio signal and the change rates of the brightness and color of the video signal. This step focuses on the physical properties of the signal energy change rather than the semantic content. The system performs multi - modal feature extraction based on the acquired audio - video signals and constructs an environmental background feature vector in combination with environmental background information. Multi - modal feature extraction means processing both audio and video signals simultaneously and extracting key features from their respective domains; the environmental background feature vector takes into account the actual environmental factors of the monitoring scene and provides environmental context for subsequent processing. The system dynamically adjusts the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector. This dynamic adjustment mechanism enables the system to adaptively modify the detection sensitivity according to environmental changes instead of using a static fixed threshold. The system compares the energy distribution characteristics of the audio - video signals extracted in real - time with the dynamically adjusted anomaly detection thresholds. When the feature values exceed the threshold range, the system determines it as an abnormal event and generates early warning information. This step realizes the automatic recognition of abnormal events and the triggering of early warnings. The system sends the early warning information from the edge device to the monitoring center through a low - bandwidth communication protocol, ensuring that the information can be transmitted in a timely and efficient manner while reducing network resource occupancy.

[0066] By collecting and analyzing the energy distribution characteristics of audio-visual signals rather than content semantic features, the computational complexity is significantly reduced, enabling the system to operate efficiently on edge devices and breaking through the computing power limitations of traditional content recognition methods. This analysis method based on the physical characteristics of signals has a small amount of calculation but is sensitive to abnormal events, and is particularly suitable for resource-constrained scenarios. Through multi-modal feature extraction and the construction of environmental background feature vectors, the system can comprehensively perceive the signal changes and environmental factors in the monitoring scenario, greatly improving the environmental adaptability and accuracy of anomaly detection. Multi-modal fusion utilizes the complementarity of audio-visual signals and enhances the system's ability to identify various abnormal events. The dynamic threshold adjustment mechanism based on the environmental background feature vector enables the system to adaptively adjust the detection sensitivity according to environmental changes, effectively solving the problem of static threshold failure in complex environments and significantly reducing the false alarm rate and missed alarm rate. This dynamic adaptation mechanism is the key for the system to maintain high accuracy under different environmental conditions. Transmitting warning information using a low-bandwidth communication protocol significantly reduces network resource occupancy and improves communication efficiency, enabling the system to operate reliably in an environment with limited network resources. At the same time, the timely transmission of warning information ensures that the monitoring center can quickly respond to abnormal events, improving the real-time performance and effectiveness of the entire monitoring system. The system can automatically optimize the calculation parameters of the anomaly detection threshold based on historical data, reducing manual intervention and improving the adaptability and accuracy of the system during long-term operation. This self-learning ability enables the system to continuously optimize its performance according to the characteristics of the actual monitoring scenario. The log formed during the self-calibration process records the historical trajectory of parameter adjustment, providing traceability and transparency of system behavior, facilitating subsequent maintenance and fault troubleshooting. These historical data can also be used for system performance evaluation and further optimization. The self-calibration mechanism on the edge device reduces the communication requirements with the central server, enabling the system to operate and optimize independently in an environment with limited network resources, improving the independence and stability of the system. Through regular self-adjustment, the system can always maintain the best performance state and effectively cope with seasonal changes and long-term trend changes in the environment.

[0067] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0068] The present invention discloses an audio-visual monitoring and warning method and system driven by a multi-modal model, which can perceive and identify abnormal events in real time and accurately in an edge computing environment. By fusing multi-modal signals and environmental background information, dynamically adjusting the anomaly detection threshold, reducing false alarms and missed alarms, and enhancing the robustness and adaptability of the monitoring system; and through lightweight model design and low-bandwidth communication protocol, achieving efficient operation and timely transmission of warning information under edge computing power limitations; wherein:

[0069] (1) By comprehensively extracting the energy distribution of audio frequencies, the brightness of video, and the change rate of color, the rich physical information of the monitoring scene is captured, reducing the limitations of single-signal analysis and improving the detection coverage of abnormal events;

[0070] (2) Combining real-time environmental illumination and noise data to construct an environmental background feature vector, realizing the adjustment of the abnormal detection threshold based on the dynamic environment, greatly enhancing the system's adaptability to complex dynamic scenes, and avoiding false positives caused by environmental factors;

[0071] (3) Through the combination and adjustment of the weight parameters in the threshold relationship formula, the system can accurately distinguish critical abnormalities according to historical means and smooth environmental changes, significantly improving the accuracy and sensitivity of abnormal detection;

[0072] (4) Comparing real-time features with dynamic thresholds to quickly capture abnormal signals and enhancing the real-time response ability of early warning;

[0073] (5) The early warning information preferentially transmits summary data, reducing the network bandwidth pressure, realizing efficient edge-center linkage, and improving the overall operation efficiency of the monitoring system. Description of the Drawings

[0074] Figure 1 is a schematic flow chart of an audio-visual monitoring and early warning method based on a multi-modal model drive provided by an embodiment of the present invention;

[0075] Figure 2 is a schematic diagram of the modules of an audio-visual monitoring and early warning system based on a multi-modal model drive provided by an embodiment of the present invention. Detailed Embodiments

[0076] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0077] As Figure 1 shown, the present application provides an audio-visual monitoring and early warning method based on a multi-modal model drive, including:

[0078] S1: Collect audio-visual signals in the monitoring scene and extract the energy distribution characteristics of the audio-visual signals, where the energy distribution characteristics at least include the frequency energy distribution of the audio signal and the change rate of the brightness and color of the video signal;

[0079] S2: Extract multi-modal features of energy distribution characteristics based on audio-visual signals, and construct an environmental background feature vector by combining environmental background information;

[0080] S3: Dynamically adjust the anomaly detection threshold of audio-visual signals based on the environmental background feature vector and historical signal data;

[0081] S4: Compare the energy distribution characteristics of the real-time extracted audio-visual signals with the anomaly detection threshold. When it exceeds the anomaly detection threshold range, it is determined as an abnormal event to obtain a warning message;

[0082] S5: Send the warning message from the edge device to the monitoring center through a low-bandwidth communication protocol.

[0083] In an embodiment provided by the present application, audio-visual acquisition devices are deployed in a monitoring scenario, including high-sensitivity microphones and high-definition cameras, for real-time acquisition of audio signals and video signals in the environment. Among them, the audio signals include environmental noise, sudden sounds, etc., and the video signals include changes in picture brightness, object movement, etc. To ensure the integrity and quality of the signals, the audio acquisition device supports wide-band audio sampling of 20 Hz - 20 kHz, and the video acquisition device supports high-resolution video recording of 1080p and above.

[0084] The acquired audio-visual signals are preliminarily preprocessed on the edge device. Specifically, frequency energy distribution characteristics are extracted from the audio signals through fast Fourier transform (FFT), and the brightness change rate and color change rate characteristics are extracted from the video signals through the inter-frame difference method. The purpose of the preprocessing process is to reduce signal noise interference and improve the accuracy of subsequent analysis. Through high-precision signal acquisition and preprocessing, it is ensured that the original characteristics of the audio-visual signals are retained, providing a reliable data basis for subsequent anomaly detection based on energy distribution.

[0085] Use a pre-trained multi-modal deep learning model to extract features from audio-visual signals on the edge device.

[0086] For audio signals, extract frequency energy distribution characteristics, including the energy ratios of low frequency (20 Hz - 200 Hz), medium frequency (200 Hz - 2 kHz), and high frequency (2 kHz - 20 kHz). The model generates an audio energy feature vector by analyzing the energy distribution in each frequency band of the spectrum map.

[0087] For video signals, extract the picture brightness change rate and color change rate characteristics. Specifically, by calculating the brightness difference and color difference between adjacent frames, a video change rate feature vector is generated.

[0088] The multi-modal deep learning model adopts a lightweight design, such as the MobileNet architecture, to adapt to the limited computing power of edge devices. The model extracts the time-frequency features of audio-visual signals through convolutional layers and pooling layers, and fuses multi-modal features through fully connected layers to output a unified feature representation.

[0089] Meanwhile, the environmental background data, including the changes in day-night light and the environmental noise level, is collected in real time through the environmental sensors (such as light sensors and noise sensors) built in the edge device. Combining historical data, an environmental background feature vector is constructed for dynamically adjusting the subsequent detection threshold.

[0090] The environmental background feature vector includes the following components:

[0091] Environmental noise level: the average noise decibel value of the current environment

[0092] Environmental light intensity: the light intensity value of the current environment

[0093] Time factor: representing the time period of a day (such as morning, noon, evening, night)

[0094] Scene type: representing the type of the monitoring scene (such as indoor, outdoor, crowded area, etc.)

[0095] Combining multi-modal feature extraction with environmental background perception can comprehensively represent the physical characteristics of audio-visual signals, avoid the dependence of traditional content recognition methods on complex scenes, and improve the robustness of the system in diverse environments.

[0096] Based on the environmental background feature vector and historical signal data, the anomaly detection threshold of the audio-visual signal is calculated dynamically.

[0097] Define the energy threshold relation of the audio signal as:

[0098] T audio =τ·E avg +ω·N env

[0099] Where, T audio is the dynamic threshold of audio energy, E avg is the historical average audio energy, N env is the current environmental noise level, and τ and ω are weighting coefficients with initial values of 0.8 and 0.2 respectively, which can be adjusted through self-calibration.

[0100] Similarly, the threshold relation of the brightness change rate of the video signal is:

[0101]

[0102] Where, T video is the dynamic threshold of the video brightness change rate, Lavg is the mean historical brightness change rate, I env is the current ambient light intensity, δ and are the weighting coefficients, with initial values of 0.7 and 0.3 respectively.

[0103] When the audio energy or video brightness change rate extracted in real time exceeds the corresponding dynamic threshold range, it is determined as an abnormal event. The abnormal detection results are stored in the edge device in the form of feature maps (including energy distribution maps and change rate curves).

[0104] The above threshold relationship realizes the dynamic adaptability of the threshold by introducing environmental variables (such as N env and I env ) and the mean historical data (such as E avg and L avg ), avoiding false alarms or missed alarms of fixed thresholds in different environments. The adjustability of the weighting coefficients further enhances the flexibility of the system. The dynamic threshold adjustment mechanism significantly improves the accuracy of abnormal detection, especially in scenarios with large changes in light or noise, reducing the false alarm rate and missed alarm rate.

[0105] When an abnormal event is detected, the edge device generates a warning message through the built-in agent module. The warning message includes the abnormal type (audio abnormality or video abnormality), abnormal timestamp, and a brief energy distribution feature map. The agent module uploads the warning message to the monitoring center through a low-bandwidth communication protocol (such as MQTT), and at the same time stores the complete data locally for subsequent analysis. The feature map summary is preferentially transmitted during the upload process to reduce network bandwidth occupancy.

[0106] Specifically, the transmission of the warning message adopts a hierarchical strategy:

[0107] The first level: Only transmit the abnormal type and timestamp (data volume < 1KB)

[0108] The second level: Transmit a brief feature map summary (data volume < 10KB)

[0109] The third level: Transmit the complete feature data according to the request of the monitoring center (data volume < 100KB)

[0110] Through the agent module and the low-bandwidth communication protocol, the rapid transmission of warning messages and the efficient utilization of resources are realized, ensuring that the monitoring center can respond to abnormal events in a timely manner.

[0111] Preferably, it further includes: regularly self-calibrating the dynamically adjusted abnormal detection threshold on the edge device based on historical data.

[0112] In an embodiment provided by the present application, the edge node periodically (e.g., every 24 hours) performs self-calibration on the weighting coefficients (such as τ, ω, δ, ) in the dynamic threshold relationship based on historical anomaly detection data and environmental background data.

[0113] The self-calibration process is as follows:

[0114] Collect anomaly detection results and environmental data within the past 24 hours;

[0115] Calculate the accuracy rate of anomaly detection (based on user feedback or system self-verification);

[0116] Use a simple linear regression method to adjust the weighting coefficients to maximize the accuracy rate;

[0117] Update the weighting coefficients in the threshold relationship;

[0118] The system also supports the user to manually fine-tune the threshold parameters or feedback false alarm / missed alarm events through the monitoring center interface. The feedback data will be used as supplementary input for self-calibration to further optimize the system performance. The combination of the self-calibration mechanism and the user feedback mechanism enables the system to continuously adapt to environmental changes and user needs, improving the stability and accuracy of long-term operation.

[0119] Preferably, in step S1, collecting the audio-visual signals in the monitoring scenario and extracting the energy distribution characteristics of the audio-visual signals includes:

[0120] Collect audio signals using a high-sensitivity microphone, and use the fast Fourier transform to extract the energy ratios of the audio signals in the low-frequency, medium-frequency, and high-frequency bands as the frequency energy distribution characteristics of the audio signals;

[0121] Collect video signals using a high-definition camera, and use the inter-frame difference method to extract the brightness change rate of each pixel point in the video signal, as well as the motion trajectory characteristics based on the optical flow method, as the change rate characteristics of the brightness and color of the video signal.

[0122] In an embodiment provided by the present application, first, audio-visual acquisition devices are deployed in the monitoring scenario. Specifically, a high-sensitivity microphone (such as a capacitive microphone with a sensitivity of not less than -40 dB) is used to collect audio signals. This microphone supports a wide frequency band sampling range (for example, 20 Hz to 20 kHz) to capture various sounds in the environment, including ambient noise, sudden sounds, etc. At the same time, a high-definition camera (such as a CMOS sensor camera supporting a resolution of 1080p or above) is deployed to collect video signals in real time, including changes in picture brightness and object motion information. The deployment location of the acquisition device should consider the characteristics of the monitoring scenario. For example, in an industrial site or a public area, ensure that the device covers key areas and avoids signal interference. The acquisition process is carried out in real time mode, and the sampling rates are set to 44.1 kHz for audio signals and 30 frames per second for video signals to ensure the timing integrity and data continuity of the signals.

[0123] After the original audio-visual signals are collected, feature extraction is performed to obtain energy distribution features. Specifically, for audio signals, the fast Fourier transform (FFT) algorithm is used to perform spectral analysis on the collected audio data. FFT is an efficient frequency domain conversion method that calculates the energy distribution of audio signals in different frequency bands by converting the time domain signal to the frequency domain. In a specific implementation, the audio spectrum is divided into three frequency bands: low frequency (20 Hz - 300 Hz), middle frequency (300 Hz - 3000 Hz), and high frequency (3000 Hz - 20 kHz). The energy proportion of each frequency band is calculated respectively. For example, the energy value is obtained by integrating the spectral density and normalized. This energy proportion feature can quantify the spectral characteristics of audio signals. For example, sudden low-frequency noise may indicate a mechanical failure, while high-frequency components may correspond to sharp sound events, thus providing basic data for anomaly detection.

[0124] For video signals, the inter-frame difference method is used to extract the luminance change rate feature. Specifically, the inter-frame difference method calculates the luminance change rate by comparing the pixel value differences between adjacent video frames. The implementation process includes: First, convert the video frames into grayscale images to simplify the calculation; then, perform pixel-level difference operations on each pair of adjacent frames to calculate the luminance difference of each pixel point. For example, use the absolute difference formula |I(t)-I(t-1)|, where I(t) represents the pixel intensity at time t; subsequently, count the change rate in the difference image, for example, calculate the proportion of changed pixels in the total pixels after thresholding as the luminance change rate feature. In addition, combined with the extraction of motion trajectory features based on the optical flow method, use algorithms such as the Farneback optical flow algorithm to calculate the optical flow field between video frames, and then infer the object motion trajectory and speed change. The optical flow method is based on the optical flow constraint equation to estimate the motion vector of pixel points between consecutive frames, thereby obtaining color and motion-related features, such as the length and direction of the motion trajectory, as the color change rate feature of the video signal. The color change rate can be indirectly represented by the hue and saturation differences in the HSV color space.

[0125] In the above feature extraction process, considering the computing resource limitations of edge devices, lightweight algorithms are used in the preprocessing steps. For example, in audio feature extraction, only calculate the energy proportion of key frequency bands instead of full-spectrum analysis; in video feature extraction, use simplified versions of inter-frame difference and optical flow calculations to reduce the computational complexity. The extracted energy distribution features are stored in vector form. For example, the audio feature vector is [low-frequency energy proportion, mid-frequency energy proportion, high-frequency energy proportion], and the video feature vector is [luminance change rate, motion trajectory speed, motion trajectory direction], which is convenient for subsequent module calls.

[0126] This embodiment ensures that the energy distribution features of audio and video signals accurately reflect the dynamic changes of the monitoring scene through high-precision signal acquisition and feature extraction based on FFT and the inter-frame difference method, avoiding the computationally intensive problems of traditional methods such as pixel-level content recognition, thereby reducing the system's dependence on high computing power and improving real-time performance and robustness. Specifically, the extraction of audio energy distribution features can effectively identify spectrum anomalies, such as detecting emergencies in a noisy environment; the extraction of video luminance and color change rate features enhances the adaptability to light changes and reduces the false detection rate. According to experimental verification, in a typical monitoring scene, this method improves the accuracy of anomaly detection by about 15% and ensures the integrity of feature data, enabling subsequent multimodal processing modules to perform efficient analysis based on reliable inputs, thus meeting the requirements of the overall technical solution of the present invention.

[0127] Preferably, in step S2, the multimodal feature extraction of energy distribution features based on audio and video signals includes:

[0128] Deploy a lightweight multi-modal deep learning model using the MobileNet architecture on edge devices;

[0129] Extract the energy proportion features of low frequency, medium frequency and high frequency from the audio signal, and extract the in-frame luminance gradient feature and the inter-frame color change rate feature from the video signal;

[0130] Use the attention mechanism to perform adaptive weight allocation on the energy proportion features of the audio and the luminance gradient features and color change rate features of the video to form a fused feature representation.

[0131] Preferably, constructing the environmental background feature vector by combining environmental background information includes:

[0132] Use the built-in light sensor and noise sensor of the edge device to collect the environmental light intensity and environmental noise level in real time;

[0133] Normalize the collected environmental light intensity and environmental noise level with the fused features, and combine the historical environmental data to construct an environmental background feature vector including timestamp, light intensity, noise level and season information.

[0134] In an embodiment provided by the present application, in order to adapt to the limited computing resources of edge devices and ensure real-time performance, this embodiment uses a lightweight multi-modal deep learning model based on the MobileNet architecture. This model is deployed on edge devices to further extract high-level semantic features from the energy distribution features of audio-visual signals. The model input is the frequency energy distribution feature of the audio signal and the luminance and color change rate features of the video signal.

[0135] For the audio signal, the model is configured to extract features in three dimensions: low-frequency energy proportion, medium-frequency energy proportion, and high-frequency energy proportion. Specifically, the result of the fast Fourier transform of the audio signal is divided into three frequency bands: low-frequency band (e.g., 20Hz - 500Hz), medium-frequency band (e.g., 500Hz - 4kHz), and high-frequency band (e.g., 4kHz - 20kHz). The model calculates the total energy of each frequency band and divides it by the total energy of the audio signal to obtain the low-frequency energy proportion, medium-frequency energy proportion, and high-frequency energy proportion. These energy proportion features can effectively characterize the frequency composition of the audio signal. For example, sudden screams are often accompanied by a significant increase in the high-frequency energy proportion, while low-pitched impacts may be manifested as an increase in the low-frequency energy proportion.

[0136] For video signals, the model is configured to extract intra-frame luminance gradient features and inter-frame color change rate features. The intra-frame luminance gradient features are obtained by calculating the luminance differences between adjacent pixel points within a video frame. For example, gradient operators such as the Sobel operator can be used to convolve each frame image to obtain luminance gradient maps in the horizontal and vertical directions. These gradient maps can reflect the edge and texture information in the image, thereby characterizing the complexity and degree of change of the picture content. The inter-frame color change rate features are obtained by comparing the color differences of corresponding pixel points between adjacent video frames. For example, the sum of the absolute values of the RGB value differences of each pixel point between adjacent frames can be calculated and used as the color change rate of that pixel point. The color change rates of all pixel points are averaged to obtain the inter-frame color change rate feature. This feature can effectively capture sudden color changes in the video picture. For example, abnormal events such as fire or flash often lead to a significant increase in the color change rate.

[0137] To more effectively fuse the features of the audio and video modalities and highlight the contribution of important features to anomaly detection, this embodiment introduces an attention mechanism into the multi-modal deep learning model. The attention mechanism is configured to perform adaptive weight allocation on the energy ratio feature of the audio and the luminance gradient feature and color change rate feature of the video. For example, in a situation with low environmental noise, the abnormal features of the audio modality may be more reliable, and the attention mechanism will assign a higher weight to the audio features; while in a scene with drastic light changes, the features of the video modality may be more sensitive, and the attention mechanism will increase the weight of the video features. Through this adaptive weight allocation, the model can more flexibly utilize multi-modal information to form a more discriminative fused feature representation, thereby improving the robustness of the system in complex environments.

[0138] To enable the anomaly detection threshold to be dynamically adjusted according to environmental changes, this embodiment constructs an environmental background feature vector in combination with environmental background information. An illumination sensor and a noise sensor are integrated on the edge device for real-time acquisition of the environmental illumination intensity and environmental noise level. The illumination sensor can use components such as a photoresistor or a photodiode to convert the environmental light intensity into an electrical signal for output; the noise sensor can use devices such as a MEMS microphone to convert the environmental sound intensity into an electrical signal for output. The collected illumination intensity and noise level data are converted through analog-to-digital conversion and used as the original input of the environmental background information.

[0139] To eliminate the dimension difference and effectively integrate the real-time environmental data with the fused features, this embodiment normalizes the collected environmental illumination intensity and environmental noise level. For example, the illumination intensity data can be mapped to the range of 0-1, and the noise level data is also normalized similarly. The normalized environmental data and the multi-modal fused features are jointly input into the subsequent dynamic threshold adjustment module.

[0140] To more comprehensively describe the environmental background and consider the regularity of environmental factors changing over time, this embodiment further combines historical environmental data to construct an environmental background feature vector that includes a timestamp, light intensity, noise level, and season information. The timestamp information is used to record the current moment, and the season information can be preset according to the current date and geographical location or obtained through a network service. The historical environmental data can be stored in the local memory of the edge device and updated regularly. When constructing the environmental background feature vector, historical environmental data similar to the current timestamp and season information can be retrieved from the historical database. For example, the average value and standard deviation of the light intensity and noise level during the same period in the past week. These historical data can reflect the long-term change trend of the environmental background and assist the system in more accurately determining whether the current abnormal event is truly abnormal. Finally, the timestamp, normalized real-time light intensity, normalized real-time noise level, season information, and the retrieved historical environmental data are combined to form an environmental background feature vector, providing comprehensive environmental context information for subsequent dynamic threshold adjustment.

[0141] This embodiment realizes the effective extraction of the signal energy distribution characteristics based on a multimodal model and constructs an environmental background feature vector by combining environmental background information. The technical effects are mainly reflected in the following aspects:

[0142] Effective fusion of multimodal features: By using a multimodal deep learning model and introducing an attention mechanism, the effective fusion and adaptive weighting of audio and video modal features are realized, enhancing the discriminability of feature representation and laying a foundation for the accuracy of subsequent anomaly detection.

[0143] Effective integration of environmental background information: By collecting and processing environmental background information such as environmental light intensity and noise level and combining historical data to construct an environmental background feature vector, reliable environmental context information is provided for dynamic threshold adjustment, enabling the system to adaptively adjust the detection sensitivity according to environmental changes.

[0144] Lightweight design and adaptation to edge computing: By adopting a lightweight model with the MobileNet architecture, the computational complexity of the model is reduced, enabling it to be deployed on resource-constrained edge devices, realizing real-time feature extraction and environmental perception, and meeting the real-time requirements of the audio and video monitoring and warning system.

[0145] Improving system robustness: The extraction of multimodal features and the integration of environmental background information enable the system to understand the monitoring scenario from multiple dimensions, reduce misjudgments that may be caused by single-modal information, and significantly improve the robustness and reliability of the system in complex environments.

[0146] Preferably, in step S3, the dynamic adjustment of the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector includes:

[0147] Construct an audio anomaly detection threshold calculation formula:

[0148] ThA(t) = μA(E) + α·σA(E)·f(t)

[0149] where ThA(t) is the audio anomaly detection threshold at time t, μA(E) is the mean of the audio energy distribution under the current environmental background feature vector E, σA(E) is the corresponding standard deviation, α is an adjustable coefficient, and f(t) is a time weighting function used to adjust the sensitivity in different time periods;

[0150] Construct a video anomaly detection threshold calculation formula:

[0151] ThV(t) = μV(E) + β·σV(E)·g(L)

[0152] where ThV(t) is the video anomaly detection threshold at time t, μV(E) is the mean of the video brightness and color change rate under the current environmental background feature vector E, σV(E) is the corresponding standard deviation, β is an adjustable coefficient, and g(L) is a light intensity weighting function used to adjust the threshold sensitivity under different lighting conditions.

[0153] In an embodiment provided by the present application, the dynamic adjustment of the audio anomaly detection threshold is implemented based on the following formula:

[0154] ThA(t) = μA(E) + α·σA(E)·f(t)

[0155] where ThA(t) represents the audio anomaly detection threshold at time t, which is the critical value for determining whether the audio signal energy is abnormal at the current moment; μA(E) represents the mean of the historical audio energy distribution under the current environmental background feature vector E, which reflects the typical level of audio energy in a similar environmental background; σA(E) represents the standard deviation of the historical audio energy distribution corresponding to the current environmental background feature vector E, which reflects the fluctuation range of audio energy in a specific environmental background; α is an adjustable coefficient used to adjust the sensitivity of the threshold to environmental changes, and the larger the value of α, the stronger the response of the threshold to environmental changes; f(t) is a time weighting function used to adjust the threshold sensitivity according to time changes. For example, in periods such as at night when higher sensitivity is required, the value of f(t) can be appropriately increased.

[0156] In practical applications, the environmental background feature vector E includes dimensions such as light intensity, environmental noise level, timestamp, and season information. When the environmental background changes, for example, when the environmental noise level increases, the system will update the environmental background feature vector E in real time. Based on the updated E, the system will recalculate μA(E) and σA(E), which are usually statistically obtained from historical data. For example, the mean and standard deviation of the audio energy distribution corresponding to different environmental background feature vectors E can be pre-stored. When the environmental noise level increases, the mean μA(E) of the historical audio energy distribution will also increase accordingly because the environmental noise itself contributes to the audio energy. At the same time, the standard deviation σA(E) may also change, depending on the stability of the noise. Therefore, according to the formula, the audio anomaly detection threshold ThA(t) will be adaptively adjusted, usually increased, to adapt to the higher background noise level and reduce false alarms caused by environmental noise.

[0157] For example, assume that during the day when the environmental noise is low, through statistical analysis of historical data, under the corresponding environmental background feature vector E-day, the mean μA(E-day) of the audio energy distribution is 30 decibels (dB), the standard deviation σA(E-day) is 5 dB, the adjustable coefficient α is set to 2, and the time-weighting function f(t) is set to 1 during the day. Then the audio anomaly detection threshold ThA(day) during the day is calculated as follows:

[0158] ThA(day) = 30 dB + 2·5 dB·1 = 40 dB

[0159] When the environment enters the night and the noise level significantly decreases, the corresponding environmental background feature vector becomes E-night. Through statistical analysis of historical data, under E-night, the mean μA(E-night) of the audio energy distribution decreases to 25 dB, the standard deviation σA(E-night) decreases to 3 dB, and the time-weighting function f(t) is set to 1.2 at night to improve the monitoring sensitivity at night. Then the audio anomaly detection threshold ThA(night) at night is calculated as follows:

[0160] ThA(night) = 25 dB + 2·3 dB·1.2 = 32.2 dB

[0161] By comparing the audio anomaly detection thresholds during the day and at night, it can be seen that the dynamic threshold adjustment mechanism can adaptively adjust the audio anomaly detection threshold according to the environmental noise level and time changes. At night when the environmental noise is low, the threshold decreases, and the system is more sensitive to abnormal sound events; during the day when the environmental noise is high, the threshold increases, which can effectively suppress false alarms caused by environmental noise.

[0162] The dynamic adjustment of the video anomaly detection threshold is implemented based on the following formula:

[0163] ThV(t)=μV(E)+β·σV(E)·g(L)

[0164] Among them, ThV(t) represents the video anomaly detection threshold at time t, which is used to determine whether the video brightness or color change rate at the current moment is abnormal; μV(E) represents the mean of the historical video brightness and color change rate under the current environmental background feature vector E, reflecting the typical level of video change under similar environmental backgrounds; σV(E) represents the standard deviation of the historical video brightness and color change rate corresponding to the current environmental background feature vector E, reflecting the fluctuation range of video changes under a specific environmental background; β is an adjustable coefficient used to adjust the sensitivity of the threshold to environmental changes. The larger the β value, the stronger the threshold response to environmental changes; g(L) is a light intensity weighting function used to adjust the threshold sensitivity according to the light intensity. For example, at night or on cloudy days with low light intensity, the value of g(L) can be appropriately reduced to increase the system's sensitivity to video brightness or color changes.

[0165] Similar to the audio threshold adjustment, when the ambient light intensity changes, the ambient background feature vector E will also be updated in real time. Based on the updated E, the system will recalculate μV(E) and σV(E). Changes in light intensity will directly affect the brightness and color of the video image. Therefore, μV(E) and σV(E) will be adaptively adjusted as the light intensity changes. Through the light intensity weighting function g(L), the video anomaly detection threshold can be further refined according to the lighting conditions.

[0166] For example, assuming that there is sufficient sunlight during the day, through historical data statistics, under the corresponding environmental background feature vector E-sunny, the mean value of the video brightness change rate μV(E-sunny) is 10 gray levels / frame, the standard deviation σV(E-sunny) is 2 gray levels / frame, the adjustable coefficient β is set to 1.5, and the light intensity weighting function g(L) is set to 1 when the light is sufficient. Then the daytime video anomaly detection threshold ThV(sunny) is calculated as follows:

[0167] ThV(sunny)=10 gray levels / frame+1.5·2 gray levels / frame·1=13 gray levels / frame

[0168] When the environment turns cloudy, the light intensity decreases, and the corresponding environmental background feature vector becomes E-cloudy. According to historical data statistics, under E-cloudy, the mean value of the video brightness change rate μV(E-cloudy) is reduced to 8 gray levels / frame, and the standard deviation σV(E-cloudy) is reduced to 1.5 gray levels / frame. The light intensity weighting function g(L) is set to 0.8 when the light is weak to improve the sensitivity under weak light conditions. The cloudy video anomaly detection threshold ThV(cloudy) is calculated as follows:

[0169] ThV(cloudy) = 8 gray levels / frame + 1.5 · 1.5 gray levels / frame · 0.8 = 9.8 gray levels / frame

[0170] Similarly, by comparing the video anomaly detection thresholds for sunny and cloudy days, it can be seen that the dynamic threshold adjustment mechanism can adaptively adjust the video anomaly detection threshold according to the change in light intensity. On cloudy days with weak light, the threshold decreases, and the system is more sensitive to changes in video brightness or color. Even minor changes may be detected. On sunny days with sufficient light, the threshold increases, which can avoid false alarms caused by light changes. For example, it can prevent misjudging normal light and shadow changes as abnormal events.

[0171] Through the above dynamic threshold adjustment method, the anomaly detection thresholds of audio and video signals can be adjusted in real time according to the environmental background feature vector. The technical effects brought by this dynamic adjustment mechanism are mainly reflected in the following aspects:

[0172] Improve environmental adaptability: The system can adapt to the influence of environmental factors such as light intensity, environmental noise level, and time change, and maintain high detection performance in various complex monitoring scenarios.

[0173] Reduce false alarm rate and missed alarm rate: The dynamic threshold can effectively distinguish the normal signal fluctuations caused by environmental changes from real abnormal events, reducing false alarms caused by environmental factors. At the same time, by reducing the threshold in environments that require high sensitivity, the detection rate of potential abnormal events is improved, and the risk of missed alarms is reduced.

[0174] Improve detection accuracy: The dynamic threshold makes anomaly detection more accurate, and can more accurately identify real abnormal events, providing a reliable basis for subsequent early warning and response.

[0175] In summary, this embodiment details the specific implementation method of dynamically adjusting the anomaly detection thresholds of audio and video signals based on the environmental background feature vector, clearly demonstrating the technical solution and advantages of the present invention in dynamic threshold adjustment.

[0176] Preferably, in step S4, when comparing the energy distribution characteristics of the real-time extracted audio and video signals with the anomaly detection threshold, and when it exceeds the anomaly detection threshold range, it is determined as an abnormal event, and the obtained warning information includes:

[0177] Comparing the energy distribution characteristics of the real-time extracted audio frequency with the anomaly detection threshold of the audio, and comparing the real-time extracted video brightness change rate characteristics with the anomaly detection threshold of the video;

[0178] When the characteristics of the audio frequency energy distribution extracted in real time exceed the abnormal detection threshold range of the audio, or the characteristics of the video brightness change rate extracted in real time exceed the abnormal detection threshold range of the video, it is determined as an abnormal event;

[0179] The abnormal detection results are stored in the edge device in the form of an energy distribution diagram and a change rate curve, and a warning message including keyword fields and an energy distribution feature map is generated.

[0180] In an embodiment provided by the present application, on the edge device, the system first obtains the characteristics of the audio-visual signal energy distribution extracted in real time. These characteristics include the frequency energy distribution of the audio signal (for example, the proportion of low-frequency, medium-frequency, and high-frequency energy) and the brightness change rate of the video signal (for example, the brightness change rate of pixel points calculated by the inter-frame difference method). At the same time, the dynamic abnormal detection thresholds calculated in step S3 are called, including the audio threshold ThA(t) and the video threshold ThV(t).

[0181] For the audio signal, the system compares the audio energy distribution characteristics EA(t) (for example, the energy proportion vector) extracted in real time with the audio abnormal detection threshold ThA(t) element by element. ThA(t) is calculated based on the environmental background feature vector E, including the mean μA(E) and the standard deviation σA(E).

[0182] For the video signal, the system compares the video brightness change rate characteristics EV(t) (for example, the average brightness change rate or the color change rate) extracted in real time with the video abnormal detection threshold ThV(t). ThV(t) is also calculated based on E, including the mean μV(E) and the standard deviation σV(E).

[0183] The comparison process uses numerical calculation methods. For example, the difference between the feature value and the threshold is calculated through subtraction operations, and it is judged whether the difference exceeds the predefined threshold range (for example, ThA(t) ± k·σA(E), where k is a system parameter, usually set to 2 or 3 to cover 95% of the normal distribution range).

[0184] If the audio feature EA(t) exceeds the range of ThA(t), or the video feature EV(t) exceeds the range of ThV(t), further abnormal analysis is triggered.

[0185] To quantify the degree of abnormality, the system calculates the deviation ratio between the real-time feature and the threshold.

[0186] Calculate the audio deviation ratio:

[0187] RA = |EA(t) - μA(E)| / σA(E),

[0188] Where EA(t) is the real-time audio energy distribution feature, and μA(E) and σA(E) are the mean and standard deviation based on the environmental background feature vector E.

[0189] Calculate the video deviation ratio:

[0190] RV = |EV(t) - μV(E)| / σV(E),

[0191] Where EV(t) is the real-time video brightness change rate feature, and μV(E) and σV(E) are similarly obtained based on E. The deviation ratios RA and RV are calculated using standard statistical methods to ensure numerical stability and comparability. For example, σA(E) and σV(E) can be estimated by the standard deviation of historical data or real-time samples.

[0192] When RA > α or RV > β (where α and β are adjustable coefficients defined in step S3, usually with an initial value of 2, indicating that exceeding 2 standard deviations is considered abnormal), the system determines it as an abnormal event. The values of α and β can be dynamically adjusted according to the environmental background to adapt to the noise levels of different scenarios.

[0193] In addition, the system considers the duration of the anomaly. For example, if the deviation ratio continuously exceeds a preset time threshold (e.g., 5 seconds), then the abnormal event is confirmed to reduce false positives caused by transient noise.

[0194] Based on the magnitudes of the calculated deviation ratios RA and RV and the duration of the anomaly, the system classifies the abnormal events.

[0195] Mild anomaly (Level 1): When RA or RV is in the range of 1.5 to 2.0 and the duration is less than 10 seconds, it indicates that the anomaly is relatively mild and may be normal fluctuations.

[0196] Moderate anomaly (Level 2): When RA or RV is in the range of 2.0 to 3.0, or the duration is between 10 and 30 seconds, it indicates that the anomaly is moderate and may involve potential risks.

[0197] Severe anomaly (Level 3): When RA or RV is greater than 3.0, or the duration exceeds 30 seconds, it indicates that the anomaly is severe and may be an emergency event.

[0198] The classification logic is implemented using simple conditional judgment rules, such as based on preset ratio thresholds and timers. Subsequently, the system generates structured warning information, including the following key fields:

[0199] Anomaly type (e.g., "audio anomaly" or "video anomaly");

[0200] Anomaly level (Level 1, Level 2, or Level 3);

[0201] Anomaly timestamp (accurate to milliseconds, based on the system clock);

[0202] Deviation ratio (specific values of RA and RV);

[0203] Energy distribution characteristic diagram (e.g., bar chart of audio energy proportion or curve graph of video brightness change rate), which graphically represents the cause of anomalies.

[0204] The characteristic diagram is generated using general drawing techniques, such as creating image files using functions based on libraries (such as Matplotlib or similar tools) on edge devices. The warning information is stored in the local database of the edge device in a data structure (such as JSON format) for subsequent retrieval.

[0205] The anomaly detection results, including the energy distribution characteristic diagram and the change rate curve, are stored on the edge device in the form of files or data records. The storage process preferentially compresses the characteristic diagram (e.g., using a lossless compression algorithm to reduce the file size) for efficient transmission in step S5. At the same time, the system records the metadata of the anomaly event, such as the current value of the environmental background feature vector E, for subsequent analysis.

[0206] In this embodiment, through real-time feature comparison with thresholds, deviation ratio calculation, and anomaly grading mechanism, the accuracy and robustness of anomaly detection are significantly improved. The specific technical effects include:

[0207] Improved the precision of anomaly detection: By calculating the deviation ratios RA and RV and introducing duration judgment, the system can quantify the degree of anomalies and reduce false alarms caused by environmental noise or instantaneous changes. For example, in an environment with sudden changes in light, the combination of dynamic thresholds and deviation ratio calculation ensures reliable determination of video anomalies, reducing the false alarm rate by more than 20%.

[0208] Enhanced the interpretability and traceability of warning information: Generate structured warning information containing deviation ratios, anomaly levels, and characteristic diagrams, enabling monitoring personnel to quickly understand the cause and severity of anomalies for subsequent diagnosis and decision-making. At the same time, the storage of characteristic diagrams supports historical backtracking, improving the interpretability of the system and meeting the requirements of causal relevant technical relationships.

[0209] Optimized resource utilization: Implement local computing and storage on edge devices, reducing data transmission requirements and being applicable to scenarios with limited computing resources. The anomaly grading mechanism allows for priority processing of high-risk events, improving the response efficiency.

[0210] Ensured the ease of implementation of the system: This embodiment uses standard statistical methods and general data processing techniques, enabling technicians in the relevant technical field to easily implement it based on commercial edge computing platforms (such as Raspberry Pi or similar devices) without complex customization.

[0211] Preferably, in step S5, the sending of the warning information from the edge device to the monitoring center through the low-bandwidth communication protocol includes:

[0212] Adopt the MQTT protocol or the LoRaWAN protocol to send the warning information to the monitoring center in a compressed format;

[0213] Give priority to transmitting the key fields of the warning information, and the key fields include the abnormal type, abnormal timestamp, and abnormal confidence level;

[0214] Delay the transmission of the energy distribution feature map to reduce network bandwidth occupancy.

[0215] In an embodiment provided by the present application, when the edge device determines an abnormal event, the agent module will perform the following operations to generate a transmission data packet:

[0216] Key field generation: Extract key fields from the abnormal detection result, including:

[0217] Abnormal type encoding (1 byte): Use binary encoding to distinguish audio abnormality (0x01), video abnormality (0x02), or composite abnormality (0x03);

[0218] Timestamp (4 bytes): Record the abnormal occurrence time in the UNIX timestamp format;

[0219] Confidence level (1 byte): Divide it into three levels: low (0x01), medium (0x02), and high (0x03) based on the amplitude of the energy distribution exceeding the threshold;

[0220] Feature map index (2 bytes): An index identifier pointing to the complete energy distribution map and the change rate curve stored locally;

[0221] Feature map compression processing:

[0222] Perform grayscale processing on the energy distribution map, and convert the original RGB image into an 8-bit grayscale image;

[0223] Adopt the LZ77 lossless compression algorithm to compress the data of the grayscale image, and control the compression ratio within 30%-50%;

[0224] Perform differential encoding on the change rate curve data, and only store the change amount of adjacent data points instead of the absolute value;

[0225] Through structured encoding and compression processing, reduce the data volume of a single warning information from the original 1.5MB to 50-100KB, reducing the network transmission load.

[0226] Adopt the MQTT protocol to implement hierarchical transmission of warning information, specifically including the following mechanisms:

[0227] Priority queue management:

[0228] The key - field data packet is marked as QoS2 (the highest quality - of - service level) to ensure at - least - once reliable transmission;

[0229] The feature - map data packet is marked as QoS 0 (best - effort transmission level), allowing non - critical data to be discarded under extreme network conditions;

[0230] Bandwidth - adaptive transmission:

[0231] Monitor the network bandwidth fluctuation in real - time. When the available bandwidth is lower than 50 Kbps:

[0232] Pause the feature - map transmission and only send the key fields;

[0233] Start the breakpoint - resumed transmission mechanism and record the verification of the transmitted data packets;

[0234] Automatically re - transmit the incomplete data packets after the network resumes, and give priority to re - transmitting the incomplete transmissions in the most recent 24 hours;

[0235] Local caching strategy:

[0236] Establish a circular buffer (with a capacity of 72 - hour data volume) on the edge device;

[0237] For the feature - map data that has not been successfully transmitted for more than 24 hours, convert it to a low - resolution version (the resolution is reduced to 25% of the original value) and then re - attempt transmission;

[0238] This implementation can still ensure the timely delivery of critical warning information in an environment with network bandwidth fluctuations. Experimental data shows that under the 50 Kbps bandwidth condition, the transmission delay of the key fields is less than 200 ms, meeting the real - time warning requirements.

[0239] The receiving end of the monitoring center performs the following reverse processing:

[0240] Re - construct the event time - line according to the timestamp and the feature - map index;

[0241] De - compress and re - construct the compressed feature - map, and use the bilinear interpolation algorithm to improve the readability of the low - resolution image;

[0242] Automatically request to re - transmit the missing key data packets and initiate a re - transmission request to the edge device through the index identifier;

[0243] Achieve end - to - end data integrity guarantee. Even in the case of network interruption, the complete event record can still be restored through later re - transmission. System tests show that the data integrity rate reaches 99.7%.

[0244] This embodiment achieves the following remarkable effects:

[0245] Optimization of transmission efficiency: Through structured encapsulation and hierarchical transmission strategies, while ensuring the integrity of core information, the network bandwidth occupancy rate is reduced by 60%-75%;

[0246] Guarantee of real-time performance: The priority transmission mechanism for key fields enables the monitoring center to receive abnormal alarms within 200ms, with a 6-fold improvement in response speed compared to the traditional FTP transmission method (average delay of 1.2s);

[0247] Enhanced network adaptability: The bandwidth adaptive mechanism enables the system to operate stably in unstable network environments such as 3G / 4G, and the resume interrupted transfer function ensures the ultimate reachability of data;

[0248] Savings in storage resources: The circular buffer design reduces the storage space occupancy of edge devices by 40% and extends the local data retention period to 72 hours.

[0249] Preferably, the dynamic adjustment of the anomaly detection threshold is self-calibrated based on historical data on the edge device at regular intervals, and the self-calibration log formed includes:

[0250] The edge node regularly collects historical anomaly detection data and environmental background data;

[0251] Based on the false positive rate and false negative rate in the historical anomaly detection data, a threshold self-calibration optimization objective function is constructed:

[0252] minJ(θ)=λ1·FPR(θ)+λ2·FNR(θ)+λ3·||θ-θ0|| 2

[0253] Where θ represents the set of parameters in the threshold calculation formula, the set of parameters includes α, β; FPR represents the false positive rate, FNR represents the false negative rate, θ0 represents the initial value of the parameter, and λ1, λ2, λ3 are weight coefficients;

[0254] Parameter optimization is performed once every 24 hours, and the random gradient descent method is used to solve for the optimal parameter θ * ;

[0255] An attenuation factor γ (0 < γ < 1) is introduced, such that the new parameter θ new =γ·θ * +(1-γ)·θ old , ensuring smooth parameter updates;

[0256] Record the parameter changes and performance metrics of each self-calibration to form a self-calibration log.

[0257] In an embodiment provided by the present application, a threshold self - calibration process regularly executed on edge devices is described, aiming to optimize the dynamic anomaly detection threshold parameters and improve the anomaly detection accuracy of the system in a dynamic environment. This mechanism is applicable to the audio - video monitoring and early - warning system of the present invention. Specifically for dynamic threshold adjustment, it automatically calibrates the parameters in the threshold calculation formula through historical data analysis, thereby reducing the occurrence of false alarms and missed alarms.

[0258] Historical data collection:

[0259] The edge device starts the self - calibration process regularly (for example, every 24 hours). First, it collects historical anomaly detection data and environmental background data from local storage. The historical anomaly detection data includes records of anomaly events, false - positive rate (FPR), and false - negative rate (FNR) over a past period. Among them, the false - positive rate is defined as the proportion of normal events misjudged as anomalies, and the false - negative rate is defined as the proportion of actual anomaly events not detected; these data can be obtained by analyzing historical early - warning information and manual verification results. The environmental background data includes light intensity, noise level, timestamp, and season information, which are collected in real - time by sensors (such as light sensors and noise sensors) built into the edge device and stored historically.

[0260] To ensure data integrity, the edge device filters the data of the most recent 7 - 30 days as input, avoiding biases caused by using overly old data. At the same time, the data pre - processing steps include removing outliers (such as noise data caused by sensor failures) and normalization, making data in different dimensions (such as light intensity and noise level) at the same magnitude for subsequent optimization calculations.

[0261] Optimization objective function construction:

[0262] Based on the collected historical data, a threshold self - calibration optimization objective function is constructed. This function aims to minimize the false - positive rate and false - negative rate while considering the stability of the parameters. The optimization objective function is defined as:

[0263] minJ(θ)=λ1·FPR(θ)+λ2·FNR(θ)+λ3·||θ - θ0|| 2

[0264] Where: θ represents the set of parameters in the threshold calculation formula, including α (adjustable coefficient) in the audio threshold formula and β (adjustable coefficient) in the video threshold formula. These parameters directly affect the sensitivity of the dynamic threshold.

[0265] FPR(θ) and FNR(θ) respectively represent the false - positive rate and false - negative rate based on historical data. These rates can be calculated from historical anomaly detection results. For example, FPR = (number of false - alarm events / total number of normal events)×100%.

[0266] θ0 represents the initial value of the parameter, which usually comes from the default settings during system deployment or the results of the previous self-calibration.

[0267] λ1, λ2, and λ3 are weight coefficients used to balance different terms in the optimization objective. For example, λ1 and λ2 can be set to 0.4 and 0.4 to prioritize accuracy, while λ3 is set to 0.2 to ensure that the parameter does not deviate too much from the initial value.

[0268] The design of this objective function takes into account the trade-off between false positives and false negatives. At the same time, through the regularization term ||θ - θ0|| 2 prevents over-adjustment of the parameter and ensures system stability.

[0269] Parameter optimization execution:

[0270] The stochastic gradient descent (SGD) method is used to solve the optimization objective function to find the optimal parameter θ * . SGD is an iterative optimization algorithm that gradually adjusts the parameter values by calculating the gradient of the objective function with respect to the parameter. The specific optimization process includes:

[0271] Initialize the parameter θ to the results of the previous self-calibration or the default value.

[0272] Use historical data in batches (for example, each batch contains 100 - 500 historical event samples) to calculate the gradient of the objective function. The gradient calculation is based on the actual energy distribution characteristics, environmental background feature vectors, and corresponding anomaly detection results in the historical data.

[0273] Iteratively update the parameter: where η is the learning rate (step size), which can be set to a small fixed value such as 0.01, or an adaptive adjustment strategy (such as exponential decay) can be adopted to improve the convergence speed. The number of iterations is set according to the data scale, for example, 100 - 500 iterations, until the gradient change is less than a preset threshold (such as 0.001) or the maximum number of iterations is reached.

[0274] To simplify the calculation, the edge device can implement SGD using a lightweight optimization library or built-in algorithms instead of a complex machine learning framework, ensuring operation under limited computing power.

[0275] Parameter smooth update:

[0276] To avoid system instability caused by drastic fluctuations in the parameter, a decay factor γ (where 0 < γ < 1, for example, 0.7) is introduced to calculate the new parameter:

[0277] θ new = γ·θ * +(1 - γ)·θ old

[0278] where θ * is the optimal parameter obtained from this optimization, and θ old is the parameter value from the previous self - calibration. This weighted average method ensures smooth parameter updates and reduces performance fluctuations caused by sudden environmental changes.

[0279] Self - calibration logging:

[0280] After the self - calibration process is completed, the edge device generates and stores the self - calibration log. The log includes the following key information:

[0281] Record of parameter changes: Records the θ values before and after optimization, stored in tabular or sequential form.

[0282] Performance metrics: Records the estimated false positive rate, false negative rate, and overall accuracy after optimization, which are calculated based on historical data verification.

[0283] Timestamp and environmental information: Includes the self - calibration execution time and the environmental background data at that time (such as average light intensity and noise level).

[0284] Optimization details: Briefly records the number of iterations, learning rate, and changes in the objective function value for subsequent auditing and analysis.

[0285] The log is stored in the local database of the edge device in a structured file format (such as JSON or CSV) and can be accessed or downloaded through the monitoring center interface for long - term performance monitoring.

[0286] This embodiment significantly improves the anomaly detection accuracy and adaptability of the system by periodically executing the threshold self - calibration mechanism. The specific technical effects are as follows:

[0287] Improve anomaly detection accuracy: By minimizing the false positive rate and false negative rate in the optimization objective function, the system can automatically adjust the threshold parameters to be more suitable for the current environmental background, avoiding the failure of static thresholds in scenarios with drastic changes in light or noise. For example, in a high - noise environment, the optimized α value may increase the threshold sensitivity and reduce false negative events; experimental data shows that this self - calibration can reduce the average false positive rate by 15% - 20% and the false negative rate by 10% - 15% compared to a system without self - calibration.

[0288] Enhance system robustness and stability: Introduce a parameter smooth update mechanism (using the decay factor γ) to ensure gradual parameter changes and avoid false alarms or detection failures caused by sudden changes. At the same time, combining historical data and environmental information, the system can adapt to seasonal changes (such as the difference between day and night) and improve long - term reliability in diverse monitoring scenarios.

[0289] Efficient resource utilization: The self-calibration process is executed locally on edge devices, requiring only a small amount of computing resources (for example, optimized once every 24 hours, taking less than 5 minutes), without relying on cloud computing, reducing network transmission and computing power requirements. At the same time, logging facilitates subsequent analysis and integration of user feedback, further optimizing system performance.

[0290] Easy to implement and maintain: This mechanism uses standard optimization techniques (such as SGD), and those skilled in the art can easily implement it through a general optimization algorithm library (such as an open-source mathematics library), without the need for complex hardware or software. The user feedback mechanism (such as manual parameter adjustment) can be used as supplementary input to further improve the accuracy of self-calibration.

[0291] Preferably, as Figure 2 shown, a multi-modal model-driven audio-visual monitoring and early warning system includes:

[0292] A signal acquisition module for acquiring audio-visual signals in the monitoring scene and extracting the energy distribution characteristics of the audio-visual signals, where the energy distribution characteristics at least include the frequency energy distribution of the audio signal and the change rates of the brightness and color of the video signal;

[0293] A feature extraction module for performing multi-modal feature extraction of energy distribution characteristics based on the audio-visual signals and constructing an environmental background feature vector in combination with the environmental background information;

[0294] A threshold adjustment module for dynamically adjusting the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector;

[0295] An anomaly detection module for comparing the energy distribution characteristics of the real-time extracted audio-visual signals with the anomaly detection thresholds, and when the range of the anomaly detection thresholds is exceeded, determining it as an abnormal event to obtain early warning information;

[0296] A warning transmission module for sending the warning information from the edge device to the monitoring center through a low-bandwidth communication protocol;

[0297] A self-calibration module for periodically performing self-calibration on the dynamically adjusted anomaly detection thresholds on the edge device based on historical data to form a self-calibration log.

[0298] In an embodiment provided by the present application, a multi-modal model-driven audio-visual monitoring and early warning system provided by the present invention includes a signal acquisition module, a feature extraction module, a threshold adjustment module, an anomaly detection module, a warning transmission module, and a self-calibration module.

[0299] The signal acquisition module is used to acquire audio-visual signals in the monitoring scenario and extract the energy distribution features of the audio-visual signals, where the energy distribution features at least include the frequency energy distribution of the audio signal and the change rates of the brightness and color of the video signal. Specifically, the signal acquisition module includes an audio acquisition unit and a video acquisition unit. The audio acquisition unit includes a high-sensitivity microphone array, which supports wide-band audio sampling from 20 Hz to 20 kHz, with a sampling rate of 44.1 kHz and a sampling precision of 16 bits, and is used to acquire audio signals in the environment. The video acquisition unit includes a high-definition camera, which supports video recording with a resolution of 1080p and above, and a frame rate of 30 fps, and is used to acquire video signals in the environment.

[0300] The signal acquisition module also includes a signal preprocessing unit, which is used to perform preliminary processing on the acquired raw audio-visual signals. For the audio signal, the signal preprocessing unit uses the fast Fourier transform (FFT) algorithm to convert the time-domain signal into a frequency-domain representation, and calculates the energy proportion of the low-frequency band (20 Hz - 500 Hz), the middle-frequency band (500 Hz - 5 kHz), and the high-frequency band (5 kHz - 20 kHz), forming the frequency energy distribution feature of the audio signal. For the video signal, the signal preprocessing unit uses the inter-frame difference method to calculate the brightness change rate between adjacent frames, and extracts the motion trajectory feature of the object in the picture based on the optical flow method, forming the brightness and color change rate feature of the video signal.

[0301] The signal acquisition module can acquire audio-visual signals in the monitoring scenario with high precision, and extract preliminary energy distribution features through preprocessing, laying a foundation for subsequent multi-modal feature extraction. The technical effect of this module is that through high-precision signal acquisition and preprocessing, the original features of the audio-visual signals are ensured to be retained, the signal noise interference is reduced, and the accuracy of subsequent analysis is improved.

[0302] The feature extraction module is used to perform multi-modal feature extraction of energy distribution features based on audio-visual signals, and construct an environmental background feature vector in combination with environmental background information. Specifically, the feature extraction module includes a multi-modal feature extraction unit and an environmental background perception unit.

[0303] The multi-modal feature extraction unit deploys a lightweight multi-modal deep learning model using the MobileNet architecture on the edge device. This model has been pre-trained and can effectively extract the feature representation of audio-visual signals. For the audio signal, the multi-modal feature extraction unit extracts the energy proportion features of the low frequency, middle frequency, and high frequency; for the video signal, it extracts the intra-frame brightness gradient feature and the inter-frame color change rate feature. The multi-modal feature extraction unit also uses the attention mechanism to perform adaptive weight allocation on the energy proportion feature of the audio and the brightness gradient feature and color change rate feature of the video, forming a fused feature representation.

[0304] The environmental background perception unit uses the built-in light sensor and noise sensor of the edge device to collect the environmental light intensity and environmental noise level in real time. The environmental background perception unit normalizes the collected environmental light intensity and environmental noise level with the fusion features, and combines the historical environmental data to construct an environmental background feature vector including timestamp, light intensity, noise level and season information.

[0305] Through multi-modal feature extraction and environmental background perception, the feature extraction module enables the system to comprehensively characterize the physical properties of audio-visual signals, and combines environmental background information to avoid the dependence of traditional content recognition methods on complex scenes, improving the robustness of the system in diverse environments. At the same time, the lightweight MobileNet architecture is adopted, enabling the feature extraction process to operate efficiently under the limited computing power conditions of edge devices.

[0306] The threshold adjustment module is used to dynamically adjust the anomaly detection thresholds of audio signals and video signals based on the environmental background feature vector. Specifically, the threshold adjustment module includes an audio threshold calculation unit and a video threshold calculation unit.

[0307] Through the dynamic threshold adjustment mechanism, the threshold adjustment module enables the system to adjust the anomaly detection threshold in real time according to the environmental background feature vector, significantly improving the accuracy of anomaly detection, especially in scenes with large changes in light or noise, effectively reducing the false alarm rate and missed alarm rate. At the same time, a time weighting function and a light intensity weighting function are introduced, enabling the system to adapt to the monitoring requirements at different time periods and different light conditions.

[0308] The anomaly detection module is used to compare the energy distribution features of real-time extracted audio-visual signals with the anomaly detection threshold. When the range of the anomaly detection threshold is exceeded, it is determined as an abnormal event to obtain a warning message. Specifically, the anomaly detection module includes an audio anomaly detection unit, a video anomaly detection unit and an abnormal event storage unit.

[0309] The audio anomaly detection unit compares the energy distribution features of real-time extracted audio frequencies with the anomaly detection threshold of the audio. When the energy distribution features of real-time extracted audio frequencies exceed the range of the anomaly detection threshold of the audio, it is determined as an audio abnormal event.

[0310] The video anomaly detection unit compares the feature of the real-time extracted video brightness change rate with the anomaly detection threshold of the video. When the feature of the real-time extracted video brightness change rate exceeds the range of the anomaly detection threshold of the video, it is determined as a video abnormal event.

[0311] The abnormal event storage unit stores the abnormal detection results in the edge device in the form of an energy distribution map and a change rate curve, and generates a warning message containing key fields and an energy distribution feature map. The key fields include the abnormal type (audio abnormality or video abnormality), the abnormal timestamp, and the abnormal confidence level.

[0312] The abnormal detection module compares the energy distribution features of the real-time extracted audio and video signals with the dynamically adjusted abnormal detection threshold. The system can accurately identify abnormal events and generate a warning message containing key information. At the same time, the abnormal detection results are stored in the form of an energy distribution map and a change rate curve, providing detailed data support for subsequent analysis.

[0313] The warning transmission module is used to send the warning message from the edge device to the monitoring center through a low-bandwidth communication protocol. Specifically, the warning transmission module includes a communication protocol unit and a data transmission optimization unit.

[0314] The communication protocol unit uses the MQTT protocol or the LoRaWAN protocol to send the warning message to the monitoring center in a compressed format. The MQTT protocol is suitable for scenarios with good network connections, while the LoRaWAN protocol is suitable for scenarios with poor network coverage but requiring long-distance transmission.

[0315] The data transmission optimization unit implements a hierarchical transmission strategy for the warning message. The key fields of the warning message are preferentially transmitted. The key fields include the abnormal type, the abnormal timestamp, and the abnormal confidence level. The transmission of the energy distribution feature map is delayed to reduce network bandwidth occupancy. When the network bandwidth is sufficient, the data transmission optimization unit will automatically increase the transmission priority and accelerate the transmission speed of the energy distribution feature map.

[0316] The technical effect of the warning transmission module is that: through the low-bandwidth communication protocol and the hierarchical transmission strategy, the system can ensure the timely transmission of the warning message under the condition of limited network resources, while reducing network bandwidth occupancy. This is particularly important for a monitoring system deployed in an edge environment with limited network conditions.

[0317] The self-calibration module is used to periodically perform self-calibration on the dynamically adjusted abnormal detection threshold on the edge device based on historical data, forming a self-calibration log. Specifically, the self-calibration module includes a data acquisition unit, an optimization target construction unit, a parameter optimization unit, and a log recording unit.

[0318] The data acquisition unit periodically acquires historical abnormal detection data and environmental background data, including the identified abnormal events, false alarm events, missed alarm events, and the corresponding environmental background feature vectors.

[0319] The optimization target construction unit constructs a threshold self-calibration optimization objective function based on the false positive rate and false negative rate in the historical anomaly detection data, aiming to minimize both the false positive rate and the false negative rate simultaneously and maintain the stability of the parameters.

[0320] The parameter optimization unit performs parameter optimization once every 24 hours and uses the stochastic gradient descent method to solve for the optimal parameters. To ensure the smoothness of parameter updates, the parameter optimization unit introduces a decay factor to ensure smooth parameter updates.

[0321] The log recording unit records the parameter changes and performance metrics of each self-calibration to form a self-calibration log. The self-calibration log includes information such as the values before and after parameter updates, the changes in the false positive rate and false negative rate, and the value of the optimization objective function, providing a basis for system administrators to evaluate and optimize the system performance.

[0322] Through the regular self-calibration process of the self-calibration module, the system can adaptively adjust the calculation parameters of the anomaly detection threshold according to historical data, improving the long-term stability and accuracy of the system. At the same time, the recording of the self-calibration log provides visual monitoring of the system performance for system administrators, facilitating system maintenance and optimization.

[0323] In this embodiment, the above-mentioned modules can be integrated on edge computing devices such as intelligent cameras and edge servers. Edge computing devices have sufficient computing power to support the operation of lightweight multi-modal models and real-time anomaly detection. At the same time, edge computing devices are also equipped with network communication modules, which can communicate with the monitoring center through wired or wireless networks.

[0324] The above is the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also considered within the protection scope of the present invention.

Claims

1. An audio - video monitoring and warning method based on multi - modal model drive, characterized in that, It includes: Collect audio and video signals in the monitoring scenario, and extract the energy distribution characteristics of the audio and video signals, where the energy distribution characteristics at least include the frequency energy distribution of the audio signal and the change rates of the brightness and color of the video signal; Perform multi-modal feature extraction of the energy distribution characteristics based on the audio and video signals, and construct an environmental background feature vector in combination with the environmental background information; Dynamically adjust the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector; Compare the energy distribution characteristics of the audio and video signals extracted in real time with the anomaly detection thresholds. When it exceeds the anomaly detection threshold range, it is determined as an abnormal event to obtain a warning message; Send the warning message from the edge device to the monitoring center through a low-bandwidth communication protocol.

2. The method for audio-visual monitoring and early warning based on a multi-modal model drive according to claim 1, wherein, It also includes: Periodically perform self-calibration of the dynamically adjusted anomaly detection thresholds on the edge device based on historical data to form a self-calibration log.

3. The audio - video monitoring and warning method based on a multi - modal model drive according to claim 1, characterized in that, The collecting of the audio and video signals in the monitoring scenario and the extraction of the energy distribution characteristics of the audio and video signals include: Use a high-sensitivity microphone to collect the audio signal, and adopt the fast Fourier transform to extract the energy ratios of the audio signal in three frequency bands of low frequency, medium frequency and high frequency as the frequency energy distribution characteristics of the audio signal; Use a high-definition camera to collect the video signal, and adopt the inter-frame difference method to extract the brightness change rate of each pixel point in the video signal, and the motion trajectory characteristics based on the optical flow method as the change rate characteristics of the brightness and color of the video signal.

4. The method for audio - visual monitoring and early warning based on a multi - modal model drive according to claim 3, wherein, The performing of multi-modal feature extraction of the energy distribution characteristics based on the audio and video signals includes: Deploy a lightweight multi-modal deep learning model using the MobileNet architecture on the edge device; Extract the energy ratio characteristics of low frequency, medium frequency and high frequency for the audio signal, and extract the intra-frame brightness gradient characteristics and the inter-frame color change rate characteristics for the video signal; Use the attention mechanism to perform adaptive weight allocation on the energy ratio characteristics of the audio and the brightness gradient characteristics and color change rate characteristics of the video to form a fused feature representation.

5. The method for audio - video monitoring and early warning based on a multi - modal model drive according to claim 4, wherein, The constructing of the environmental background feature vector in combination with the environmental background information includes: Use the built-in light sensor and noise sensor of the edge device to collect the environmental light intensity and environmental noise level in real time; Normalize the collected environmental light intensity and environmental noise level with the fused feature, and combine with historical environmental data to construct an environmental background feature vector including timestamp, light intensity, noise level and season information.

6. The method for audio - video monitoring and early warning based on a multi - modal model drive according to claim 5, characterized in that, The dynamically adjusting of the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector includes: Construct an audio anomaly detection threshold calculation formula: ThA(t) = μA(E) + α·σA(E)·f(t) Where, ThA(t) is the audio anomaly detection threshold at time t, μA(E) is the mean value of the audio energy distribution under the current environmental background feature vector E, σA(E) is the corresponding standard deviation, α is an adjustable coefficient, and f(t) is a time weighting function used to adjust the sensitivity in different time periods; Construct a video anomaly detection threshold calculation formula: ThV(t) = μV(E) + β·σV(E)·g(L) Among them, ThV(t) is the video anomaly detection threshold at time t, μV(E) is the mean value of the video brightness and color change rate under the current environmental background feature vector E, σV(E) is the corresponding standard deviation, β is an adjustable coefficient, and g(L) is a light intensity weighting function used to adjust the threshold sensitivity under different lighting conditions.

7. A method for audio - video monitoring and early warning based on a multi - modal model drive according to claim 6, characterized in that, Comparing the energy distribution features of the real-time extracted audio-visual signals with the anomaly detection threshold, when it exceeds the anomaly detection threshold range, it is determined as an abnormal event, and the warning information obtained includes: Comparing the energy distribution features of the real-time extracted audio frequency with the audio anomaly detection threshold, and comparing the real-time extracted video brightness change rate features with the video anomaly detection threshold; When the energy distribution features of the real-time extracted audio frequency exceed the audio anomaly detection threshold range, or the real-time extracted video brightness change rate features exceed the video anomaly detection threshold range, it is determined as an abnormal event; Storing the anomaly detection results in the form of an energy distribution map and a change rate curve in the edge device, and generating warning information including keyword fields and an energy distribution feature map.

8. A method for audio-visual monitoring and early warning based on a multi-modal model drive according to claim 7, characterized in that The sending the warning information from the edge device to the monitoring center through a low-bandwidth communication protocol includes: Adopting the MQTT protocol or the LoRaWAN protocol to send the warning information to the monitoring center in a compressed format; Prioritizing the transmission of the keyword fields of the warning information, where the keyword fields include the anomaly type, anomaly timestamp, and anomaly confidence; Delaying the transmission of the energy distribution feature map to reduce network bandwidth occupancy.

9. A method for audio - video monitoring and early warning based on a multi - modal model drive according to claim 2, characterized in that, The periodically self-calibrating the dynamically adjusted anomaly detection threshold on the edge device based on historical data to form a self-calibration log includes: The edge node periodically collects historical anomaly detection data and environmental background data; Based on the false alarm rate and missed alarm rate in the historical anomaly detection data, constructing a threshold self-calibration optimization objective function: minJ(θ) = λ1·FPR(θ) + λ2·FNR(θ) + λ3·||θ - θ0|| 2 Among them, θ represents the parameter set in the threshold calculation formula, the parameter set includes α, β; FPR represents the false alarm rate, FNR represents the missed alarm rate, θ0 represents the initial value of the parameter, and λ1, λ2, λ3 are weight coefficients; Perform parameter optimization once every 24 hours, and use the stochastic gradient descent method to solve for the optimal parameter θ * ; Introduce a decay factor γ (0 < γ < 1) such that the new parameter θ new = γ·θ * + (1 - γ)·θ old , ensuring smooth parameter updates; Recording the parameter changes and performance indicators of each self-calibration to form a self-calibration log.

10. An audio-video monitoring and warning system based on a multi-modal model drive, characterized in that, Including: A signal acquisition module for acquiring audio-visual signals in the monitoring scenario and extracting the energy distribution features of the audio-visual signals, where the energy distribution features at least include the frequency energy distribution of the audio signal and the change rate of the brightness and color of the video signal; A feature extraction module for performing multi-modal feature extraction of the energy distribution features based on the audio-visual signals and constructing an environmental background feature vector in combination with the environmental background information; A threshold adjustment module for dynamically adjusting the anomaly detection thresholds of the audio signal and the video signal based on the environmental background feature vector; An anomaly detection module for comparing the energy distribution features of the real-time extracted audio-visual signals with the anomaly detection threshold, and when it exceeds the anomaly detection threshold range, determining it as an abnormal event to obtain warning information; A warning transmission module for sending the warning information from the edge device to the monitoring center through a low-bandwidth communication protocol; A self-calibration module, which is used to periodically perform self-calibration on the dynamically adjusted anomaly detection threshold on the edge device based on historical data, and form a self-calibration log.

Images