Multi-scene voice alarm system based on video AI behavior analysis

Abstract

The invention relates to the technical field of computer vision and intelligent monitoring, and particularly discloses a multi-scene voice alarm system based on video AI behavior analysis. The system comprises a video acquisition and preprocessing module, a multi-mode environment perception module, a core AI behavior identification and scene understanding module, a dynamic alarm strategy generator and a programmable voice synthesis and broadcast control module. According to the system, the alarm volume and content details are dynamically adjusted by sensing environmental noise and scene personnel density in real time, and the behavior position and the scene structure are fused to generate alarm information with path guidance, so that full-dimension adaptive optimization of the alarm volume, the content and a broadcast strategy is realized; and the audibility, the pertinence and the field response efficiency of the alarm information are obviously improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and intelligent monitoring, and particularly relates to a multi-scene voice alarm system based on video AI behavior analysis. BACKGROUND

[0002] Video monitoring and intelligent analysis technology is an important part of the security field, which automatically identifies and understands the monitoring picture through computer vision algorithm, and is widely used in public safety, traffic management, industrial inspection and other scenes. Triggering voice alarm based on video analysis results has become a key technology direction to improve the efficiency of on-site response and achieve proactive security.

[0003] The prior art usually simply connects the video AI behavior recognition module and the voice alarm module. When the system detects a preset abnormal behavior (such as intrusion, loitering, falling), a fixed voice alarm content is triggered and broadcasted at a preset volume. This mode exposes significant defects in complex and variable actual application scenarios: in environments such as factory workshops and traffic intersections where there is continuous or intermittent strong noise, the alarm voice of fixed volume is easily covered by environmental noise, resulting in ineffective transmission of alarm information to on-site personnel, which poses a serious safety hazard.

[0004] At the same time, the existing system lacks perception and utilization of dynamic context information such as crowd density in the monitoring scene, and cannot adaptively adjust the degree of detail or broadcast strategy of the alarm content according to the busy degree of the scene (such as sparse or dense crowds). In a crowded area, it may cause repeated alarms and information overload, disrupting normal order; while in a sparse area, it may not provide effective guidance due to too brief information, resulting in insufficient pertinence and effectiveness of the alarm. SUMMARY

[0005] The purpose of the present application is to provide a multi-scene voice alarm system based on video AI behavior analysis to solve the problems in the prior art that the alarm volume is fixed and easily covered by environmental noise, and the alarm content cannot be adaptively adjusted according to the dynamic context information of the scene, resulting in ineffective or inefficient transmission of alarm information.

[0006] The present application provides a multi-scene voice alarm system based on video AI behavior analysis, which is a closed-loop adaptive system integrating environmental perception, behavior recognition, context analysis and intelligent decision-making. The system includes a video acquisition and preprocessing module, a multi-modal environmental perception module, a core AI behavior recognition and scene understanding module, a dynamic alarm strategy generator, and a programmable voice synthesis and broadcast control module. The above-mentioned modules work cooperatively through the data bus and control bus defined internally in the system.

[0007] The video acquisition and preprocessing module is used to acquire raw video streams from network cameras deployed in the monitoring area. This module performs decoding, noise reduction, format normalization, and keyframe extraction on the raw video stream, generating a standardized video frame sequence, which is then output to the core AI behavior recognition and scene understanding module. Simultaneously, this module separates the audio channel from the raw video stream and outputs the raw environmental audio signal to the multimodal environment perception module.

[0008] The multimodal environmental perception module is used for real-time quantification of the acoustic and optical environmental conditions at the monitoring site. This module includes a high-precision acoustic analysis submodule and an ambient light intensity perception submodule. The acoustic analysis submodule receives the raw environmental audio signal from the video acquisition and preprocessing module, and first performs frame-by-frame windowing processing on the audio signal, with a frame length of 20 milliseconds and a frame shift of 10 milliseconds.

[0009] Next, the A-weighted sound pressure level of each frame of audio signal is calculated, and the A-weighted sound pressure level values ​​of all frames in the past 5 seconds are counted. The 95th percentile of these values ​​is taken as the current ambient noise baseline value, in decibels. The ambient light intensity sensing submodule extracts the average pixel brightness of a specified region of interest from the standardized video frame sequence output by the video acquisition and preprocessing module, as a reference value for ambient light intensity.

[0010] The core AI behavior recognition and scene understanding module is the intelligent hub of the system, running two deep neural network models in parallel: a behavior recognition model and a scene parsing model. The behavior recognition model is a multi-object spatiotemporal graph convolutional network pre-trained on a large-scale abnormal behavior dataset. This model receives video frame sequences as input, extracts key point sequences from the human skeleton, constructs a spatiotemporal graph, and identifies various preset abnormal behaviors, including but not limited to area intrusion, loitering, leaving objects behind, falling, and running violently. For each identified behavior, the model outputs a behavior type label, a behavior confidence score, and the bounding box coordinates of the behavior's location.

[0011] The scene analysis model is a convolutional neural network based on semantic segmentation, which also receives video frame sequences. This model performs pixel-level classification on each frame, outputting a heatmap of people density, a static obstacle distribution map, and a mask for passable areas within the scene. The scene analysis model further calculates the number of people within a specified monitoring area from the people density heatmap and combines this with historical frame data to calculate the trend of people density changes, ultimately outputting the current scene's people density level, which is divided into three discrete states: sparse, normal, and dense.

[0012] The dynamic alarm policy generator is used to generate optimal alarm execution parameters based on the real-time output of the multimodal environment perception module and the core AI behavior recognition and scene understanding module. This generator incorporates a policy mapping matrix and an adaptive adjustment algorithm. The policy mapping matrix is ​​a two-dimensional lookup table, where the row index represents the scene's personnel density level and the column index represents the abnormal behavior type.

[0013] Each cell of the matrix stores a set of basic alarm content template identifiers and broadcast priority weights. The dynamic alarm strategy generator first retrieves the corresponding basic alarm content templates and priority weights from the strategy mapping matrix based on the identified behavior type and current personnel density level. Subsequently, the generator initiates an adaptive adjustment algorithm. The volume adjustment sub-process of this algorithm reads the current environmental noise baseline value provided by the multimodal environment perception module and calculates the target alarm volume based on a preset speech intelligibility model.

[0014] The speech clarity model specifies that the target alarm volume value is equal to the ambient noise baseline value plus a dynamic signal-to-noise ratio margin. This margin is linearly adjusted according to the behavior confidence score. For every 0.1 increase in confidence, the signal-to-noise ratio margin increases by 2 dB, but the total margin range is constrained to between 10 and 25 dB.

[0015] The content optimization sub-process of the algorithm calculates the optimal path direction description to reach the behavior occurrence point based on the bounding box coordinates of the behavior occurrence location and the passable area mask output by the scene parsing model, and inserts this direction description information into the specified placeholder in the basic alarm content template.

[0016] Finally, the dynamic alarm policy generator outputs a structured alarm instruction package, which includes the target alarm volume, the final alarm text content incorporating path guidance, and the broadcast priority.

[0017] The programmable speech synthesis and broadcast control module receives and executes alarm command packets from the dynamic alarm policy generator. This module includes a text-to-speech engine and a multi-channel digital audio processor. The text-to-speech engine employs a deep learning-based end-to-end speech synthesis model to convert the final alarm text content in the alarm command packet into highly natural-looking speech waveform data.

[0018] The multi-channel digital audio processor is responsible for post-processing and power amplification of the generated speech waveform. Its processing flow includes: First, based on the target alarm volume value in the alarm command packet, the amplitude of the voice waveform is adjusted by a digital gain controller.

[0019] Next, a dynamic range compressor is applied to control the peak value of the voice signal below the safe limit to prevent overload distortion.

[0020] Finally, the processor distributes the processed multi-channel audio signals to the corresponding power amplifiers and speaker arrays for playback, based on the speaker layout configured in the system. This module also manages the alarm task queue, sorting and scheduling multiple alarm commands triggered simultaneously according to their playback priority. Higher-priority alarms can interrupt or overwrite lower-priority alarms that are currently being played.

[0021] In one embodiment of the present invention, the policy mapping matrix in the dynamic alarm policy generator has alarm content templates that are differentiated according to personnel density levels. When the personnel density level is sparse, the alarm content template includes detailed behavioral descriptions, precise location information, and specific handling suggestions.

[0022] When the personnel density level is normal, the alarm content template is simplified to a description of behavior and location information. When the personnel density level is high, the alarm content template is further simplified to a brief prompt of key behavior types and directional guidance to avoid information overload.

[0023] Furthermore, the scene analysis model in the core AI behavior recognition and scene understanding module uses a sliding window weighted average method to calculate the personnel density level. The specific process is as follows: The statistical time window is set to 30 seconds. The number of people in each second within the window is recorded at 1-second intervals. Data closer to the current moment is given higher weight. The current population density is estimated by weighted average calculation. Then, based on the preset threshold ranges of sparse, normal, and dense levels, it is mapped to the corresponding population density level.

[0024] Furthermore, the acoustic analysis submodule of the multimodal environment perception module is also used for noise spectrum characteristic analysis. This submodule performs a fast Fourier transform on the raw environmental audio signal, calculates its power spectral density, and identifies characteristic frequency bands with energy significantly higher than the average level. When the identified characteristic frequency band overlaps with the system's preset typical continuous industrial noise frequency band by more than 70%, the acoustic analysis submodule sends a noise type identifier as steady-state noise to the dynamic alarm strategy generator. Otherwise, it is identified as transient or mixed noise. When calculating the target alarm volume, the adaptive adjustment algorithm of the dynamic alarm strategy generator adds an extra 3 dB compensation to the final volume value if the noise type is identified as steady-state noise, in order to counteract the masking effect of steady-state noise on speech.

[0025] Furthermore, the multi-channel digital audio processor of the programmable speech synthesis and broadcast control module integrates echo cancellation and noise suppression functions. This processor estimates and eliminates acoustic echoes in real time by referencing the signal output from the speaker and the feedback signal collected by the microphone, using a normalized least mean square adaptive filtering algorithm.

[0026] Meanwhile, by combining the noise spectrum characteristics provided by the multimodal environment perception module, targeted spectral reduction processing is performed on the speech band to further improve the clarity of the broadcast speech in noisy environments.

[0027] Furthermore, the system also includes an offline strategy optimization and model update module. This module periodically collects system operation logs, which include environmental noise data, triggered alarm commands, scene status, and alarm effectiveness scores entered through a manual feedback interface.

[0028] This module utilizes this log data to iteratively optimize the policy mapping matrix and adaptive adjustment algorithm parameters in the dynamic alarm policy generator through reinforcement learning algorithms. Simultaneously, this module injects newly labeled behavior samples into the training pipeline of the core AI behavior recognition and scene understanding module, enabling incremental learning and online updates of the behavior recognition model and scene analysis model.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention introduces a multimodal environmental perception module to achieve accurate and real-time quantification of environmental noise levels and dynamically calculates alarm volume based on a speech clarity model. This technology completely changes the fixed-volume alarm mode, ensuring that alarm voice maintains sufficient signal-to-noise ratio and audibility in any noisy environment. It fundamentally solves the technical problem of alarm information being masked by environmental noise and significantly improves the reliability of the alarm system in high-noise scenarios such as industrial workshops and transportation hubs.

[0030] 2. This invention utilizes a scene analysis model within its core AI behavior recognition and scene understanding module to analyze the density level and spatial structure of personnel in the monitored area in real time, enabling the system to perceive the dynamic contextual information of the scene. The dynamic alarm strategy generator uses this information, through a preset strategy mapping matrix, to intelligently adapt the level of detail in the alarm content. It provides concise prompts to prevent interference in densely populated areas and detailed guidance to facilitate handling in sparsely populated areas. This content-adaptive mechanism ensures that the amount of information output in the alarm message precisely matches the needs of the scene, greatly enhancing the relevance of the alarms and the reception efficiency for on-site personnel.

[0031] 3. This invention constructs a complete technical closed loop from environmental perception, behavior recognition, and contextual understanding to strategy generation and speech synthesis. The dynamic alarm strategy generator integrates multi-source information such as acoustic environment, visual behavior, and scene status for joint decision-making, and innovatively combines behavior location with passable areas of the scene to generate path guidance, giving alarm content both warning and navigation functions. Deep collaboration among all modules of the entire system achieves full-dimensional adaptive optimization of alarm volume, alarm content, and broadcast strategy, forming a highly intelligent, scene-aware next-generation voice alarm solution that effectively improves the overall early warning efficiency and response level of security systems. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the overall technical solution architecture of the multi-scenario voice alarm system based on video AI behavior analysis proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the dynamic alarm strategy generator in this invention, which integrates multi-source information to make decisions. Figure 3 This is a logical flow diagram of the core AI behavior recognition and scene understanding module in this invention; Figure 4 This is a schematic diagram of the multi-level interaction relationship and data flow of the multimodal environment perception and programmable speech synthesis and broadcast control module in this invention; Figure 5 This is a logical framework diagram of the offline strategy optimization and model update module in this invention to achieve system self-evolution. Detailed Implementation

[0033] Please refer to the attached document. Figures 1 to 5 This invention provides a multi-scenario voice alarm system based on video AI behavior analysis. This system is a closed-loop adaptive system integrating environmental perception, behavior recognition, contextual analysis, and intelligent decision-making. The system includes a video acquisition and preprocessing module, a multimodal environmental perception module, a core AI behavior recognition and scene understanding module, a dynamic alarm strategy generator, and a programmable speech synthesis and playback control module.

[0034] The aforementioned modules work together through a data bus and a control bus defined within the system. The data bus is responsible for transmitting data streams such as video frames, audio signals, structured analysis results, and alarm commands, while the control bus is responsible for transmitting control signals such as module start / stop, parameter configuration, and status synchronization, ensuring strict synchronization of the entire system in terms of timing and logic.

[0035] The video acquisition and preprocessing module is the system's data input terminal, and its hardware foundation consists of one or more network cameras deployed in the monitored area. This module continuously receives raw video streams from the network cameras via a network protocol stack, using real-time transmission protocols or real-time messaging protocols. The raw video streams are typically encapsulated in H.264 or H.265 encoding formats and may contain multiple audio channels.

[0036] This module first calls the corresponding hardware decoder or software decoding library to decode the encoded video stream, restoring the original YUV or RGB image sequence. The decoding process is accompanied by a bitstream verification mechanism; when a data packet loss or verification error is detected, the module triggers a retransmission request or uses the previous frame's data to hide the error. The decoded original video frames then enter the preprocessing pipeline.

[0037] The first step in preprocessing is joint spatial and frequency domain denoising, using a threshold denoising algorithm based on wavelet transform to effectively suppress Gaussian noise and salt-and-pepper noise in the image, thereby improving the accuracy of subsequent analysis.

[0038] The second step is format standardization. Regardless of the resolution and frame rate of the input video, the preprocessing module will use bilinear interpolation algorithm to uniformly scale it to 1920×1080 pixels, stabilize the frame rate at 25 frames per second, and uniformly convert the color space to RGB three-channel format.

[0039] The third step is keyframe extraction. The preprocessing module calculates the average absolute difference between consecutive frames. When the difference exceeds a preset threshold of 50, it is determined that a significant scene change has occurred, and the frame is marked as a keyframe. At the same time, the system also ensures that at least one frame is extracted as a keyframe every 2 seconds. These standardized video frames are arranged in timestamp order to form a video frame sequence, which is cached in a first-in-first-out queue, waiting to be read by the core AI behavior recognition and scene understanding module.

[0040] Meanwhile, the module extracts the audio elementary stream from the container of the original video stream using an audio separator. This audio is typically in PCM format, either mono or stereo, with a sampling rate of 16 kHz and a bit depth of 16 bits. This raw environmental audio signal is then directly output to the acoustic analysis submodule of the multimodal environment perception module.

[0041] The multimodal environment perception module is responsible for real-time quantitative monitoring of the acoustic and optical environmental conditions on-site, providing the system with accurate environmental context parameters. Please refer to the attached document. Figure 4 This module includes a high-precision acoustic analysis submodule and an ambient light intensity sensing submodule, which work in parallel.

[0042] The acoustic analysis submodule receives the raw environmental audio signal from the video acquisition and preprocessing module. This submodule first performs frame-by-frame windowing on the continuous audio signal to balance temporal resolution and frequency stability. The frame length is fixed at 20 milliseconds, corresponding to 320 sampling points at a 16 kHz sampling rate; the frame shift is 10 milliseconds, meaning there is 50% overlap between adjacent frames.

[0043] The windowing function uses a Hamming window to reduce spectral leakage. For each frame of windowed audio signal, the submodule calculates its A-weighted sound pressure level. The calculation process is as follows: First, the frame data is subjected to a Fast Fourier Transform to obtain the spectrum. Then, each frequency component is weighted according to the A-weighting curve specified in the international standard IEC 61672:2013 to simulate the human ear's sensitivity to different frequencies of sound. Finally, the weighted spectral energy is summed and converted into a decibel value to obtain the instantaneous A-weighted sound pressure level of that frame.

[0044] To obtain a stable and representative ambient noise level, the submodule maintains a 5-second circular buffer, continuously storing the latest calculated instantaneous sound pressure level (SPL) value. Every 100 milliseconds, the submodule sorts all values ​​in the buffer and takes the 95th percentile as the current ambient noise baseline. For example, if the buffer contains 500 values, the 475th value after sorting is taken as the output. This baseline value is measured in decibels (dB), reflecting a higher level of background noise in the current environment, rather than instantaneous peak noise.

[0045] In addition, the acoustic analysis submodule performs noise spectral characteristic analysis. This submodule performs a 2048-point Fast Fourier Transform on the raw ambient audio signal to calculate its power spectral density.

[0046] Subsequently, the algorithm calculates the average energy of each frequency band within the 1 / 3 octave band from 63 Hz to 8 kHz and compares it with the average energy of the entire band. When the energy of a certain 1 / 3 octave band exceeds the average energy of the entire band by more than 10 dB, that band is marked as a characteristic band. The system internally stores several typical continuous industrial noise spectrum templates, such as fan noise mainly concentrated in the 125 Hz to 500 Hz range, and compressor noise concentrated in the 250 Hz to 1 kHz range.

[0047] The submodule calculates the overlap between the identified feature frequency bands and these preset template frequency bands. When the overlap exceeds 70%, the acoustic analysis submodule determines that the current noise type is steady-state noise and sends a status flag indicating that the noise type is steady-state noise to the dynamic alarm strategy generator. Otherwise, it is identified as transient or mixed noise. All acoustic analysis results, including environmental noise baseline values, noise type identifiers, and characteristic frequency band distribution vectors, are packaged into an acoustic environmental state data structure and published via a data bus.

[0048] The ambient light intensity sensing submodule extracts ambient light intensity reference values ​​from the standardized video frame sequence output by the video acquisition and preprocessing module. This submodule defines one or more rectangular regions of interest on the video frames. These regions typically avoid direct light sources and dark objects, representing typical reflective surfaces in the scene, such as the ground and walls.

[0049] For each frame, the submodule extracts the brightness values ​​of all pixels within a specified region of interest. In the RGB color space, the brightness value is calculated using the formula: Brightness = 0.299 × Red channel value + 0.587 × Green channel value + 0.114 × Blue channel value. Subsequently, the arithmetic mean of the brightness of all pixels within this region is calculated as a reference value for the ambient light intensity of that frame, with a value ranging from 0 to 255.

[0050] To smooth out instantaneous fluctuations in illumination, the submodule employs a first-order low-pass filter to filter the average brightness of consecutive frames, with a time constant of 2 seconds. The final output ambient light intensity reference value is used to assist the system in determining whether it is currently daytime, nighttime, or artificial lighting conditions; this information can serve as input for future expansion functions.

[0051] The core AI behavior recognition and scene understanding module is the system's intelligent hub, responsible for parsing high-level behavioral semantics and scene structure information from visual data. Please refer to the appendix. Figure 3 This module runs two deep neural network models in parallel: an action recognition model and a scene parsing model. The two models share a standardized video frame sequence provided by the video acquisition and preprocessing module as input, but perform different inference tasks.

[0052] The behavior recognition model is a multi-object spatiotemporal graph convolutional network pre-trained on a large-scale abnormal behavior dataset. Its inference process consists of three stages.

[0053] The first stage is human detection and keypoint extraction. The model first uses a target detection head based on the YOLOv5 architecture to locate all human targets in the input video frames and output their bounding boxes. For each detected human bounding box, the model uses a lightweight pose estimation network, such as a variant of OpenPose, to extract the coordinates of 17 two-dimensional skeletal keypoints of the human, including the nose, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0054] The second stage is the construction of the spatiotemporal graph. For a video segment consisting of T consecutive frames, the model constructs a spatiotemporal graph for each human body. The nodes of the graph are the 17 keypoints in each frame, so the total number of nodes is 17×T. The edges of the graph are of two types: spatial edges connect the physical connections of the human body between different keypoints within the same frame, and temporal edges connect the positions of the same keypoint between adjacent frames.

[0055] The third stage is graph convolution and classification. The constructed spatiotemporal graph is fed into a multi-layer spatiotemporal graph convolutional network. This network learns the dynamic patterns of human joint movements by alternately performing graph convolution operations in the spatial and temporal dimensions.

[0056] Finally, the network outputs a classification result for the behavior of each person in the video clip. The system has preset several abnormal behavior types, including area intrusion, loitering, leaving objects behind, falling, and running wildly.

[0057] For each identified behavior, the model outputs a structured description of the behavior event, which includes a behavior type label, a behavior confidence score between 0 and 1, and the bounding box coordinates of the location where the behavior occurred.

[0058] The bounding box coordinates are represented in image pixel coordinates, in the format of top-left x-coordinate, top-left y-coordinate, width, and height. All identified events are encapsulated into a list, sorted by confidence level from highest to lowest, and then output.

[0059] The scene parsing model is a semantic segmentation-based convolutional neural network with an architecture using DeepLabv3+, optimized for surveillance scenarios. This model performs pixel-level classification on each input frame and outputs three parallel semantic segmentation maps.

[0060] The first is a heatmap of people density. This map is not a simple binary mask of people, but rather a value is predicted for each pixel, representing the probability that the pixel belongs to the people area. After Gaussian blurring, a continuous heatmap distribution is formed.

[0061] The second is a static obstacle distribution map, which separates fixed obstacles in the scene such as counters, railings, equipment, and walls.

[0062] The third is the passable area mask, which identifies the ground or areas where people are allowed to move freely. The scene analysis model further uses the personnel density heatmap to count the number of people within the specified monitoring area.

[0063] The statistical method involves performing connected component analysis on pixels with a probability value exceeding 0.5 in the heatmap, with each connected component counted as one person. Combined with historical frame data, the model calculates the trend in the number of people. Specifically, a sliding window weighted average method is used: the statistical time window length is set to 30 seconds, and the number of people is recorded at 1-second intervals within each second, forming a sequence.

[0064] Each value in the sequence is assigned a weight, with the weight coefficient decaying exponentially over time; data closer to the current moment has a higher weight. A weighted average is used to calculate the current population density estimate. The system presets a threshold range for population density levels: when the estimate is less than 5 people, it is mapped to a sparsity level. When the estimated value is between 5 and 20 people, it is mapped to the normal level; When the estimated number is greater than 20 people, it is mapped to a density level.

[0065] Finally, the scene analysis model outputs a scene state data structure, including a heatmap of personnel density, a static obstacle distribution map, a passable area mask, and discrete personnel density levels.

[0066] The dynamic alarm strategy generator is the core of the system's decision-making process. It generates optimal alarm execution parameters based on the real-time outputs of the multimodal environment perception module and the core AI behavior recognition and scene understanding module. Please refer to the attached document. Figure 2 The generator has a built-in policy mapping matrix and an adaptive adjustment algorithm.

[0067] The strategy mapping matrix is ​​a two-dimensional lookup table data structure stored in memory. Its row index represents the scene's personnel density level, i.e., sparse, normal, and dense values. The column index represents the abnormal behavior type, i.e., area intrusion, loitering, leaving items behind, falling, running wildly, etc.

[0068] Each cell in the matrix stores a set of basic alarm content template identifiers and a broadcast priority weight value. The alarm content template is a predefined text string containing placeholders. For example, for the "area intrusion" behavior, the template at the sparse level might be "Attention, personnel detected loitering near the eastern restricted area. Please leave this area immediately." The priority weight is an integer ranging from 1 to 10, with higher values ​​indicating more urgent alarms.

[0069] The dynamic alarm strategy generator works as follows: Upon receiving a list of behavioral events from the core AI behavior recognition and scene understanding module, the generator iterates through each valid event in the list. For each event, it first uses its behavior type label and the current personnel density level obtained from the scene analysis model as the row and column keys to query the strategy mapping matrix, retrieving the corresponding basic alarm content template and priority weight. Subsequently, the generator initiates an adaptive adjustment algorithm, which includes two parallel sub-processes: volume adjustment and content optimization.

[0070] The volume adjustment subprocess reads the current ambient noise baseline value provided by the multimodal environment perception module and calculates the target alarm volume based on a preset speech intelligibility model. The speech intelligibility model specifies that the target alarm volume value equals the ambient noise baseline value plus a dynamic signal-to-noise ratio margin. This margin is linearly adjusted based on the behavior confidence score output by the behavior recognition model.

[0071] The adjustment rule is: the basic signal-to-noise ratio (SNR) margin is 10 dB; for every 0.1 increase in confidence level, the SNR margin increases by 2 dB. For example, when the confidence level is 0.7, the increased margin is... The total margin is 14 dB. However, the total margin range is constrained to between 10 and 25 dB, that is, when the calculated result is below 10 dB, it is taken as 10 dB, and when it is above 25 dB, it is taken as 25 dB.

[0072] Furthermore, if the noise type identified by the acoustic analysis submodule is steady-state noise, an additional 3 dB compensation is added to the final calculated target alarm volume value to counteract the stronger masking effect of steady-state noise on speech. The final target alarm volume value is a numerical value in decibels.

[0073] The content optimization sub-process aims to make alert content more action-oriented. This sub-process calculates the optimal path direction description to the point where the behavior occurred based on the location bounding box coordinates in the behavioral event and the passable area mask output by the scene parsing model.

[0074] The specific steps are as follows: First, the center point coordinates of the behavior bounding box are used as the target point. Then, on the passable area mask, starting from the entrance of the monitored area or the preset responder's permanent location, the A* pathfinding algorithm is used to calculate the shortest path avoiding static obstacles. Finally, the macroscopic direction of this path is converted into a natural language description, such as "Go 10 meters north from your current location, then turn right." This directional description information is inserted into the designated placeholder in the basic alarm content template, replacing the original general location terms to form the final alarm text content.

[0075] Finally, the dynamic alarm policy generator packages all the above information into a structured alarm command package. This command package is encapsulated in JSON format and contains the following key fields: target alarm volume, final alarm text content incorporating path guidance, broadcast priority, source behavior type, and timestamp. This alarm command package is then published to the system's data bus.

[0076] The programmable speech synthesis and broadcast control module is the system's execution terminal, used to receive and execute alarm command packets from the dynamic alarm policy generator. Please refer to the attached document again. Figure 4 This module includes a text-to-speech engine and a multi-channel digital audio processor.

[0077] The text-to-speech engine employs an end-to-end speech synthesis model based on deep learning, such as VITS or FastSpeech 2. The engine receives the final alarm text content from the alarm instruction packet and first performs front-end text processing, including text regularization, word segmentation, part-of-speech tagging, and prosodic prediction. Then, the back-end acoustic model generates the corresponding Mel spectrogram based on the predicted prosodic features.

[0078] Finally, the vocoder converts the Mel spectrogram into highly natural-sounding speech waveform data at a sampling rate of 24 kHz, mono. The engine supports multiple timbre options and fine-tunes the speech rate and tone according to the urgency of the alarm; high-priority alarms use a more urgent and higher-pitched speech style.

[0079] The multi-channel digital audio processor is responsible for professional post-processing and power amplification of the generated speech waveform. Its processing flow is a rigorous digital signal processing chain. First, the digital gain controller adjusts the amplitude of the speech waveform according to the target alarm volume value in the alarm command packet. The processor internally maintains a reference sound pressure level calibration table, mapping the decibel value of the digital gain to the actual speaker output sound pressure level to ensure accurate volume control.

[0080] Next, the speech signal is fed into a dynamic range compressor with a threshold set to -3 dB, a compression ratio of 4:1, and start-up and release times of 5 milliseconds and 100 milliseconds, respectively. This ensures that the peak value of the speech signal is smoothly controlled below safe limits, preventing amplifier overload or speaker distortion. Then, the advanced algorithms integrated into the processor begin to work.

[0081] The echo cancellation subsystem uses a normalized least mean square adaptive filtering algorithm to estimate and eliminate acoustic echoes in real time by referencing the signal output from the loudspeaker and the signal collected by feedback microphones deployed in the environment. The filter order is 512. The noise suppression subsystem combines the noise spectrum characteristics provided by the multimodal environment perception module to perform targeted spectral attenuation processing in the speech band, thereby improving speech intelligibility.

[0082] Finally, the processor distributes the processed audio signals according to the speaker layout strategy configured in the system. For example, the speaker group corresponding to the area where the behavior occurs plays at full volume, while the adjacent area plays at a lower volume, creating a sound field focusing effect. The processed multi-channel digital audio signals are sent to the power amplifiers and speaker arrays in the corresponding areas through digital audio interfaces for electroacoustic conversion and playback.

[0083] This module also manages a priority-based alarm task queue. When multiple alarm command packets are received simultaneously, the module sorts them according to their broadcast priority. The queue scheduler uses a preemptive strategy, where a higher-priority alarm can immediately interrupt or overwrite a lower-priority alarm that is being broadcast. For interrupted alarms, if their priority is still valid, they will be re-queued for broadcast after the current higher-priority alarm has finished broadcasting.

[0084] As one embodiment of the present invention, the strategy mapping matrix in the dynamic alarm strategy generator has its alarm content templates designed with profound differentiation based on personnel density levels. When the personnel density level is sparse, the alarm content template includes detailed behavioral descriptions, precise location information, and specific handling suggestions, such as "Warning: A black package has been found left in passage number three. Please have nearby security personnel go to check immediately and contact the bomb disposal team."

[0085] When the personnel density level is normal, the alarm content template is simplified to a description of behavior and location information, such as "Please note that someone has been loitering at counter number seven for an extended period of time." When the personnel density level is high, the alarm content template is further simplified to a brief prompt of key behavior type and directional guidance, such as "Turn left ahead, be careful to avoid people who have fallen," in order to avoid information overload and panic in noisy and crowded environments.

[0086] Furthermore, the scene analysis model in the core AI behavior recognition and scene understanding module uses a sliding window weighted average method to calculate the personnel density level. The specific process is as follows: a statistical time window of 30 seconds is set, and the number of people is recorded every second within the window, forming a sequence. Each value in the sequence is assigned a weight, with the weight coefficient decaying exponentially over time; data closer to the current moment has a higher weight. The current personnel density estimate is obtained through weighted averaging, and then mapped to the corresponding personnel density level based on preset threshold ranges for sparse, normal, and dense levels. This method effectively smooths out instantaneous fluctuations in personnel counts, providing a more stable judgment of scene status.

[0087] Furthermore, the acoustic analysis submodule of the multimodal environment perception module is also used for noise spectrum characteristic analysis. This submodule performs a fast Fourier transform on the raw environmental audio signal, calculates its power spectral density, and identifies characteristic frequency bands with energy significantly higher than the average level. When the identified characteristic frequency band overlaps with the system's preset typical continuous industrial noise frequency band by more than 70%, the acoustic analysis submodule sends a noise type identifier as steady-state noise to the dynamic alarm strategy generator. Otherwise, it is identified as transient or mixed noise. When calculating the target alarm volume, the adaptive adjustment algorithm of the dynamic alarm strategy generator adds an extra 3 dB compensation to the final volume value if the noise type is identified as steady-state noise, in order to counteract the long-term masking effect of steady-state noise on speech.

[0088] Furthermore, the multi-channel digital audio processor in the programmable speech synthesis and broadcast control module integrates echo cancellation and noise suppression functions. This processor estimates and eliminates acoustic echoes in real time by referencing the signal output from the speaker and the feedback signal collected by the microphone, using a normalized least mean square adaptive filtering algorithm. Simultaneously, combined with the noise spectrum characteristics provided by the multimodal environment perception module, targeted spectral attenuation processing is performed on the speech band to further improve the clarity and intelligibility of the broadcast speech in noisy environments.

Claims

1. A multi-scenario voice alarm system based on video AI behavior analysis, characterized in that, include: The video acquisition and preprocessing module is used to acquire raw video streams from network cameras deployed in the monitoring area, perform decoding, noise reduction, format standardization and key frame extraction operations on the raw video streams, generate standardized video frame sequences, and separate the audio channels in the raw video streams to output raw environmental audio signals. A multimodal environment perception module is used for real-time quantitative monitoring of the acoustic and optical environment conditions at the site. The core AI behavior recognition and scene understanding module is used to run the behavior recognition model and the scene analysis model in parallel. A dynamic alarm strategy generator is used to generate optimal alarm execution parameters by comprehensively making decisions based on the real-time output of the multimodal environment perception module and the core AI behavior recognition and scene understanding module. The programmable speech synthesis and broadcast control module is used to receive and execute alarm instruction packets from the dynamic alarm strategy generator.

2. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 1, characterized in that, The multimodal environment perception module includes a high-precision acoustic analysis submodule and an ambient light intensity perception submodule; The high-precision acoustic analysis submodule receives the original ambient audio signal, performs frame-by-frame windowing processing on the audio signal, calculates the A-weighted sound pressure level of each frame of the audio signal, and counts the A-weighted sound pressure level values ​​of all frames in the past 5 seconds, taking the 95th percentile as the current ambient noise baseline value; the ambient light intensity sensing submodule extracts the average pixel brightness of a specified region of interest from the standardized video frame sequence as a reference value for ambient light intensity.

3. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 2, characterized in that, The behavior recognition model receives the standardized video frame sequence as input, extracts the human skeleton key point sequence and constructs a spatiotemporal graph, identifies a variety of preset abnormal behaviors, and outputs behavior type labels, behavior confidence scores and bounding box coordinates of the location where the behavior occurs. The scene analysis model receives the standardized video frame sequence, performs pixel-level classification on each frame, outputs a heat map of personnel density, a static obstacle distribution map, and a passable area mask in the scene, counts the number of people in the specified monitoring area from the personnel density heat map, calculates the changing trend of personnel number in combination with historical frame data, and finally outputs the current scene personnel density level divided into three discrete states: sparse, normal, and dense.

4. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 3, characterized in that, The dynamic alarm strategy generator has a built-in strategy mapping matrix and an adaptive adjustment algorithm. The strategy mapping matrix is ​​a two-dimensional lookup table, with row indexes representing scene personnel density levels and column indexes representing abnormal behavior types. Each cell of the matrix stores a set of basic alarm content template identifiers and broadcast priority weights. The dynamic alarm strategy generator first retrieves the corresponding basic alarm content template and priority weight from the strategy mapping matrix based on the identified behavior type and the current personnel density level, and then starts the adaptive adjustment algorithm. The adaptive adjustment algorithm includes a volume adjustment sub-process and a content optimization sub-process; The volume adjustment subprocess reads the current ambient noise reference value and calculates the target alarm volume based on a preset speech intelligibility model. The speech intelligibility model specifies that the target alarm volume value is equal to the ambient noise reference value plus a dynamic signal-to-noise ratio margin, which is linearly adjusted according to the behavior confidence score. The content optimization subprocess calculates the optimal path direction description to the behavior occurrence point based on the bounding box coordinates of the behavior occurrence location and the passable area mask, and inserts this direction description information into the specified placeholder of the basic alarm content template to form the final alarm text content. The dynamic alarm strategy generator outputs a structured alarm instruction package that includes the target alarm volume, the final alarm text content, and the broadcast priority.

5. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 4, characterized in that, The programmable speech synthesis and broadcast control module includes a text-to-speech engine and a multi-channel digital audio processor. The text-to-speech engine uses an end-to-end speech synthesis model based on deep learning to convert the final alarm text content into speech waveform data. The multi-channel digital audio processor adjusts the amplitude of the voice waveform through a digital gain controller according to the target alarm volume value, uses a dynamic range compressor to control the peak value of the voice signal below the safety limit, and distributes the processed multi-channel audio signals to the power amplifiers and speaker arrays in the corresponding areas for broadcasting according to the speaker layout configured in the system. The programmable speech synthesis and broadcast control module also manages the alarm task queue, and sorts and schedules multiple alarm commands that are triggered at the same time according to their broadcast priority.

6. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 5, characterized in that, The alarm content templates in the strategy mapping matrix are designed differently according to personnel density levels; When the personnel density level is sparse, the alarm content template includes a detailed description of behavior, accurate location information, and specific handling suggestions; When the personnel density level is normal, the alarm content template is simplified to a behavior description and location information; When the personnel density level is dense, the alarm content template is simplified to a brief prompt with key behavior types and directional guidance.

7. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 6, characterized in that, The calculation of personnel density levels in the scene analysis model adopts the sliding window weighted average method; the specific process is as follows: The statistical time window is set to 30 seconds. The number of people in each second within the window is recorded at 1-second intervals. Data closer to the current moment is given higher weight. The current population density is estimated by weighted average calculation. Then, based on the preset threshold ranges of sparse, normal, and dense levels, it is mapped to the corresponding population density level.

8. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 7, characterized in that, The high-precision acoustic analysis submodule is also used to perform noise spectrum characteristic analysis; this submodule performs a fast Fourier transform on the original environmental audio signal, calculates its power spectral density, and identifies characteristic frequency bands with energy significantly higher than the average level; When the overlap between the identified characteristic frequency band and the system's preset typical continuous industrial noise frequency band exceeds 70%, the high-precision acoustic analysis submodule sends a noise type identifier to the dynamic alarm strategy generator as steady-state noise; otherwise, it is identified as transient or mixed noise. When calculating the target alarm volume, the adaptive adjustment algorithm of the dynamic alarm strategy generator adds an extra 3 dB compensation to the final volume value if the noise type is identified as steady-state noise.

9. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 8, characterized in that, The multi-channel digital audio processor integrates echo cancellation and noise suppression functions; The processor uses a normalized least mean square adaptive filtering algorithm to estimate and eliminate acoustic echoes in real time by referencing the signal output from the speaker and the feedback signal collected by the microphone. Simultaneously, based on the noise spectrum characteristics provided by the multimodal environment perception module, targeted spectral reduction processing is performed on the speech audio segment.

10. The multi-scenario voice alarm system based on video AI behavior analysis according to claim 9, characterized in that, It also includes offline strategy optimization and model update modules; The offline strategy optimization and model update module periodically collects system operation logs, which include environmental noise data, triggered alarm commands, scene status, and alarm effectiveness scores entered through the manual feedback interface. The module uses this log data to iteratively optimize the policy mapping matrix and the parameters of the adaptive adjustment algorithm in the dynamic alarm policy generator through reinforcement learning algorithms.

Images