Campus dangerous behavior partition early warning system based on multi-modal perception

Abstract

This invention relates to the field of intelligent security monitoring technology and discloses a campus dangerous behavior zoning early warning system based on multimodal perception. The system includes a depth camera, an omnidirectional microphone, and an edge computing server. A static mapping database is built within the server. Real-time depth video is used to lock onto the target body and extract features of non-target obstructions. Combined with zoning parameters, the baseline sound field features are corrected to derive the dynamic theoretical sound mix ratio. Simultaneously, the airborne physical transmission time of the direct sound wave is calculated based on the target's three-dimensional coordinates. Audio time segments are aligned and truncated to extract the actual sound mix ratio value. Finally, the spatial physical parameter deviation between the dynamic theoretical and actual sound mix ratios is calculated. When the deviation meets the tolerance condition, the audiovisual confidence level is fused and an early warning is output. This invention filters out erroneous correlation interference when actions and sounds occur independently at different locations through physical spatial consistency verification, reducing the false alarm rate in complex environments.

Description

Technical Field

[0001] This invention relates to the field of intelligent security monitoring technology, specifically to a campus dangerous behavior zoning early warning system based on multimodal perception. Background Technology

[0002] With the increasing demand for campus security, surveillance systems are gradually evolving from single-mode video monitoring to multi-modal audiovisual fusion. Existing dangerous behavior early warning systems typically calculate the anomaly probabilities of the two modalities independently when processing visual and audio data, and then perform simple numerical superposition or logical binding. However, in actual campus environments, there are often densely populated areas, spatial obstructions, and complex wall reflections, resulting in significant differences in acoustic attenuation characteristics across different areas.

[0003] Because sound travels much slower than the speed of light and is significantly affected by the physical structure of the environment, existing systems lack verification of the physical consistency of audiovisual features in terms of temporal transmission delay and three-dimensional spatial location. This deficiency prevents the system from determining whether abnormal sounds and abnormal movements truly originate from the same source. For example, when a normal running action occurs in one location on the monitoring screen, and high-frequency noise interference happens to occur in another spatial location outside the screen, the system may easily misassociate spatially independent actions with sound and trigger an alarm. Furthermore, existing technologies often use globally fixed fusion decision parameters, failing to adaptively compensate for dynamic changes in obstructions and the sound field characteristics of different areas, further hindering the reduction of false alarm rates in multimodal early warning systems in complex campus scenarios. Summary of the Invention

[0004] The technical problem solved by this invention is that existing campus security monitoring systems, when issuing warnings of dangerous behaviors, usually simply superimpose or independently judge the recognition results of visual and audio modalities, which easily leads to the incorrect association of irrelevant visual actions and abnormal sounds occurring in different spatial locations, resulting in a high false alarm rate.

[0005] To address the above problems, the present invention provides the following technical solution:

[0006] The first aspect of this invention provides a campus hazardous behavior zoning early warning system based on multimodal perception, including a depth camera, an omnidirectional microphone, and an edge computing server; the depth camera and the omnidirectional microphone are installed in the campus early warning area and are respectively communicatively connected to the edge computing server; the edge computing server is equipped with:

[0007] The acoustic calibration module is used to construct a static mapping database between three-dimensional spatial coordinates and reference sound field characteristics;

[0008] The topology extraction module is used to lock the target body based on the real-time depth video data collected by the depth camera device and extract the absolute timestamp of visual action, calculate the three-dimensional world coordinates of the target body, extract the non-target occlusion features between the target body and the omnidirectional microphone and generate the confidence level of visual dangerous behavior, and determine the warning zone to which the target body belongs in the pre-established warning zone parameter table.

[0009] The parameter compensation module is used to combine the features of non-target obstructions and the associated parameters corresponding to the current warning zone to correct the reference sound field features corresponding to the target location in the static mapping database and output the dynamic theoretical sound mixing ratio.

[0010] The audio alignment module is used to calculate the airborne physical transmission time of the direct sound wave based on the three-dimensional world coordinates of the target body, combine the visual action absolute timestamp to extract the corresponding audio time segment from the audio data collected by the omnidirectional microphone, and extract the actual sound mixing ratio value and audio anomaly confidence.

[0011] The verification decision module is used to calculate the spatial physical parameter deviation between the dynamic theoretical sound mix ratio and the actual sound mix ratio. When the spatial physical parameter deviation meets the environmental tolerance value condition corresponding to the warning zone, the confidence of visual dangerous behavior and the confidence of audio anomaly are fused and calculated, and the warning command is output when the joint warning score reaches the trigger threshold.

[0012] Furthermore, the specific process of constructing the static mapping database by the acoustic calibration module is as follows:

[0013] The campus early warning area is divided into grids to obtain multiple spatial grid sampling nodes, and room impulse response signals are collected at each spatial grid sampling node.

[0014] Calculate the reference direct sound to reverberant sound energy ratio corresponding to each spatial grid sampling node, and calculate the time difference between the first effective local peak time after the main lobe of the direct sound and the initial arrival time of the direct sound, as the early reflection delay time;

[0015] A spatial mapping relationship is established between the three-dimensional spatial coordinates of each spatial grid sampling node and the corresponding reference direct sound to reverberation energy ratio and early reflection delay time, forming a static mapping database.

[0016] Furthermore, the specific process of the topology extraction module extracting features of non-target occluders is as follows:

[0017] A spatial geometric channel is constructed with the line connecting the three-dimensional world coordinates of the target body to the three-dimensional coordinates of the omnidirectional microphone as the central axis;

[0018] The spatial geometry channel is reprojected onto the two-dimensional depth image to generate an image region mask, and the object located between the target body and the omnidirectional microphone and falling inside the spatial geometry channel is selected as a group of non-target occluders.

[0019] The effective acoustic projection area and depth distribution values ​​of each non-target occluder in the non-target occluder group are extracted as non-target occluder features.

[0020] Furthermore, the specific process by which the parameter compensation module outputs the dynamic theoretical sound mixing ratio is as follows:

[0021] Substitute the effective acoustic projection area and depth distribution values ​​of each non-target obstruction into the spatial loss attenuation model, and combine them with the sound absorption coefficients corresponding to the non-target obstructions to calculate the spatial acoustic attenuation compensation factor.

[0022] Extract the environmental attenuation compensation weight corresponding to the current warning zone from the warning zone parameter table, and extract the reference direct sound and reverberant sound energy ratio corresponding to the target body position from the static mapping database;

[0023] The dynamic theoretical sound mixing ratio is output after numerical correction by subtracting the product of the environmental attenuation compensation weight and the spatial acoustic attenuation compensation factor from the ratio of the reference direct sound to the reverberant sound energy.

[0024] Furthermore, the specific process by which the audio alignment module extracts the corresponding audio time segment is as follows:

[0025] The physical straight-line distance is calculated based on the three-dimensional world coordinates of the target body and the three-dimensional coordinates of the omnidirectional microphone. The physical straight-line distance is then divided by the speed of sound in air to obtain the physical transmission time in air.

[0026] The starting boundary time of the audio time segment is calculated by adding the absolute timestamp of the visual action to the physical transmission time in the air and subtracting the preset safety redundancy prefix time.

[0027] The early reflection delay time of the corresponding location is extracted from the static mapping database. The cutoff boundary time of the audio time segment is calculated by adding the early reflection delay time and the preset analysis window length to the starting boundary time.

[0028] Furthermore, the specific process by which the audio alignment module extracts the actual sound mixing ratio and audio anomaly confidence level is as follows:

[0029] Calculate the average short-time energy of the extracted audio segment. When the average short-time energy is greater than or equal to the background noise level threshold, perform blind reverberation assessment on the audio segment and separate the estimated direct sound energy and reverberant sound energy.

[0030] Based on the estimated direct sound energy and reverberant sound energy, the energy ratio of direct sound to reverberant sound is calculated in the logarithmic energy domain to obtain the actual sound mixing ratio value; at the same time, the audio anomaly confidence level is generated based on the degree of energy transcendence of the extracted audio segment relative to the background noise and the stability of the blind reverberation assessment results.

[0031] Furthermore, the specific process of verification and fusion performed by the verification decision module is as follows:

[0032] Subtracting the actual sound mixing ratio from the dynamic theoretical sound mixing ratio yields the deviation of spatial physical parameters.

[0033] When the deviation of spatial physical parameters is greater than the environmental tolerance value corresponding to the warning zone, it is determined that the visual modality and audio modality do not meet the consistency condition in physical space, the event is classified as scattered interference and the current warning triggering process is terminated.

[0034] When the deviation of spatial physical parameters is less than or equal to the environmental tolerance value corresponding to the warning zone, the confidence of visual dangerous behavior and the confidence of audio anomaly are weighted and summed according to the visual modality fusion weight and audio modality fusion weight corresponding to the current warning zone, and the joint warning score is output.

[0035] Furthermore, the specific configuration of the early warning zone parameter table is as follows:

[0036] Record the spatial boundary data of each warning zone; and under the identifier of each warning zone, bind and store the corresponding environmental attenuation compensation weight, environmental tolerance value, visual modal fusion weight, audio modal fusion weight and joint warning trigger threshold;

[0037] The topology extraction module determines the warning zone to which the target body belongs by matching the three-dimensional world coordinates of the target body with the spatial boundary data of each warning zone.

[0038] Furthermore, when the minimum distance from the three-dimensional world coordinates of the target body to the boundary of the adjacent warning zone is less than the preset zone boundary buffer threshold, the topology extraction module maintains the zone determination result of the previous moment as the current zone determination output.

[0039] Furthermore, the specific process of the topology extraction module locking the target body and generating a visual dangerous behavior confidence score is as follows: deploy a human pose estimation model to extract skeletal key points from real-time depth video data, calculate the three-dimensional displacement variance of each skeletal key point within a time sliding window; calculate the proportion of skeletal key points with a three-dimensional displacement variance greater than a first threshold to obtain the abnormal action ratio; when the abnormal action ratio is greater than a preset ratio threshold, lock the current object as the target body and generate a visual dangerous behavior confidence score.

[0040] Furthermore, after generating the image region mask, the topology extraction module combines the target body's three-dimensional bounding box, key point distribution range, and depth continuity features to directly remove pixels belonging to the target body itself within the mask.

[0041] A second aspect of this invention provides a campus dangerous behavior zoning early warning method based on multimodal perception, applied to the edge computing server in the aforementioned system, comprising the following steps:

[0042] Construct a static mapping database between three-dimensional spatial coordinates and reference sound field characteristics;

[0043] Based on real-time depth video data, the target body is located and the absolute timestamp of visual action is extracted. The three-dimensional world coordinates are calculated and the features of non-target occlusion objects are extracted. The confidence level of visual dangerous behavior is generated and the warning zone to which the target body belongs is determined.

[0044] The baseline sound field characteristics are corrected by combining the characteristics of non-target obstructions and related parameters, and the dynamic theoretical sound mixing ratio is output.

[0045] The airborne physical transmission time of the direct sound wave is calculated based on the target's three-dimensional world coordinates. The audio time segment is extracted by combining the absolute timestamp of the visual action, and the actual sound mixing ratio and audio anomaly confidence are extracted.

[0046] The spatial physical parameter deviation between the dynamic theoretical sound mix ratio and the actual sound mix ratio is calculated. When the physical deviation meets the environmental tolerance condition, the multimodal confidence is fused, and an early warning is triggered when the score reaches the standard.

[0047] This invention provides a campus hazardous behavior zoning early warning system based on multimodal perception. It has the following beneficial effects:

[0048] 1. This invention establishes a static mapping database of physical space and, in real-time, combines the features of non-target occlusion objects within the scene to calculate the dynamic theoretical sound mix ratio in the current environment. The dynamic theoretical sound mix ratio is compared with the actual sound mix ratio extracted from the audio to obtain the deviation of spatial physical parameters. Based on the objective physical laws of acoustic propagation, this mechanism verifies whether visually locked abnormal actions and auditorily captured abnormal sounds originate from the same three-dimensional spatial location, filtering out splicing interference caused by actions and sounds occurring independently at different locations, thus reducing the false alarm rate in complex environments.

[0049] 2. This invention establishes a dual acoustic compensation algorithm based on dynamic occlusion and zoned attenuation. It utilizes depth images to construct spatial geometric channels, extracts the acoustic projection area and depth distribution of objects, calculates spatial acoustic attenuation compensation factors, and then combines these with specific environmental attenuation compensation weights for different warning zones to correct the baseline sound field characteristics. This processing method can adaptively adjust the acoustic propagation theoretical reference value when scenes change dynamically, such as crowd movement or object stacking, ensuring the data accuracy of the physical feature comparison stage.

[0050] 3. This invention calculates the physical transmission time of sound in the air based on the three-dimensional coordinates of the visual target, and combines this with the early reflection delay time in the sound field environment to extract an effective analysis window of the audio data on the time axis. This method avoids the audio-visual asynchrony problem caused by the slow speed of sound, ensuring that the subsequent blind reverberation evaluation targets the target soundprint segment that is highly corresponding to the visual action.

[0051] 4. This invention enables the system to adopt differentiated fusion decision criteria for different areas such as indoor spaces, corridors, or playgrounds by configuring a warning partition parameter table that includes environmental tolerance values ​​and modal weights; at the same time, it introduces partition boundary buffer decision to maintain state stability when the target is active at the partition boundary, avoiding system logic oscillation caused by frequent switching of calculation parameters. Attached Figure Description

[0052] Figure 1 This is a diagram illustrating the architecture of the indoor hazardous behavior zoning early warning system based on multimodal perception, as described in this invention.

[0053] Figure 2 This is a flowchart of the indoor hazardous behavior zoning early warning method based on multimodal perception of the present invention;

[0054] Figure 3 This is a schematic diagram illustrating the principle of indoor physical space gridding and static mapping of reference sound field characteristics in this invention.

[0055] Figure 4 This is a schematic diagram illustrating the principle of visual event detection and 3D topology analysis of the present invention.

[0056] Figure 5 This is a schematic diagram illustrating the principle of indoor early warning zone mapping and zone parameter reading of the present invention.

[0057] Figure 6 This is a schematic diagram illustrating the principle of dynamic compensation of environmental acoustic parameters field of view topology in this invention.

[0058] Figure 7 This is a schematic diagram of the physical delay-driven adaptive audio time window alignment principle of the present invention;

[0059] Figure 8 This is a schematic diagram of the cross-modal physical causal cross-validation and decision-making mechanism of the present invention;

[0060] Figure 9 This is a schematic diagram illustrating the application deployment of the local early warning area in a campus teaching building according to the present invention;

[0061] Figure 10 This is a comparison chart of the early warning performance of different schemes of the present invention;

[0062] Figure 11 This is a graph showing the change in recognition performance under different occlusion conditions according to the present invention;

[0063] Figure 12 This is a statistical comparison chart of the deviations in space physics parameters of the present invention.

[0064] Among them, 10 is a depth camera; 20 is an omnidirectional microphone; 30 is an edge computing server; 100 is an acoustic calibration module; 200 is a topology extraction module; 300 is a parameter compensation module; 400 is an audio alignment module; and 500 is a verification decision module. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0066] See attached document Figure 1 This invention provides an indoor hazardous behavior zoning early warning system based on multimodal perception, comprising: a depth camera device 10, an omnidirectional microphone 20, and an edge computing server 30. The edge computing server 30 is equipped with an acoustic calibration module 100, a topology extraction module 200, a parameter compensation module 300, an audio alignment module 400, and a verification decision module 500.

[0067] A depth camera 10 and an omnidirectional microphone 20 are fixedly installed in the indoor early warning area. The depth camera 10 collects real-time depth video data of the indoor area and converts the local depth coordinates into absolute three-dimensional world coordinates. The omnidirectional microphone 20 collects acoustic environmental signals of the indoor area and converts them into audio data. Its installation coordinates in physical space are pre-set known three-dimensional coordinate values. The depth camera 10 and the omnidirectional microphone 20 are respectively connected to the edge computing server 30 through a data communication interface. The acoustic calibration module 100, topology extraction module 200, parameter compensation module 300, audio alignment module 400, and verification decision module 500 sequentially complete data interaction within the edge computing server 30 according to the processing flow.

[0068] The edge computing server 30 stores an indoor early warning zoning parameter table. This table records the spatial boundary data, zoning identifier, risk level, environmental tolerance value, joint early warning trigger threshold, and corresponding zoning response strategy for each early warning zoning zone. After outputting the three-dimensional world coordinates of the target body, the topology extraction module 200 matches these coordinates with the spatial boundary data in the indoor early warning zoning parameter table to determine the early warning zoning zone to which the target body belongs. The parameter compensation module 300 calls the environmental attenuation compensation parameters corresponding to the early warning zoning zone to perform zoning correction on the theoretical sound field parameters at the target location. The verification decision module 500 calls the environmental tolerance value, fusion weight, and joint early warning trigger threshold corresponding to the early warning zoning zone to output a dangerous behavior early warning command with the zoning identifier.

[0069] See attached document Figure 2 This invention provides an indoor hazardous behavior zoning early warning method based on multimodal perception, comprising the following steps:

[0070] S10, during the initial system configuration phase, acquire the spatial acoustic parameters of the indoor warning area, collect the room impulse response of each spatial grid node, extract the direct sound and reverberant sound energy ratio parameters, and construct a static mapping database between three-dimensional spatial coordinates and reference sound field characteristics.

[0071] S20: Receive real-time depth video data, lock the target body by analyzing the displacement variance of key points of the human skeleton, calculate the three-dimensional world coordinates of the target body, and construct a spatial geometric channel based on the three-dimensional world coordinates of the target body and the three-dimensional values ​​of the omnidirectional microphone 20; traverse the environmental depth images within the spatial geometric channel, and calculate the acoustic projection area and depth distribution values ​​of the non-target occlusion group.

[0072] S30: Based on the three-dimensional world coordinates of the target body, determine the warning zone to which the target body belongs in the pre-established indoor warning zone parameter table, read the environmental attenuation compensation parameters, environmental tolerance values, modal fusion weights and joint warning trigger thresholds corresponding to the warning zone, and use the read parameters as input parameters for subsequent dynamic theoretical sound mixing ratio correction and zone warning decision.

[0073] S40, extract the depth distribution values ​​and acoustic projection area of ​​the non-target obstruction group, and substitute them into the spatial loss attenuation model to calculate the spatial acoustic attenuation compensation factor; apply the spatial acoustic attenuation compensation factor to the static mapping database, perform numerical correction for the coordinates of the target body, and output the dynamic theoretical sound mixing ratio.

[0074] S50: Calculate the airborne physical transmission time of the direct sound wave based on the three-dimensional world coordinates of the target body. Combine the physical transmission time with the early reflected sound delay time parameter in the static mapping database, extract the audio time segment in the audio data buffer sequence, and perform a blind reverberation evaluation algorithm on the audio time segment to extract the actual sound mixing ratio value.

[0075] S60: Perform difference calculation on the dynamic theoretical sound mixing ratio and the actual sound mixing ratio to obtain the deviation amount, and compare the deviation amount with the environmental tolerance value corresponding to the current warning zone; when the deviation amount is greater than the environmental tolerance value corresponding to the current warning zone, terminate the joint warning triggering process in the current time period; when the deviation amount is less than or equal to the environmental tolerance value corresponding to the current warning zone, enter the joint warning scoring process, and combine the modal fusion weight and joint warning triggering threshold corresponding to the current warning zone to perform fusion calculation on the confidence of visual dangerous behavior and the confidence of audio anomaly; when the joint warning score reaches the joint warning triggering threshold corresponding to the current warning zone, output the multimodal danger warning triggering command.

[0076] The technical details of each of the above steps will be explained in detail below with reference to the accompanying drawings.

[0077] See attached document Figure 3 In this embodiment, the acoustic calibration module 100 acquires the spatial acoustic parameters of the indoor warning area during the initial system configuration phase, which is used to establish the reference sound field characteristics under steady-state conditions and form a static mapping database for subsequent dynamic theoretical sound mixing ratio correction and audio time window construction.

[0078] Within the indoor warning area, the acoustic calibration module 100 establishes a three-dimensional Cartesian coordinate system and determines the fixed three-dimensional values ​​of the omnidirectional microphone 20 in this coordinate system. The origin of the coordinate system can be selected as a fixed physical boundary vertex of the indoor space, and the direction of the coordinate axes is determined according to the room structure.

[0079] In order to enable the sound field characteristics of different spatial locations to be discretely sampled and participate in subsequent mapping, the acoustic calibration module 100 divides the personnel activity space into grids according to the geometric dimensions of the indoor area, and obtains multiple spatial grid sampling nodes.

[0080] The spacing between spatial grid sampling nodes is related to the size of the warning area and the highest evaluation frequency of the sound wave. As a preferred method, the distance between adjacent sampling nodes is set to... The speed of sound in air is The highest evaluation frequency is ,but Preferred satisfaction:

[0081] ;

[0082] in, Indicates the spatial grid sampling interval. This indicates the speed of sound in air. This represents the highest evaluation frequency corresponding to the sound field calibration. After determining the sampling interval through the above constraints, the absolute coordinate position of each spatial grid sampling node can be determined using an equidistant sampling method.

[0083] After the spatial grid is divided, the acoustic calibration module 100 collects the room impulse response signal at each sampling node of the spatial grid. The room impulse response characterizes the overall transmission characteristics of sound waves when the sound source is located in a specific spatial position, after direct propagation, boundary reflection, scattering, and absorption, before reaching the omnidirectional pickup 20. In specific measurements, the test sound source can be placed sequentially at each sampling node of the spatial grid, and its emission of test sound waves can be controlled. The fixedly installed omnidirectional pickup 20 synchronously receives and converts the test sound waves into electrical signals. As a preferred method, the test sound source uses a sinusoidal sweep frequency signal or a maximum length sequence signal. For the emission control, synchronous acquisition, and deconvolution calculation of the test sound waves, those skilled in the art can use well-known techniques in the field of architectural acoustics measurement, which will not be elaborated here.

[0084] After obtaining the original acoustic response signal, the acoustic calibration module 100 performs time axis truncation processing on the signal to remove the environmental noise segment before the arrival of the direct sound, and retain the effective response portion including the main lobe of the direct sound and the subsequent reverberation attenuation segment.

[0085] Let the first The room impulse response measured at each spatial grid sampling node is: The direct sound begins to arrive at the time of arrival. The cutoff time when the response signal decays to near the background noise level is .in, Indicates the spatial grid sampling node number, Indicates time, Indicates the first The room impulse response corresponding to each sampling node Indicates the start and end times of the direct sound. Indicates the deadline for the valid response segment.

[0086] As a preferred method, The time when the envelope of the response signal first exceeds the background noise threshold can be determined. The time when the response signal decays to near the background noise level can be determined. If the signal-to-noise ratio of the response signal at a certain sampling node is lower than the preset effective threshold, the node can be remeasured; if an effective response signal cannot be obtained after repeated measurements, the interpolation results of neighboring effective nodes can be used to supplement the node to avoid gaps in the static mapping database.

[0087] After obtaining the effective room impulse response, the acoustic calibration module 100 calculates the energy ratio of the reference direct sound to the reverberant sound corresponding to each sampling node. Let the first sampling node be... The three-dimensional spatial coordinates of each spatial grid sampling node are: The time window width used to separate the direct acoustic main energy region from the subsequent reverberation energy region is [width value missing]. The energy ratio of the reference direct sound to the reverberant sound corresponding to that node is... It can be represented as:

[0088] ;

[0089] in, Indicates the first The ratio of reference direct sound to reverberant sound energy at each spatial grid sampling node. , , This represents the three-dimensional spatial coordinates of the sampling node. This represents the instantaneous energy distribution of the room's impulse response in the time domain. This represents the width of the time window for integrating the direct sound energy. As a preferred method, The value ranges from 2 milliseconds to 5 milliseconds, and its specific value can be determined based on the size of the indoor space and the duration of the direct sound main lobe.

[0090] While calculating the energy ratio of the reference direct sound to the reverberant sound, the acoustic calibration module 100 also extracts the early reflection delay time corresponding to each sampling node. Let the first effective local peak time after the main lobe of the direct sound be... The early reflection delay time at the corresponding coordinate point It can be represented as:

[0091] ;

[0092] in, Indicates the first Early reflection delay time at each spatial grid sampling node This indicates the moment of the first effective reflection peak detected after reaching the main lobe of the direct sound. To avoid misinterpreting the fluctuations at the tail of the direct sound as reflection peaks, it is preferable to [do something] at [a certain point]. The first effective local peak is then searched within the subsequent time interval.

[0093] After all spatial grid sampling nodes have completed data extraction, the acoustic calibration module 100 will convert the three-dimensional spatial coordinates... As an index, the corresponding reference direct sound to reverberant sound energy ratio and early reflection delay time Establish spatial mapping relationships. For continuous spatial locations not directly covered by sampling nodes, spatial interpolation algorithms can be used for estimation. As a preferred method, inverse distance weighted interpolation or kriging interpolation algorithms are used. If the location to be estimated coincides with a known sampling node, the measured value of the corresponding sampling node is directly used.

[0094] After spatial interpolation, the acoustic calibration module 100 stores the three-dimensional spatial coordinates and the corresponding reference direct sound to reverberant sound energy ratio and early reflection delay time in the data storage medium, forming a static mapping database between the three-dimensional spatial coordinates and the reference sound field characteristics. This static mapping database serves as a reference for the system under steady-state conditions and is subsequently called by the parameter compensation module 300 and the audio alignment module 400 to complete the dynamic theoretical sound mixing ratio correction and audio time window construction, respectively.

[0095] See attached document Figure 4 In this embodiment, the topology extraction module 200 receives real-time depth video data transmitted by the depth camera device 10, extracts personnel posture information and spatial geometric information to determine the position of the target body, and establishes a sound wave propagation spatial geometric channel between the target body and the omnidirectional microphone 20, so as to output the geometric features of the obstruction required for subsequent environmental acoustic compensation.

[0096] The depth video data is preferably RGB-D data composed of synchronized RGB and depth images. To extract human skeletal key points from the depth video data, a human pose estimation model is deployed within the topology extraction module 200. As a preferred approach, the human pose estimation model employs a high-resolution network structure. The model input is a uniformly sized four-channel image tensor with dimensions H×W×4, where H represents the input image height, W represents the input image width, and the four channels correspond to the red, green, blue, and depth channels, respectively. Before the input data enters the model, the RGB images undergo scaling and pixel normalization, and the depth images undergo invalid depth compensation and numerical normalization to reduce the impact of different acquisition distances and lighting conditions on the key point detection results.

[0097] The front end of the human pose estimation model is a convolutional feature extraction layer, which is used to extract human contours and local texture features from the input image; the middle part is a parallel multi-resolution feature branch, and the branches are fused through feature exchange units to simultaneously retain high-resolution spatial positioning information and deep semantic information; the output end adopts a heatmap regression method to output the corresponding two-dimensional probability heatmap and key point detection confidence for predefined human skeletal key points.

[0098] As one implementation method, predefined keypoints include standard human skeletal points such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. After analyzing the peak positions of the keypoint heatmap, the two-dimensional pixel coordinates of each keypoint in the current video frame can be obtained; combined with the depth value corresponding to the pixel position, the three-dimensional coordinates of the keypoint in the camera coordinate system can be recovered.

[0099] For the specific implementation of recovering three-dimensional coordinates from two-dimensional pixel coordinates and depth values, those skilled in the art can complete it based on the pinhole imaging model and depth camera calibration parameters. The specific process is well-known in the field and will not be elaborated here.

[0100] To ensure the applicability of the human pose estimation model in indoor early warning scenarios, during the model construction phase, supervised training is conducted using a labeled sample dataset that includes indoor monitoring perspectives, multi-pose human activities, and different occlusion conditions. The input data in the training samples are RGB-D images, and the label data consists of manually annotated positions of each human skeletal keypoint in the image coordinate system. During training, the mean squared error between the predicted heatmap and the actual keypoint heatmap is used as the loss function, and the model parameters are iteratively updated using a gradient descent optimization algorithm until the loss value converges or the validation set error reaches the preset convergence condition. After model deployment, the topology extraction module 200 inputs real-time depth video data frame by frame according to the video frame order and outputs the skeletal keypoint coordinates and corresponding confidence scores of each human target in the current frame.

[0101] After obtaining the skeletal keypoints, the topology extraction module 200 analyzes the spatial displacement characteristics of the target body in conjunction with the continuous frame tracking results. If no skeletal keypoints meeting the minimum confidence requirement are detected in the current video frame, abnormal action judgment is not performed, and the process directly proceeds to the next frame to avoid incorrect judgments due to missing keypoints. For stable human targets existing in multiple consecutive frames, the system... The three-dimensional displacement variance of each bone keypoint is calculated within a sliding time window. Taking a key skeletal point as an example, its three-dimensional displacement variance It can be represented as:

[0102] ;

[0103] in, Indicates the first The three-dimensional displacement variance of each skeletal keypoint within a time sliding window; This indicates the total number of consecutive video frames contained within the time-sliding window; Indicates the first The key skeletal points are at the first The three-dimensional spatial position vector in the frame; Indicates the first The average three-dimensional spatial position vector of each skeletal keypoint within the time sliding window; This represents the Euclidean norm. As a preferred method, The number of video frames is determined based on the video frame rate and the duration of the action, typically corresponding to 1 to 2 seconds, to cover a complete abnormal action cycle.

[0104] To determine whether a person is in a state of strenuous activity, the topology extraction module 200 statistically determines whether the key point detection confidence requirement is met and the displacement variance is greater than the first threshold. The percentage of skeletal keypoints. Let the number of keypoints satisfying the condition be... The total number of key points that meet the minimum detection confidence requirement is The proportion of abnormal actions It can be represented as:

[0105] ;

[0106] in, This indicates the percentage of abnormal actions taken by the current personnel target; This indicates that the displacement variance is greater than the first threshold. Number of key points; This represents the total number of valid key points involved in the determination. As a preferred method, when... Greater than the preset ratio threshold At that time, it was determined that the person was in a state of violent limb movement, and the target body was locked; when Not greater than At that time, it was determined to be a routine activity.

[0107] First threshold The value was determined based on the displacement statistics of normal walking and strenuous activity samples, and its preferred range is 0.05 square meters to 0.2 square meters; the proportion threshold... The target selection ratio is determined based on the distribution ratio of key points involved in vigorous movement under different types of dangerous behavior, preferably between 0.3 and 0.6. If multiple candidate targets that meet the criteria exist at the same time, they can be ranked according to the proportion of abnormal movements, target tracking stability, and average confidence of key points, with the highest-ranked target being selected as the current target body. This process avoids the problem of target selection uncertainty when multiple targets exist simultaneously.

[0108] After locking onto the target body, the topology extraction module 200 further generates a confidence level for visual dangerous behavior. As one implementation method, Based on the proportion of abnormal actions The determination is based on a comprehensive assessment of the average confidence level of valid keypoints, the number of frames with sustained abnormal actions, and the target tracking stability, and is normalized to a value range of 0 to 1. If there is severe missing keypoints, an unstable proportion of abnormal actions, or target tracking is interrupted in the current frame, then... Record it as invalid, or output it with a preset low confidence level, so as to avoid directly entering the subsequent fusion decision when there is insufficient visual information.

[0109] Furthermore, the topology extraction module 200 extracts the representative spatial position of the target body and transforms it from the camera coordinate system to the absolute three-dimensional world coordinate system. As a preferred embodiment, the representative spatial position is taken as the three-dimensional centroid position of the effective skeletal key points of the target body. Let the three-dimensional coordinates of the target body in the camera coordinate system be... After performing coordinate transformation on the camera intrinsic and extrinsic parameter matrices obtained by the depth camera device 10 in advance, the three-dimensional coordinates of the target body in the absolute three-dimensional world coordinate system are obtained. .in, This indicates the absolute spatial location of the target body within the indoor warning zone. These coordinates serve as a spatial index for subsequent processing, including warning zone matching, constructing sound wave propagation geometry, and calculating audio time windows.

[0110] After obtaining the three-dimensional world coordinates of the target body, the topology extraction module 200 establishes the spatial geometric channel between the target body and the omnidirectional microphone 20, which is used to approximately characterize the main spatial range of direct sound wave propagation. Let the fixed position of the omnidirectional microphone 20 in the absolute three-dimensional world coordinate system be... Then, based on the target body coordinates To the omnidirectional pickup at coordinate 20 The lines connecting the points form the direct sound propagation path. Around this propagation path, the topology extraction module 200 constructs a spatial geometric channel as the occlusion analysis range. As a preferred approach, the spatial geometric channel is approximated using a frustum or truncated cone, with its vertex located at the target body position and its central axis extending along the line connecting the target body to the omnidirectional pickup 20. The channel angle is determined based on the effective propagation range corresponding to the first Fresnel zone. By employing the aforementioned approximate geometric channel, obstructions that have a major impact on the direct sound propagation can be included in the subsequent analysis range while keeping computational complexity controllable.

[0111] After constructing the spatial geometric channel, the topology extraction module 200 reprojects the boundary of the spatial geometric channel onto the current depth image coordinate system, generating a corresponding two-dimensional image region mask, and extracting non-target occlusion groups within the mask range. To avoid mistaking the target body itself as an occlusion, the system combines the target body's three-dimensional bounding box, key point distribution range, and depth continuity to remove pixels belonging to the target body itself. For the remaining candidate pixel regions within the mask area, the topology extraction module 200 further filters them based on their depth value, connectivity, and spatial position relative to the sound wave propagation path, retaining only objects located between the target body and the omnidirectional microphone 20 and falling inside the spatial geometric channel as non-target occlusion groups. This processing avoids misjudging occlusions solely based on the depth relationship relative to the camera, thus making subsequent compensation parameters more consistent with the actual spatial relationship of sound wave propagation. If no non-target occlusions meeting the conditions are detected in the current frame, the number of occlusions is recorded as zero, and the corresponding occlusion feature is set to null or zero to ensure that subsequent compensation calculations can still be performed normally.

[0112] For the identified first For a non-target occluder, the topology extraction module 200 calculates its effective acoustic projection area. and depth distribution numerical .in, Indicates the first The actual physical projected area of ​​a non-target obstruction perpendicular to the direction of sound wave propagation. Indicates the first The straight-line distance in three-dimensional physical space from the geometric center point of a non-target occluder to the omnidirectional microphone 20. The effective acoustic projection area can be calculated based on the number of effective contour pixels of the occluder in the two-dimensional depth image and its corresponding average depth value, combined with the pinhole imaging model; depth distribution values. The distance is calculated using the Euclidean distance between the geometric center of the obstruction and the coordinates of the omnidirectional microphone 20.

[0113] Through the above processing, the topology extraction module 200 converts the personnel action information and environmental occlusion information in the depth video data into geometric feature quantities required for subsequent dynamic compensation of environmental acoustic parameters, thereby completing visual event detection and three-dimensional topology analysis.

[0114] See attached document Figure 5 In this embodiment, after obtaining the three-dimensional world coordinates of the target body, the topology extraction module 200 performs spatial partition mapping on the indoor warning area to determine the warning partition to which the current event belongs, and reads the parameter set corresponding to the warning partition from the indoor warning partition parameter table for the parameter compensation module 300 and the verification decision module 500 to call in subsequent processing.

[0115] The indoor early warning zone is partitioned during the initial system configuration phase. The edge computing server 30 spatially divides the indoor early warning zone based on room structure, access areas, equipment deployment locations, and security requirements, and generates an indoor early warning zone parameter table. As one implementation method, the ground activity area can be divided into several closed planar polygonal regions, which are then combined with the height range of personnel activity to form corresponding three-dimensional early warning zones. Let the total number of all early warning zones be... , No. The spatial area of ​​each early warning zone is denoted as ,but It can be represented as:

[0116] ;

[0117] in, Represents the coordinates of any point in a three-dimensional world coordinate system; Indicates the first Each early warning zone is a closed planar area on a horizontal ground surface; Indicates the first The lower boundary height of each early warning zone; Indicates the first The upper boundary height of each warning zone. As a preferred method, Ground height can be taken. The ceiling height or the effective monitoring height can be used. For closed planar areas... The establishment of the boundary can be determined based on the room floor plan, the calibration results of the depth camera 10, and on-site measurement data. For boundary vertex acquisition, boundary fitting, and determination of whether a point is inside the polygon, those skilled in the art can use conventional geometric modeling methods. The specific process is well-known in the field and will not be elaborated here.

[0118] After completing the spatial partitioning, the edge computing server 30 establishes a corresponding parameter set for each early warning partition. Let the first... The parameter set for each warning zone is: ,but It can be represented as:

[0119] ;

[0120] in, Indicates the first Environmental attenuation compensation weights corresponding to each early warning zone; Indicates the first The environmental tolerance value corresponding to each early warning zone; Indicates the first The fusion weights of visual modalities in each early warning zone; Indicates the first Fusion weights of audio modalities in each early warning zone; Indicates the first The joint early warning trigger threshold corresponding to each early warning zone; Indicates the first Risk level or response level identifier for each warning zone.

[0121] High-risk areas, stairwells, equipment room entrances, and areas prone to falls can be classified as higher-risk areas, while ordinary activity areas can be classified as lower-risk areas. The environmental tolerance values, fusion weights, and joint warning trigger thresholds for different warning zones can be determined based on historical sample data, unloaded environmental test data, and on-site calibration data. The statistical analysis, regression analysis, and threshold selection used in the zone parameter calibration can be implemented using conventional parameter calibration methods by those skilled in the art, and will not be elaborated upon here.

[0122] Topology extraction module 200 outputs the three-dimensional world coordinates of the target body. Then, the coordinates are compared with the spatial regions of each early warning zone. Perform a match. When At that time, it was determined that the target body was located at the first There are several warning zones. If the target is located near the boundary of an adjacent zone, a spatial buffer distance can be set for the zone boundary to avoid frequent jumps in the zone identifier in consecutive frames. Let the target coordinates be... The minimum distance between the boundaries of each early warning zone is The boundary buffer threshold is Then when At this time, the system maintains the partition determination result of the previous moment, or combines the tracking results of multiple consecutive frames to output a stable partition determination.

[0123] in, Indicates the target coordinates up to the 1st The minimum distance between the boundaries of each early warning zone; This represents the partition boundary buffer threshold, measured in meters. As a preferred method, The determination can be based on the ranging error of the depth camera 10, the target's movement speed, and the spacing between partition boundaries. If the same target moves across partitions within consecutive frames, the warning partition corresponding to the latest stability determination result is used as the current partition identifier, and this partition identifier is sent to the parameter compensation module 300 and the verification decision module 500. If the target coordinates fail to fall directly into any warning partition in individual frames, or simultaneously satisfy multiple partition boundary constraints, the determination can be made based on the valid partition identifier of the previous moment or the minimum boundary distance principle, thereby avoiding blank states during the partition mapping process.

[0124] See attached document Figure 6In this embodiment, the parameter compensation module 300 receives the geometric features of the non-target obstruction group output by the topology extraction module 200, and dynamically corrects the reference sound field features in the static mapping database in combination with the parameters associated with the corresponding early warning zone.

[0125] During indoor acoustic propagation, obstructions along the propagation path typically have a greater impact on the direct sound energy than on the overall distribution of the reverberant sound field. Therefore, when there are non-target obstructions between the target body and the omnidirectional pickup 20, the ratio of direct sound to reverberant sound energy measured at the receiver usually decreases. Accordingly, the parameter compensation module 300 converts the number, projected area, spatial location, and sound absorption characteristics of the obstructions into acoustic attenuation compensation amounts, and corrects the reference direct sound to reverberant sound energy ratio at the target location, thereby obtaining a dynamic theoretical sound mixing ratio corresponding to the current spatial topology.

[0126] The parameter compensation module 300 reads the total number of non-target occluders identified at the current moment. And the effective acoustic projection area corresponding to each non-target obstruction. and depth distribution numerical Considering that not all non-target obstructions in the propagation path are human bodies, the parameter compensation module 300 uses corresponding sound absorption coefficients for different types of obstructions.

[0127] Let the first The sound absorption coefficient corresponding to each non-target obstruction is: To avoid an abnormally amplified compensation value due to an excessively small denominator when the geometric center of the obstruction is too close to the omnidirectional microphone 20, a lower depth limit is set to... The corrected depth amount and spatial acoustic attenuation compensation factor They can be represented as:

[0128] ;

[0129] ;

[0130] in, This represents the spatial acoustic attenuation compensation factor, used to characterize the overall sound energy loss caused by non-target obstructions within the current direct sound propagation channel; This represents the total number of non-target occluders extracted within the spatial geometric channel; Indicates the sequence number of the non-target occluder; Indicates the first The effective acoustic projection area of ​​a non-target obstruction perpendicular to the direction of sound wave propagation, in square meters; Indicates the first The straight-line distance in three-dimensional physical space from the geometric center point of a non-target obstruction to the omnidirectional microphone 20, in meters; This represents the effective depth after introducing a lower bound constraint. This represents the lower limit of depth. As a preferred method, The value can be determined based on the installation distance of the omnidirectional microphone 20, the depth measurement resolution, and the near-field measurement stability, and its preferred range is 0.1 meters to 0.3 meters.

[0131] For sound absorption coefficient The determination of the first, if the first If a non-target occluder is a non-target human body, then The average sound absorption coefficient of the human body can be used. This coefficient can be obtained through pre-calibration in a noise-absorbing environment, based on the thickness of the clothing worn by people in the warning scenario and the dominant frequency band of indoor sound waves. The preferred value range is 0.3 to 0.6. If the... If the non-target obstruction is furniture, door, box, or other common indoor object, then The sound absorption coefficient of the corresponding material can be read from the preset material acoustic parameter table. If the type of occlusion cannot be stably distinguished within the current frame, the pre-calibrated default sound absorption coefficient can be called. Approximate calculations are performed to ensure continuous execution of compensation calculations. If the effective acoustic projection area of ​​a candidate obstruction... If the depth data is zero, missing, or the geometric center point cannot be stably determined, the candidate occluder will not be included in the summation term. If there are no non-target occluders satisfying the conditions within the current spatial geometric channel (i.e., N=0), then... =0. This avoids algorithm interruption in unobstructed scenes, invalid measurement scenes, and scenes where individual occlusion features are missing.

[0132] After obtaining the spatial acoustic attenuation compensation factor, the parameter compensation module 300 uses the target body's three-dimensional world coordinates output by the topology extraction module 200. As an index, the static mapping database established during the initial system configuration phase is accessed to extract the reference direct sound and reverberant sound energy ratio corresponding to the current target location. The parameter compensation module 300 simultaneously uses the current warning partition number sent by the topology extraction module 200. Read the environmental attenuation compensation weight corresponding to the warning zone from the indoor warning zone parameter table. The dynamic theoretical acoustic mixing ratio at the current moment is calculated by combining the aforementioned spatial acoustic attenuation compensation factor. The formula for its calculation is:

[0133] ;

[0134] in, Indicates the first The dynamic theoretical sound mixing ratio obtained under the constraint of each early warning zone; This represents the ratio of the direct sound energy to the reverberant sound energy at the target location reference, retrieved from the static mapping database. This indicates the environmental degradation compensation weight corresponding to the current warning zone; This represents the spatial acoustic attenuation compensation factor. If the corrected... If the output is lower than the minimum effective sound mixing ratio pre-calibrated by the system, the output will be based on the minimum effective sound mixing ratio to avoid abnormal values ​​that have no actual physical meaning under extreme obstruction conditions.

[0135] The above-mentioned processing yielded This represents the ratio of direct sound to reverberant sound energy that the omnidirectional pickup 20 should theoretically receive, under the combined constraints of the current target spatial location, current obstruction distribution, and the current acoustic environment of the warning zone, if the target body is the source of the current sudden sound. The parameter compensation module 300 sends this dynamic theoretical sound mixing ratio to the verification decision module 500 as a reference benchmark for subsequent cross-modal spatial consistency verification.

[0136] See attached document Figure 7 In this embodiment, the audio alignment module 400 receives the target body's three-dimensional world coordinates and visual action absolute timestamps output by the topology extraction module 200, and determines the audio analysis time window corresponding to the current visual event in the continuously input audio data stream based on the physical propagation delay between the target position and the omnidirectional microphone 20, thereby extracting effective audio segments for acoustic parameter calculation.

[0137] The absolute timestamp of the visual action here is This represents the absolute timestamp of the keyframe when the target body is locked and the abnormal action judgment condition is met. As a preferred approach, when multiple consecutive frames meet the abnormal action judgment condition, the timestamp of the frame where the proportion of abnormal actions first exceeds the judgment threshold can be taken as the timestamp. Alternatively, take the timestamp of the keyframe with the highest proportion of abnormal actions as... .

[0138] The audio alignment module 400 aligns the target body's three-dimensional world coordinates. and the fixed three-dimensional coordinates of the omnidirectional pickup 20 Calculate the physical straight-line distance between the target location and the omnidirectional microphone 20. The formula for its calculation is:

[0139] ;

[0140] in, The distance from the target sound-emitting location to the receiver of the omnidirectional microphone 20 is expressed in meters. Represents the three-dimensional world coordinates of the target body; This represents the fixed three-dimensional coordinates of the omnidirectional microphone 20. Based on the physical straight-line distance, the audio alignment module 400 further calculates the airborne physical transmission time of the direct sound wave. The formula for its calculation is:

[0141] ;

[0142] in, Indicates the physical transmission time of a direct sound wave, measured in seconds; This represents the speed of sound in air. As a preferred method, The speed of sound is taken as 340 m / s under standard ambient temperature. When the system is equipped with an indoor temperature sensor, the air velocity can also be corrected based on the real-time temperature to reduce the impact of ambient temperature fluctuations on the propagation delay calculation results.

[0143] After obtaining the propagation delay, the audio alignment module 400 combines the visual action absolute timestamp. Construct the start and end boundaries of the audio analysis time window. Start boundary of the audio time window. It can be represented as:

[0144] ;

[0145] in, Indicates the starting boundary time of the audio analysis time window; This indicates the safety redundancy prefix time, used to prevent the rising edge of the direct sound from being truncated. As a preferred method, The value ranges from 5 milliseconds to 15 milliseconds.

[0146] To ensure that the extracted audio clips not only include the direct main lobe but also cover early reflections and the effective segments used for subsequent reverberation statistical analysis, the audio alignment module 400 uses the target body's three-dimensional world coordinates as an index to extract the early reflection delay time corresponding to that spatial location from the aforementioned static mapping database. And based on this, determine the cutoff boundary of the audio time window. . It can be represented as:

[0147] ;

[0148] in, Indicates the cutoff boundary time of the audio analysis time window; This represents the early reflected acoustic delay time of the target location retrieved from the static mapping database; This indicates the length of the analysis window, used to cover the reverberation attenuation statistical section. As a preferred method, The value is set based on the average reverberation time in the room, ranging from 200 milliseconds to 500 milliseconds.

[0149] If the target location does not coincide with the sampled coordinates in the static mapping database, the aforementioned spatial interpolation method or nearest neighbor coordinate matching method can be used to obtain the corresponding coordinates. .like If the readable starting point is earlier than the current circular buffer queue, then the readable starting point is used as the actual interception starting point; if If the end point of the current buffer queue is written later, parameter parsing will be performed after the corresponding audio data is written, in order to avoid the current time window becoming invalid due to insufficient buffer boundaries.

[0150] The raw audio stream continuously acquired by the omnidirectional microphone 20 is temporarily stored in a circular buffer queue of the edge computing server 30. The audio alignment module 400, based on... and The corresponding audio segment is extracted from the circular buffer queue, and a validity screening is performed before blind reverberation evaluation. Let the discrete sampling sequence of the extracted audio segment be... The total number of sampling points for the segment is The average short-time energy of the audio segment is... It can be represented as:

[0151] ;

[0152] in, This represents the average short-time energy of the extracted audio segment; Indicates the first The amplitude of each sampling point; This represents the total number of sampling points involved in the calculation.

[0153] Will Compared with background noise level threshold When comparing, If, at that time, it is determined that there is no valid sound sufficient to support the calculation of acoustic parameters within the current time window, the blind reverberation evaluation process for the current time window is terminated, and an audio invalid flag is output. The background noise level threshold is... The level can be determined based on the mean or root mean square level of background audio energy collected during the system's idle operation phase. As a preferred method, The energy threshold can be set to correspond to a background noise mean of 3 to 10 dB. This avoids performing invalid calculations on silent segments, low signal-to-noise ratio segments, or segments with occasional acquisition anomalies.

[0154] when At that time, the audio alignment module 400 performs blind reverberation evaluation on the audio segment to estimate the direct sound energy and reverberant sound energy reaching the receiver. As one implementation, the audio segment is first framed and windowed to extract the energy envelope of each frame; then, based on the rapid rise portion of the beginning of the audio segment and the attenuation envelope of the end segment, the direct sound dominant section and the reverberation attenuation section are divided; subsequently, the reverberation attenuation slope is fitted in the logarithmic energy domain, or the maximum likelihood estimation method is used to separately estimate the direct sound energy and reverberant sound energy. Let the estimated direct sound energy be... The reverberant energy is The actual sound mixing ratio It can be represented as:

[0155] ;

[0156] in, Indicates the actual sound mixing ratio; This represents the estimated direct sound energy; This represents the estimated reverberant energy. If the blind reverberation assessment process encounters insufficient effective frames, estimation residuals exceeding a preset threshold, or... If the value approaches zero, the estimated result of the actual sound mixing ratio for the current time window is deemed invalid, and an invalid audio flag is output to prevent abnormal estimates from being passed into the subsequent verification decision process.

[0157] Output actual sound mixing ratio Simultaneously, the audio alignment module 400 can also generate audio anomaly confidence scores. Audio anomaly confidence level The energy excess of the extracted audio segment relative to the background noise, the stability of the blind reverberation assessment results, and the effective sound duration can be comprehensively determined and normalized to a numerical range of 0 to 1. Therefore, the output of the audio alignment module 400 includes not only the actual sound mixing ratio used for spatial consistency verification. It also includes audio anomaly confidence levels that can be used for subsequent fusion and determination. .

[0158] Through the above processing, the start and end positions of the audio analysis time window are established in correspondence with the target spatial position, sound wave propagation delay, and early indoor reflection characteristics, so that the audio segments used for subsequent cross-modal verification are consistent with the current visual event in the time dimension.

[0159] See attached document Figure 8In this embodiment, the verification decision module 500 receives the dynamic theoretical sound mixing ratio output by the parameter compensation module 300, the actual sound mixing ratio output by the audio alignment module 400, and the audio anomaly confidence level. It then combines this with the visual hazard behavior confidence level output by the topology extraction module 200 to perform cross-modal spatial consistency verification and fusion decision-making for the current event. The purpose is to determine whether the abnormal action locked by the visual side and the valid sound detected by the audio side originate from the same physical space event, thereby filtering out interference signals that coincidentally overlap in time but are spatially separated.

[0160] After obtaining the output of the preceding module, the verification decision module 500 calculates the dynamic theoretical acoustic mixing ratio under the current warning zone. Mixing with actual sound Deviation of spatial physical parameters between The formula for its calculation is:

[0161] ;

[0162] in, Indicates the first The deviation of spatial physical parameters within each warning zone; The parameter compensation module 300 indicates that the parameter compensation module is in the first stage. The dynamic theoretical sound mixing ratio output under the constraints of each early warning zone; This indicates the actual sound mix ratio estimated by the audio alignment module 400 based on the aligned audio segments. Because... It reflects the degree of deviation between the visually inferred sound field parameters and the measured audio sound field parameters. Therefore, the smaller the value, the stronger the correspondence between the current visual event and the audio event in physical space.

[0163] The verification decision module 500 reads the environmental tolerance value corresponding to the current warning zone from the indoor warning zone parameter table. and will and The comparison is performed. If any necessary input is missing at the current moment, including invalid three-dimensional world coordinates of the target body, failure to generate dynamic theoretical sound mix ratio, failure to estimate actual sound mix ratio, lack of confidence in visual dangerous behavior, or lack of confidence in audio anomaly, the verification decision module 500 directly outputs a blocking flag to terminate the joint early warning process of the current time window, so as to avoid generating incorrect decisions under incomplete information conditions.

[0164] when When this occurs, the verification decision module 500 determines that the current visual modality and audio modality do not meet the consistency condition in physical space and classifies the event as stray interference. This situation indicates that the abnormal action captured in the visual image and the sound detected in the audio channel do not have sufficient spatial homology, and may originate from off-screen sounds, occasional impact sounds in adjacent areas, or other interference events triggered by non-target locations. In this case, the verification decision module 500 blocks the joint early warning command output within the current event time window and performs cleanup or reset processing on the current cache state, causing the system to return to continuous monitoring state.

[0165] when At this point, the verification decision module 500 determines that the current visual modality and audio modality meet the spatial consistency condition and enters the fusion decision process. Based on this, the verification decision module 500 calls the visual modality fusion weight corresponding to the current warning partition. Audio modality fusion weights and joint early warning trigger threshold Confidence level for visually dangerous behaviors and audio anomaly confidence Perform fusion calculations. Joint early warning score. It can be represented as:

[0166] ;

[0167] in, Indicates the first Joint early warning scores under each early warning zone; This represents the confidence level of the visual hazardous behavior output by the topology extraction module 200. This indicates the confidence level of audio anomalies output by the audio alignment module 400; and These represent the fusion weights of the visual and audio modalities in the current warning zone, respectively. As one implementation method, and All values ​​were normalized to the range of 0 to 1 before participating in the fusion calculation, and .

[0168] Verification decision module 500 will Joint early warning trigger threshold corresponding to the current early warning zone Comparison. When When, generate a formal early warning instruction; when In this case, only the current event record is retained, without triggering a formal alarm. Thus, spatial consistency determination is used to filter out non-similar interference, and joint early warning score is used to determine whether the current homogeneous event reaches the preset alarm intensity. The two respectively bear consistency constraints and alarm triggering constraints in the decision-making process.

[0169] Formal early warning commands may include the current early warning zone number, the risk level of the early warning zone, the three-dimensional world coordinates of the target body, the event timestamp, the dynamic theoretical sound mix ratio, the actual sound mix ratio, the confidence level of visual dangerous behavior, the confidence level of audio anomalies, and the corresponding audiovisual data segments. Upon receiving the early warning command, the security monitoring management platform can execute differentiated responses based on the early warning zone number and risk level. As one implementation method, high-risk zones correspond to audio-visual alarms and platform pop-up windows, medium-risk zones correspond to platform prompts and video recording markers, and ordinary zones correspond to event archiving. Through the above processing, the visual modality and audio modality form a complete closed loop in the physical space dimension and the judgment intensity dimension, thereby achieving cross-modal dangerous behavior early warning decision-making corresponding to spatial location.

[0170] Furthermore, to aid in understanding the practical application and technical effects of the present invention, the present invention will be further described below in conjunction with specific application embodiments in a campus setting.

[0171] See attached document Figure 9 In this embodiment, the warning area is located on the east side of the staircase platform on the second floor of a middle school teaching building and its adjacent short corridor. The local warning area is approximately 4.8 meters long and 3.6 meters wide, including the staircase exit, the central area of ​​the staircase platform, and the short corridor transition area connecting to the platform. This location experiences high pedestrian traffic during breaks and get out of class times, posing risks of both normal student activities such as walking, talking, and lingering, as well as dangerous behaviors such as pushing, chasing, falling, and colliding with railings. Therefore, it is suitable as a typical deployment scenario for the present invention.

[0172] In this embodiment, the depth camera 10 is installed above the stair platform against the wall at a height of 2.7 meters, with a downward viewing angle of approximately 28°, and its field of view can completely cover the local warning area. The omnidirectional microphone 20 is installed in the middle of the top surface of the local warning area at a height of 2.8 meters, and is used to collect environmental acoustic signals in this area. The edge computing server 30 is located in the low-voltage room on the same floor and is connected to the depth camera 10 and the omnidirectional microphone 20 via a wired network. After the system is deployed, indoor acoustic calibration, spatial coordinate system establishment, static mapping database generation, and warning zone parameter initialization are first completed under open field conditions.

[0173] Based on the spatial location and risk characteristics of the local warning area, the area is further divided into three warning zones. Zone 1 is the area connecting the stair treads and the platform; Zone 2 is the central area of ​​the stair platform; and Zone 3 is the short corridor transition area outside the platform. Zone 1 corresponds to the high-risk zone, Zone 2 to the medium-risk zone, and Zone 3 to the conventional-risk zone. During system initialization, the edge computing server 30 establishes an indoor warning zone parameter table and configures environmental attenuation compensation weights, environmental tolerance values, visual modal fusion weights, audio modal fusion weights, and joint warning trigger thresholds for each of the three zones. As an implementation approach, Zone 1 uses a smaller environmental tolerance value and a lower joint warning trigger threshold to improve the warning sensitivity to events such as pushing, falls, and railing collisions; Zone 3 uses a relatively larger environmental tolerance value and a higher joint warning trigger threshold to reduce false triggers caused by ordinary passage, short-term stops, and minor noise. Therefore, the "zonal early warning" in this embodiment is manifested as follows: within the same local early warning area, different zoning parameters are used for subsequent parameter compensation, spatial consistency verification and fusion decision based on the risk level and environmental characteristics of different spatial sub-regions, and the corresponding zoning is identified in the final output.

[0174] During the trial operation, the depth camera 10 continuously collects RGB-D video data within the local warning area, and the omnidirectional microphone 20 continuously collects audio data within the area. The edge computing server 30 processes the two types of data synchronously. Taking a real-world example of a student falling during a push-and-shove incident during the break between the second and third periods in the morning, the system first detects a significant increase in the displacement variance of key points on the student's torso, hips, and knees in consecutive video frames. The proportion of abnormal movements exceeds a preset threshold, and the system then identifies the student as the target body and maps its spatial location to a zone. Simultaneously, a student passing by in parallel is identified as a non-target occlusion in the propagation path and its occlusion effect is factored into the parameter compensation process. The audio alignment module 400 extracts the corresponding audio segment from the circular buffer queue based on the absolute timestamp of the visual action and the propagation delay from the target position to the omnidirectional microphone 20. This audio segment contains a distinct impact sound and a short reverberation tail. After completing the spatial consistency check, the verification decision module 500 further combines the confidence levels of visual dangerous behavior and audio anomalies for a fusion judgment, and outputs a formal warning result with a zone-one identifier. Upon manual review, the incident was confirmed to be a genuine fall, and the system output result was consistent with the actual area where the incident occurred.

[0175] In this embodiment, a single depth camera 10 and a single omnidirectional microphone 20 correspond to a local warning area, which is further divided into multiple warning zones. If the campus needs to cover multiple stair platforms, multiple floor nodes, or multiple public areas of teaching buildings, multiple depth cameras 10 and omnidirectional microphones 20 can be deployed to correspond to multiple local warning areas. Within each local warning area, partition mapping, dynamic theoretical sound mixing ratio correction, audio time window alignment, spatial consistency verification, and fusion decision are performed according to the scheme of this invention. Therefore, the technical advantage of this invention does not rely on a single device covering a large area, but is reflected in the fact that within a single local warning area, differentiated parameter calls and differentiated warning outputs can be achieved based on the risk level and environmental characteristics of different spatial sub-regions, and can be extended to multiple local warning areas with the same processing mechanism.

[0176] To further verify the practical effectiveness of the invention in a campus setting, a two-week trial run was conducted in the aforementioned teaching building's local warning area. The test period covered peak activity times such as before school arrival, between classes, before and after lunch break, and before dismissal in the afternoon, totaling 132 hours of operation. During the trial run, 412 valid event samples were collected and manually verified, including 102 dangerous events and 310 non-dangerous events. Dangerous events included 12 instances of pushing at stairwells, 26 instances of falling on platforms, 18 instances of collisions between platforms and railings, and 46 instances of chasing and bumping. Non-dangerous events included normal walking, conversations during breaks, going up and down stairs, dropping backpacks, ordinary door opening and closing, and multiple people walking side-by-side. All event samples retained video clips, audio clips, event times, and manual verification tags.

[0177] See attached document Figure 10 In this embodiment, to facilitate the evaluation of the effectiveness of the present invention, two sets of comparative schemes are set up. Comparative scheme A only uses the visual abnormal action detection results for alarm determination; comparative scheme B adds audio abnormality detection on the basis of visual abnormal action detection, but does not perform warning zone mapping, dynamic theoretical sound mixture ratio correction, and spatial consistency verification. The present invention, as scheme C, further introduces warning zone parameters, dynamic theoretical sound mixture ratio correction, audio time window alignment, and cross-modal spatial consistency verification on the basis of visual abnormal action detection and audio abnormality detection. Using the results of manual review as a benchmark, the accuracy, false alarm rate, false negative rate, and F1 score of the three schemes on 412 event samples are statistically analyzed.

[0178] Experimental results show that, compared to scheme A, a total of 126 warnings were triggered, with 89 correct warnings, 37 false alarms, and 13 missed warnings, achieving an accuracy rate of 87.9%, a false alarm rate of 11.9%, a missed alarm rate of 12.7%, and an F1 score of 0.781. Scheme B triggered 112 warnings, with 93 correct warnings, 19 false alarms, and 9 missed warnings, achieving an accuracy rate of 93.2%, a false alarm rate of 6.1%, a missed alarm rate of 8.8%, and an F1 score of 0.869. The proposed solution triggered 104 warnings, with 97 correct warnings, 7 false alarms, and 5 missed warnings, achieving an accuracy rate of 97.1%, a false alarm rate of 2.3%, a missed alarm rate of 4.9%, and an F1 score of 0.942. These results demonstrate that by introducing partitioning parameters and spatial consistency verification, the system's ability to identify real dangerous events and suppress non-dangerous events is improved.

[0179] In terms of the sources of false alarms, the false alarms of Comparative Solution A mainly occur in scenarios such as running quickly during breaks, multiple people converging, and backpacks falling to the ground; the false alarms of Comparative Solution B mainly occur in scenarios where visually unusual movements coincide with occasional impact sounds from adjacent areas; the few false alarms of the present invention mainly occur in situations where multiple people are gathered, there is significant mutual obstruction, and strong instantaneous noise is present. Therefore, it is evident that the present invention has a better ability to distinguish non-dangerous events in complex break-time environments.

[0180] See attached document Figure 11 In this embodiment, the trial run samples were divided into three categories based on the number of non-target obstructions between the target body and the omnidirectional microphone 20: no obstruction, single-person obstruction, and multiple-person obstruction. There were 156 no-obstruction event samples, 143 single-person obstruction event samples, and 113 multiple-person obstruction event samples. Statistical results show that the accuracy rates of the proposed solution in the three scenarios are 96.8%, 95.1%, and 93.7%, respectively; while the accuracy rates of the comparative solution B in the three scenarios are 92.4%, 89.8%, and 85.6%, respectively. As the number of non-target obstructions in the propagation path increases, the recognition performance of the comparative solution B decreases significantly, while the proposed solution, due to the introduction of spatial acoustic attenuation compensation factors and zonal environmental compensation parameters, can still maintain good recognition stability in scenarios such as crowded passage during breaks.

[0181] To further verify the ability of the present invention to suppress stray interference, an additional interference scenario was set up during the test, in which abnormal movements occurred within the screen while simultaneously causing sounds of tables and chairs being dragged, doors slamming, and objects falling outside the local warning area. A total of 31 typical stray interference event samples were collected. Statistical results show that compared to Scheme B, this invention falsely triggered 9 such events, while the present invention falsely triggered 2. This demonstrates that the present invention, by utilizing the spatial consistency constraint between the dynamic theoretical sound mix ratio and the actual sound mix ratio, can distinguish between campus interference events that overlap in time but do not have spatial homogeneity, thereby improving the reliability of the system in actual campus environments.

[0182] See attached document Figure 12 In this embodiment, the distribution of spatial physical parameter deviations corresponding to real dangerous events and stray interference events is statistically analyzed. The results show that the spatial physical parameter deviations corresponding to real dangerous events are mainly concentrated in a smaller range, while the spatial physical parameter deviations corresponding to stray interference events are generally larger, indicating a significant difference in distribution between the two types of events. Therefore, by introducing spatial consistency verification, the present invention can provide additional physical causal constraints for early warning of dangerous behaviors on campus, improving the reliability of the early warning results.

[0183] Furthermore, the zoning statistics show that Zone 1 recorded 41 hazardous events, with 40 correct warnings; Zone 2 recorded 33 hazardous events, with 31 correct warnings; and Zone 3 recorded 28 hazardous events, with 26 correct warnings. This indicates that after adopting the zoning parameters, the system exhibits high warning sensitivity in the area connecting stair treads and platforms, while maintaining a low false alarm level in the short corridor transition area outside the platform. This demonstrates that the zoning warning system in this invention is not merely used to identify the location of events, but actually participates in the differentiated application of environmental tolerance values, fusion weights, and alarm thresholds.

[0184] In summary, in the practical application scenario of a local public area in a school teaching building, the solution of this invention can provide zoned early warning for dangerous behaviors such as pushing at stairwells, student falls, and platform collisions. Moreover, it still has good recognition stability and false alarm control capabilities even under conditions of obstruction, reverberation, and interference from outside the area, and has strong practical application value.

[0185] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A campus hazardous behavior zoning early warning system based on multimodal perception, comprising a depth camera, an omnidirectional microphone, and an edge computing server; wherein the depth camera and the omnidirectional microphone are installed within the campus early warning area and are respectively communicatively connected to the edge computing server; characterized in that, The edge computing server is deployed with: The acoustic calibration module is used to construct a static mapping database between three-dimensional spatial coordinates and reference sound field characteristics; The topology extraction module is used to lock the target body and extract the absolute timestamp of visual action based on the real-time depth video data collected by the depth camera device, calculate the three-dimensional world coordinates of the target body, extract the non-target occlusion features between the target body and the omnidirectional microphone and generate the confidence level of visual dangerous behavior, and determine the warning zone to which the target body belongs in the pre-established warning zone parameter table. The parameter compensation module is used to combine the features of the non-target obstruction and the associated parameters corresponding to the current warning zone to correct the reference sound field features corresponding to the target position in the static mapping database and output the dynamic theoretical sound mixing ratio. The audio alignment module is used to calculate the airborne physical transmission time of the direct sound wave based on the three-dimensional world coordinates of the target body, combine the absolute timestamp of the visual action to extract the corresponding audio time segment from the audio data collected by the omnidirectional microphone, and extract the actual sound mixing ratio value and audio anomaly confidence. The verification decision module is used to calculate the spatial physical parameter deviation between the dynamic theoretical sound mixing ratio and the actual sound mixing ratio. When the spatial physical parameter deviation meets the environmental tolerance value condition corresponding to the warning zone, the confidence level of the visual dangerous behavior and the confidence level of the audio anomaly are fused and calculated, and a warning command is output when the joint warning score reaches the trigger threshold.

2. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 1, characterized in that, The specific process by which the acoustic calibration module constructs the static mapping database is as follows: The campus early warning area is divided into grids to obtain multiple spatial grid sampling nodes, and room impulse response signals are collected at each spatial grid sampling node. Calculate the reference direct sound to reverberant sound energy ratio corresponding to each spatial grid sampling node, and calculate the time difference between the first effective local peak time after the main lobe of the direct sound and the initial arrival time of the direct sound, as the early reflection delay time; A spatial mapping relationship is established between the three-dimensional spatial coordinates of each spatial grid sampling node and the corresponding reference direct sound to reverberation energy ratio and early reflection delay time, forming the static mapping database.

3. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 1, characterized in that, The specific process by which the topology extraction module extracts features of non-target occluders is as follows: A spatial geometric channel is constructed with the line connecting the three-dimensional world coordinates of the target body to the three-dimensional coordinates of the omnidirectional microphone as the central axis; The spatial geometric channel is reprojected onto a two-dimensional depth image to generate an image region mask, and object objects located between the target body and the omnidirectional microphone and falling inside the spatial geometric channel are selected as a group of non-target occluders. The effective acoustic projection area and depth distribution values ​​of each non-target occluder in the non-target occluder group are extracted as the non-target occluder features.

4. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 3, characterized in that, The specific process by which the parameter compensation module outputs the dynamic theoretical sound mixing ratio is as follows: Substitute the effective acoustic projection area and depth distribution values ​​of each non-target obstruction into the spatial loss attenuation model, and combine them with the sound absorption coefficients corresponding to the non-target obstructions to calculate the spatial acoustic attenuation compensation factor. Extract the environmental attenuation compensation weight corresponding to the current warning zone from the warning zone parameter table, and extract the reference direct sound and reverberant sound energy ratio corresponding to the target body position from the static mapping database; The dynamic theoretical sound mixing ratio is output after numerical correction by subtracting the product of the environmental attenuation compensation weight and the spatial acoustic attenuation compensation factor from the ratio of the reference direct sound to the reverberant sound energy.

5. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 2, characterized in that, The specific process by which the audio alignment module extracts the corresponding audio time segment is as follows: The physical straight-line distance is calculated based on the three-dimensional world coordinates of the target body and the three-dimensional coordinates of the omnidirectional microphone. The physical straight-line distance is then divided by the speed of sound in air to obtain the physical transmission time in air. The starting boundary time of the audio time segment is calculated by adding the absolute timestamp of the visual action to the physical transmission time of the air and subtracting the preset safety redundancy prefix time. The early reflection delay time of the corresponding position is extracted from the static mapping database. The early reflection delay time and the preset analysis window length are added to the starting boundary time to calculate the cutoff boundary time of the audio time segment.

6. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 1, characterized in that, The specific process by which the audio alignment module extracts the actual sound mixing ratio and audio anomaly confidence level is as follows: Calculate the average short-time energy of the extracted audio segment. When the average short-time energy is greater than or equal to the background noise level threshold, perform blind reverberation assessment on the audio segment and separate the estimated direct sound energy and reverberant sound energy. Based on the estimated direct sound energy and reverberant sound energy, the energy ratio of direct sound to reverberant sound is calculated in the logarithmic energy domain to obtain the actual sound mixing ratio value; at the same time, the audio anomaly confidence level is generated based on the degree of energy transcendence of the extracted audio segment relative to the background noise and the stability of the blind reverberation assessment results.

7. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 1, characterized in that, The specific process of the verification decision module performing verification and fusion is as follows: Subtract the actual sound mixing ratio from the dynamic theoretical sound mixing ratio to obtain the spatial physical parameter deviation. When the deviation of the spatial physical parameters is greater than the environmental tolerance value corresponding to the warning zone, it is determined that the visual modality and the audio modality do not meet the consistency condition in physical space, the event is classified as scattered interference and the current warning triggering process is terminated. When the deviation of the spatial physical parameters is less than or equal to the environmental tolerance value corresponding to the warning zone, the confidence of the visual dangerous behavior and the confidence of the audio anomaly are weighted and summed according to the visual modality fusion weight and audio modality fusion weight corresponding to the current warning zone, and the joint warning score is output.

8. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 1, characterized in that, The specific configuration of the early warning partition parameter table is as follows: Record the spatial boundary data of each warning zone; and under the identifier of each warning zone, bind and store the corresponding environmental attenuation compensation weight, environmental tolerance value, visual modal fusion weight, audio modal fusion weight and joint warning trigger threshold; The topology extraction module determines the warning zone to which the target body belongs by matching the three-dimensional world coordinates of the target body with the spatial boundary data of each warning zone.

9. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 8, characterized in that, When the minimum distance from the three-dimensional world coordinates of the target body to the boundary of the adjacent warning zone is less than the preset zone boundary buffer threshold, the topology extraction module maintains the zone determination result of the previous moment as the current zone determination output.

10. The campus dangerous behavior zoning early warning system based on multimodal perception according to claim 3, characterized in that, The specific process of the topology extraction module locking the target body and generating the confidence score of visual dangerous behavior is as follows: deploying a human posture estimation model to extract skeletal key points from real-time depth video data, and calculating the three-dimensional displacement variance of each skeletal key point within a time sliding window; The proportion of abnormal movements is obtained by statistically analyzing the percentage of skeletal keypoints with a three-dimensional displacement variance greater than the first threshold. When the proportion of abnormal actions exceeds a preset proportion threshold, the current object is locked as the target body and the confidence level of the visual dangerous behavior is generated. Furthermore, after generating the image region mask, the topology extraction module combines the target body's three-dimensional bounding box, key point distribution range, and depth continuity features to directly remove pixels belonging to the target body itself within the mask.

Images