A multi-modal security warning method and system

Abstract

The present invention discloses a multimodal-based security warning method and system, which relates to the technical field of behavior recognition. By acquiring audio and video data of the target area and preprocessing the audio and video data, video feature data and audio feature data are obtained; the video feature data is used as the input of the pre-trained behavior recognition model to obtain a first detection result; the audio feature data is used as the input of the pre-trained speech recognition model to obtain a second detection result; if the first detection result is abnormal or the second detection result is abnormal, a warning message is generated to prompt the staff to pay attention to the situation in the area. By automatically analyzing the audio and video data, abnormal behaviors can be quickly screened out, avoiding possible negligence in manual monitoring; and the warning message prompts the staff to handle it in time to avoid further deterioration of the security incident, greatly shortening the response time, improving the timeliness of event processing, and improving the overall efficiency of the security system.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior recognition, and in particular to a multi-modal security warning method and system. Background Art

[0002] With the development of society and the advancement of science and technology, public safety issues are receiving increasing attention. Whether it is a home, commercial place, or public area, security systems are playing a vital role. Traditional security monitoring mainly relies on video surveillance equipment to achieve real-time monitoring of the target area, and monitoring personnel discover potential security threats by monitoring video images. Through monitoring, security personnel can observe the video images in real time, identify and judge potential abnormal behaviors. Especially in some scenes with frequent dynamic changes and complex information, security personnel can use experience and intuition to promptly identify and intervene in abnormal behaviors.

[0003] However, with the increase in security needs and the expansion of monitoring areas, relying solely on manual monitoring is far from meeting the security requirements of modern society. Traditional manual monitoring methods are inefficient, and security personnel cannot carefully check every monitoring screen. Even experienced staff will face problems such as decreased attention and fatigue during long-term monitoring tasks, resulting in the inability to detect and deal with abnormal behavior or dangerous situations in a timely manner. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of low efficiency mentioned in the above background technology and to propose a multi-modal security warning method and system.

[0005] In a first aspect of the present invention, a multi-modal security early warning method is provided, the method comprising:

[0006] Obtain audio and video data and light intensity of the target area;

[0007] Preprocessing the audio and video data to obtain video feature data and audio feature data;

[0008] If the light intensity is greater than a preset threshold, the video feature data is used as an input of a pre-trained behavior recognition model to obtain a first detection result; the first detection result includes normal and abnormal;

[0009] Using the audio feature data as an input of a pre-trained speech recognition model to obtain a second detection result; the second detection result includes normal and abnormal;

[0010] If the first detection result is abnormal or the second detection result is abnormal, a warning message is generated.

[0011] Optionally, the audio and video data includes video signal data and audio signal data; and the preprocessing of the audio and video data to obtain video feature data and audio feature data includes:

[0012] According to a preset resolution, each frame of the video signal data is resized to obtain video feature data;

[0013] The audio signal data is denoised according to a preset bandpass filter to obtain audio feature data.

[0014] Optionally, the behavior recognition model is an improved SlowFast model; the original SlowFast model includes a Slow branch and a Fast branch; each branch includes a convolution layer, a pooling layer and four residual blocks; the specific improvements include:

[0015] A hybrid attention module is embedded in each branch of the original SlowFast model; the calculation formula of the hybrid attention module is:

[0016]

[0017] Where x is the input of the hybrid attention module; f maxpool represents the maximum pooling; f avgpool represents average pooling; represents the convolution operation; σ is the sigmoid function; W1 is the channel temporal attention weight; Represents element multiplication; concat represents channel concatenation; W2 is the spatial-temporal attention weight; y1 is the feature after channel-temporal attention weighting; y2 is the feature after spatial-temporal attention weighting, and is also the output of the hybrid attention module.

[0018] Optionally, each residual block includes multiple bottleneck structures, each bottleneck structure consists of three convolutional layers; the embedding position of the hybrid attention module includes:

[0019] The first residual block of the Slow branch is between the second convolution layer and the third convolution layer of each bottleneck structure;

[0020] The second residual block of the Slow branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure;

[0021] The first residual block of the Fast branch is between the second convolution layer and the third convolution layer of each bottleneck structure;

[0022] The second residual block of the Fast branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure.

[0023] Optionally, the method further includes:

[0024] If the illumination intensity is not greater than the preset threshold, performing gamma correction on the video feature data to obtain enhanced video feature data;

[0025] The enhanced video feature data is used as input to the pre-trained behavior recognition model to obtain a first detection result.

[0026] A second aspect of the present invention provides a multi-modal security warning system, the system comprising:

[0027] A data acquisition module is used to obtain audio and video data and light intensity of the target area;

[0028] A preprocessing module, used for preprocessing the audio and video data to obtain video feature data and audio feature data;

[0029] A video recognition module, configured to use the video feature data as an input of a pre-trained behavior recognition model to obtain a first detection result if the light intensity is greater than a preset threshold; the first detection result includes normal and abnormal;

[0030] An audio recognition module, used to use the audio feature data as an input of a pre-trained speech recognition model to obtain a second detection result; the second detection result includes normal and abnormal;

[0031] The early warning module is used to generate early warning information if the first detection result is abnormal or the second detection result is abnormal.

[0032] Optionally, the audio and video data includes video signal data and audio signal data; and the preprocessing module includes:

[0033] A video processing module, used for scaling each frame of the video signal data according to a preset resolution to obtain video feature data;

[0034] The audio processing module is used to denoise the audio signal data according to a preset bandpass filter to obtain audio feature data.

[0035] Optionally, the behavior recognition model is an improved SlowFast model; the original SlowFast model includes a Slow branch and a Fast branch; each branch includes a convolution layer, a pooling layer and four residual blocks; the specific improvements include:

[0036] A hybrid attention module is embedded in each branch of the original SlowFast model; the calculation formula of the hybrid attention module is:

[0037]

[0038] Where x is the input of the hybrid attention module; f maxpool represents the maximum pooling; f avgpool represents average pooling; represents the convolution operation; σ is the sigmoid function; W1 is the channel temporal attention weight; Represents element multiplication; concat represents channel concatenation; W2 is the spatial-temporal attention weight; y1 is the feature after channel-temporal attention weighting; y2 is the feature after spatial-temporal attention weighting, and is also the output of the hybrid attention module.

[0039] Optionally, each residual block includes multiple bottleneck structures, each bottleneck structure consists of three convolutional layers; the embedding position of the hybrid attention module includes:

[0040] The first residual block of the Slow branch is between the second convolution layer and the third convolution layer of each bottleneck structure;

[0041] The second residual block of the Slow branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure;

[0042] The first residual block of the Fast branch is between the second convolution layer and the third convolution layer of each bottleneck structure;

[0043] The second residual block of the Fast branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure.

[0044] Optionally, the system further comprises a brightness enhancement module, wherein:

[0045] The brightness enhancement module is used to perform gamma correction on the video feature data to obtain enhanced video feature data if the light intensity is not greater than the preset threshold;

[0046] The video recognition module is also used to use the enhanced video feature data as input of a pre-trained behavior recognition model to obtain a first detection result.

[0047] Beneficial effects of the present invention:

[0048] The present invention proposes a multimodal-based security warning method, which includes: acquiring audio and video data and light intensity of a target area; preprocessing the audio and video data to obtain video feature data and audio feature data; if the light intensity is greater than a preset threshold, using the video feature data as an input of a pre-trained behavior recognition model to obtain a first detection result; the first detection result includes normal and abnormal; using the audio feature data as an input of a pre-trained speech recognition model to obtain a second detection result; the second detection result includes normal and abnormal; if the first detection result is abnormal or the second detection result is abnormal, generating warning information.

[0049] By automatically analyzing audio and video data, abnormal behavior can be quickly screened out, avoiding possible negligence in manual monitoring; and early warning information can be used to prompt staff to handle it in a timely manner to avoid further deterioration of security incidents, greatly shortening the response time, improving the timeliness of event handling, and improving the overall efficiency of the security system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described below in conjunction with the accompanying drawings.

[0051] Figure 1 A flowchart of a multi-modal security warning method is provided for an embodiment of the present invention;

[0052] Figure 2 The embodiment of the present invention provides an original network structure diagram of a SlowFast model;

[0053] Figure 3 is a schematic diagram of the structure of a hybrid attention module provided by an embodiment of the present invention;

[0054] Figure 4 An embodiment of the present invention provides a schematic diagram of a bottleneck structure;

[0055] Figure 5 An architecture diagram of a multi-modal security warning system is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The embodiment of the present invention provides a multi-modal security warning method. Figure 1 , Figure 1A flowchart of a multi-modal security warning method provided by an embodiment of the present invention. The method comprises the following steps:

[0058] S101, obtaining audio and video data and light intensity of a target area.

[0059] S102, pre-processing the audio and video data to obtain video feature data and audio feature data.

[0060] S103: If the light intensity is greater than a preset threshold, the video feature data is used as an input of a pre-trained behavior recognition model to obtain a first detection result.

[0061] S104: Use the audio feature data as input to a pre-trained speech recognition model to obtain a second detection result.

[0062] S105: If the first detection result is abnormal or the second detection result is abnormal, a warning message is generated.

[0063] Among them, the first test result includes normal and abnormal; the second test result includes normal and abnormal.

[0064] A multimodal security warning method provided by an embodiment of the present invention can quickly screen out abnormal behaviors by automatically analyzing audio and video data, thus avoiding possible negligence in manual monitoring; and prompt staff to handle the incident in a timely manner through early warning information, thus avoiding further deterioration of the security incident, greatly shortening the response time, improving the timeliness of incident handling, and improving the overall efficiency of the security system.

[0065] In one implementation, behavior annotations may include fighting, running, falling, climbing over railings, walking, smoking, etc. In different occasions, various behaviors are divided into normal and abnormal. For example, in a residential area, fighting, falling, etc. are classified as abnormal, while walking, running, smoking, etc. are classified as normal. In places where smoking is prohibited, such as scenic spots, smoking is classified as abnormal. The division of normal and abnormal behaviors according to the scene can provide more accurate warnings. The behavior recognition model can obtain the probability of various behaviors, and take the behaviors with probabilities greater than the accuracy threshold as the model output, and there can be multiple. If there is abnormal behavior, the first detection result is abnormal.

[0066] In one implementation, the speech recognition model can be composed of a speech-to-text sub-model and a text sentiment analysis model. The speech-to-text sub-model can use the DeepSpeech model, and the text analysis model can use the BERT model. SOS voice messages can be marked as abnormal and model training can be performed, so that the speech recognition model can detect the surrounding critical situations in a timely manner, and then feedback to the staff for timely processing.

[0067] In one implementation, the warning information may include the monitoring device number to remind the staff to pay attention to the monitoring situation.

[0068] In one embodiment, the audio and video data includes video signal data and audio signal data; step S102 includes:

[0069] Step 1: scaling each frame of the video signal data according to a preset resolution to obtain video feature data.

[0070] Step 2: De-noise the audio signal data according to a preset bandpass filter to obtain audio feature data.

[0071] In one implementation, each frame of the video meets the input format requirements of the model through resizing. The passband of the bandpass filter can be set to the human hearing range. De-noising can make the audio signal cleaner and avoid the interference of environmental noise on subsequent speech recognition, thereby improving the quality and reliability of audio data.

[0072] In one embodiment, the behavior recognition model is a modified SlowFast model. Figure 2 , Figure 2 This is an original network structure diagram of a SlowFast model provided by an embodiment of the present invention. As shown in the figure, the original SlowFast model includes a Slow branch, a Fast branch, and a prediction network. The horizontal connection between branches is realized by time-strided convolution; each branch includes a convolution layer, a pooling layer, and four residual blocks. The Slow branch and the Fast branch are used for feature extraction, and the prediction network is used for behavior recognition and classification.

[0073] Specific improvements include:

[0074] A hybrid attention module is embedded in each branch of the original SlowFast model.

[0075] See also Figure 3 , Figure 3 : is a structural diagram of a hybrid attention module provided by an embodiment of the present invention. As shown in the figure, the hybrid attention module HAM is composed of a channel timing attention module M1 and a spatial timing attention module M2. The calculation formula of the hybrid attention module is:

[0076]

[0077] Among them, x is the input of the hybrid attention module; f maxpool represents the maximum pooling; f avgpool represents average pooling; represents the convolution operation; σ is the sigmoid function; W1 is the channel temporal attention weight; represents element-wise multiplication; concat represents channel concatenation; W2 is the spatial-temporal attention weight; y1 is the feature weighted by the channel-temporal attention; y2 is the feature weighted by the spatial-temporal attention, and is also the output of the mixed attention module.

[0078] In one implementation, the attention module can effectively suppress irrelevant or redundant features, preventing the model from over-relying on unimportant details, thereby improving the efficiency and accuracy of the model. By adding the attention module, the SlowFast model can adaptively focus on the most important features in different time and space dimensions.

[0079] In one implementation, the channel temporal attention module M1 extracts channel information through convolution operations rather than traditional fully connected layers, emphasizing the capture of local features. And through 3D convolution, channel features and timing information are captured simultaneously, so that the model can more effectively learn the time and channel dependencies in the video. The spatial temporal attention module M2 can capture the spatial and temporal features in the video, and further optimize these features through convolution operations, helping the model to more accurately locate key action information. By connecting in series rather than in parallel, the hybrid attention module HAM ensures effective interaction between different features, thereby obtaining a richer feature representation.

[0080] In one embodiment, each residual block includes multiple bottleneck structures, each bottleneck structure consists of three convolutional layers; see Figure 4 , Figure 4 It is a schematic diagram of a bottleneck structure provided by an embodiment of the present invention. Figure 4 (a) shows the original bottleneck structure; Figure 4 (b) shows the bottleneck structure embedded with the hybrid attention module.

[0081] The embedding locations of the hybrid attention module include:

[0082] The first residual block of the Slow branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure;

[0083] The second residual block of the Slow branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure;

[0084] The first residual block of the Fast branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure;

[0085] The second residual block of the Fast branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure.

[0086] Among them, the first residual block of the Slow branch is Figure 2 Res2_s in the Slow branch; the second residual block of the Slow branch is Figure 2 Res3_s in ; the first residual block of the Fast branch is Figure 2 Res2_f in ; the second residual block of the Fast branch is Figure 2 Res3_f in.

[0087] In one implementation, each bottleneck structure consists of three layers of convolutions. The second convolution layer is the key layer for transforming channel information, while the third convolution layer is responsible for changing the channel dimension. A hybrid attention module is embedded between the two layers, which can introduce an attention mechanism in the process of channel feature transformation and optimize the selection and representation of key features.

[0088] In one implementation, a hybrid attention module is embedded in the Slow and Fast branches respectively, so that both branches can get attention enhancement at the time scale they are good at. This enhancement not only improves the ability of a single branch, but also further improves the spatiotemporal modeling ability of the overall model through subsequent fusion.

[0089] In one embodiment, the method further comprises:

[0090] Step 1: If the light intensity is not greater than a preset threshold, gamma correction is performed on the video feature data to obtain enhanced video feature data.

[0091] Step 2: Using the enhanced video feature data as input to a pre-trained behavior recognition model to obtain a first detection result.

[0092] In one implementation, the preset threshold can be set to 50 lux. Gamma correction can optimize the brightness of video images under low-light conditions, making dark details clearer and bright parts less overexposed, thereby improving image quality. In low-light conditions, ungamma-corrected video features may cause the model to be unable to accurately extract key features. Enhancing input video features through gamma correction can significantly improve the accuracy of the behavior recognition model in low-light environments.

[0093] The embodiment of the present invention provides a multi-modal security warning system. Figure 5 , Figure 5 An architecture diagram of a multi-modal security warning system provided in an embodiment of the present invention.

[0094] The system includes:

[0095] The data acquisition module is used to obtain the audio and video data and light intensity of the target area.

[0096] The preprocessing module is used to preprocess the audio and video data to obtain video feature data and audio feature data.

[0097] The video recognition module is used to use the video feature data as input to a pre-trained behavior recognition model to obtain a first detection result if the light intensity is greater than a preset threshold.

[0098] The audio recognition module is used to use the audio feature data as an input of a pre-trained speech recognition model to obtain a second detection result.

[0099] The early warning module is used to generate early warning information if the first detection result is abnormal or the second detection result is abnormal.

[0100] Among them, the first test result includes normal and abnormal; the second test result includes normal and abnormal.

[0101] A multimodal security warning system provided by an embodiment of the present invention can quickly screen out abnormal behaviors by automatically analyzing audio and video data, thereby avoiding possible negligence in manual monitoring; and prompting staff to handle timely through early warning information to avoid further deterioration of security incidents, thereby greatly shortening the response time, improving the timeliness of event processing, and improving the overall efficiency of the security system.

[0102] In one embodiment, the audio and video data includes video signal data and audio signal data; the preprocessing module includes:

[0103] The video processing module is used to scale the size of each frame of the video signal data according to a preset resolution size to obtain video feature data.

[0104] The audio processing module is used to denoise the audio signal data according to a preset bandpass filter to obtain audio feature data.

[0105] In one embodiment, the system further comprises a brightness enhancement module, wherein:

[0106] The brightness enhancement module is used to perform gamma correction on the video feature data to obtain enhanced video feature data if the light intensity is not greater than a preset threshold.

[0107] The video recognition module is also used to use the enhanced video feature data as input of the pre-trained behavior recognition model to obtain a first detection result.

[0108] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A multi-modal security warning method, characterized in that: The method comprises: Obtain audio and video data and light intensity of the target area; Preprocessing the audio and video data to obtain video feature data and audio feature data; If the light intensity is greater than a preset threshold, the video feature data is used as an input of a pre-trained behavior recognition model to obtain a first detection result; the first detection result includes normal and abnormal; Using the audio feature data as an input of a pre-trained speech recognition model to obtain a second detection result; the second detection result includes normal and abnormal; If the first detection result is abnormal or the second detection result is abnormal, a warning message is generated.

2. The multi-modal security warning method according to claim 1, characterized in that: The audio and video data includes video signal data and audio signal data; the preprocessing of the audio and video data to obtain video feature data and audio feature data includes: According to a preset resolution, each frame of the video signal data is resized to obtain video feature data; The audio signal data is denoised according to a preset bandpass filter to obtain audio feature data.

3. The multi-modal security warning method according to claim 1, characterized in that: The behavior recognition model is an improved SlowFast model; the original SlowFast model includes a Slow branch and a Fast branch; each branch includes a convolution layer, a pooling layer and four residual blocks; the specific improvements include: A hybrid attention module is embedded in each branch of the original SlowFast model; the calculation formula of the hybrid attention module is: Where x is the input of the hybrid attention module; f maxpool represents the maximum pooling; f avgpool represents average pooling; represents the convolution operation; σ is the sigmoid function; W1 is the channel temporal attention weight; Represents element multiplication; concat represents channel concatenation; W2 is the spatial-temporal attention weight; y1 is the feature after channel-temporal attention weighting; y2 is the feature after spatial-temporal attention weighting, and is also the output of the hybrid attention module.

4. The multi-modal security warning method according to claim 3, characterized in that: Each residual block includes multiple bottleneck structures, each of which consists of three convolutional layers; the embedding position of the hybrid attention module includes: The first residual block of the Slow branch is between the second convolution layer and the third convolution layer of each bottleneck structure; The second residual block of the Slow branch is between the second convolution layer and the third convolution layer of each bottleneck structure; The first residual block of the Fast branch is between the second convolution layer and the third convolution layer of each bottleneck structure; The second residual block of the Fast branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure.

5. The multi-modal security warning method according to claim 1, characterized in that: The method further comprises: If the illumination intensity is not greater than the preset threshold, performing gamma correction on the video feature data to obtain enhanced video feature data; The enhanced video feature data is used as input to the pre-trained behavior recognition model to obtain a first detection result.

6. A multi-modal security warning system, characterized in that: The system comprises: A data acquisition module is used to obtain audio and video data and light intensity of the target area; A preprocessing module, used for preprocessing the audio and video data to obtain video feature data and audio feature data; A video recognition module, configured to use the video feature data as an input of a pre-trained behavior recognition model to obtain a first detection result if the light intensity is greater than a preset threshold; the first detection result includes normal and abnormal; An audio recognition module, used to use the audio feature data as an input of a pre-trained speech recognition model to obtain a second detection result; the second detection result includes normal and abnormal; The early warning module is used to generate early warning information if the first detection result is abnormal or the second detection result is abnormal.

7. The multi-modal security warning system according to claim 6, characterized in that: The audio and video data includes video signal data and audio signal data; The preprocessing module comprises: A video processing module, used for scaling each frame of the video signal data according to a preset resolution to obtain video feature data; The audio processing module is used to denoise the audio signal data according to a preset bandpass filter to obtain audio feature data.

8. The multi-modal security warning system according to claim 6, characterized in that: The behavior recognition model is an improved SlowFast model; the original SlowFast model includes a Slow branch and a Fast branch; each branch includes a convolution layer, a pooling layer and four residual blocks; the specific improvements include: A hybrid attention module is embedded in each branch of the original SlowFast model; the calculation formula of the hybrid attention module is: Where x is the input of the hybrid attention module; f maxpool represents the maximum pooling; f avgpool represents average pooling; represents the convolution operation; σ is the sigmoid function; W1 is the channel temporal attention weight; Represents element multiplication; concat represents channel concatenation; W2 is the spatial-temporal attention weight; y1 is the feature after channel-temporal attention weighting; y2 is the feature after spatial-temporal attention weighting, and is also the output of the hybrid attention module.

9. The multi-modal security warning system according to claim 8, characterized in that: Each residual block includes multiple bottleneck structures, each of which consists of three convolutional layers; the embedding position of the hybrid attention module includes: The first residual block of the Slow branch is between the second convolution layer and the third convolution layer of each bottleneck structure; The second residual block of the Slow branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure; The first residual block of the Fast branch is between the second convolution layer and the third convolution layer of each bottleneck structure; The second residual block of the Fast branch is between the second convolutional layer and the third convolutional layer of each bottleneck structure.

10. The multi-modal security warning system according to claim 6, characterized in that: The system also includes a brightness enhancement module, wherein: The brightness enhancement module is used to perform gamma correction on the video feature data to obtain enhanced video feature data if the light intensity is not greater than the preset threshold; The video recognition module is also used to use the enhanced video feature data as input of a pre-trained behavior recognition model to obtain a first detection result.

Images