Intelligent security alarm system and method based on information fusion technology

By combining multimodal information collected by sound sensors and cameras in the smart security alarm system, using artificial intelligence for semantic analysis, the problem of traditional security systems capturing hidden abnormalities and environmental interference is solved, and more accurate monitoring and alarm is achieved.

CN117975638BActive Publication Date: 2024-11-05陕西瑞欣科技发展有限公司
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
CN202410074748.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-18
Publication Date
2024-11-05
Estimated Expiration
2044-01-18

AI Technical Summary

Technical Problem

Traditional security systems rely on image information to capture hidden or invisible abnormal situations, and are easily disturbed by environmental factors, resulting in false alarms or false alarms, resulting in monitoring blind spots that cannot fully and accurately determine whether there are abnormalities in the monitoring area.

Method used

Using a smart security alarm system based on information fusion technology, the sound signals and videos of the monitoring area are collected in real time through sound sensors and cameras, and an artificial intelligence-based signal analysis and data processing algorithm are introduced at the back end to perform semantic analysis, combining sound semantics and Video semantics perform abnormal detection and alarm.

Benefits of technology

Through the analysis and fusion characterization of multimodal information, real-time security monitoring and alarms can be achieved, and accurate detection and alarm capabilities of abnormal situations in the monitoring area can be improved, and false alarms and false alarms can be reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117975638B_ABST
    Figure CN117975638B_ABST
Patent Text Reader

Abstract

The present application discloses a smart security alarm system and method based on information fusion technology, which collects the sound signal and monitoring video of the monitoring area in real time through sound sensors and cameras, and introduces signal analysis and data processing algorithms based on artificial intelligence at the back end to perform semantic analysis of the sound signal and the monitoring video, so as to comprehensively detect and alarm abnormal situations by combining the sound semantics and video semantics of the monitoring area. In this way, it is possible to comprehensively perform smart security monitoring through the analysis and fusion representation of multimodal information of the monitoring area, thereby realizing real-time security monitoring and alarming, and improving the ability to accurately detect and alarm abnormal situations in the monitoring area.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent alarm, and more specifically, to an intelligent security alarm system and method based on information fusion technology. Background Art

[0002] With the continuous advancement of science and technology, the field of smart security has also developed rapidly, and the smart security alarm system has become an important part of social security. The smart security alarm system is a system that uses advanced information technology and sensor equipment to achieve real-time monitoring and alarm of the monitored area, which can provide strong protection for social security.

[0003] However, traditional security systems mainly rely on cameras to capture and analyze image information of the security monitoring area. However, image information alone may not be able to fully capture some hidden or invisible abnormalities, such as behavior in low-light environments, abnormalities in sound generation, etc. In addition, when using image information for abnormality detection, traditional security systems are easily disturbed by environmental factors such as light, shadows, and weather, resulting in false alarms or false alarms. For example, changes in light or shaking leaves may be misjudged as abnormal conditions, resulting in unnecessary alarms. Therefore, relying solely on image information for security monitoring may lead to monitoring blind spots, and it is impossible to fully and accurately determine whether there are abnormal conditions in the monitoring area.

[0004] Therefore, an intelligent security alarm system based on information fusion technology is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a smart security alarm system and method based on information fusion technology, which collects the sound signal and monitoring video of the monitoring area in real time through sound sensors and cameras, and introduces artificial intelligence-based signal analysis and data processing algorithms at the back end to perform semantic analysis of the sound signal and the monitoring video, so as to combine the sound semantics and video semantics of the monitoring area to comprehensively detect and alarm abnormal situations. In this way, it is possible to comprehensively perform smart security monitoring through the analysis and fusion representation of multimodal information of the monitoring area, thereby realizing real-time security monitoring and alarm, and improving the ability to accurately detect and alarm abnormal situations in the monitoring area.

[0006] According to one aspect of the present application, there is provided a smart security alarm system based on information fusion technology, which includes:

[0007] A data acquisition module, used to obtain monitoring video and sound signals of the monitored area in a predetermined time period collected by the sound sensor and the camera;

[0008] A video semantic understanding module, used for extracting features from the surveillance video through a video encoder based on a deep neural network model to obtain a video semantic understanding feature map of the surveillance area;

[0009] A sound signal framing module, used for performing frame processing on the sound signal to obtain a sequence of sound signal frames;

[0010] A sound signal frame amplitude spectrum calculation module, used to calculate the amplitude spectrum of each sound signal frame in the sequence of sound signal frames to obtain a sequence of sound signal frame amplitude spectrum vectors;

[0011] A sound signal frame amplitude spectrum semantic association analysis module is used to perform sound signal frame amplitude spectrum feature association analysis on the sequence of the sound signal frame amplitude spectrum vectors to obtain a sound signal semantic understanding feature vector;

[0012] A monitoring area multimodal fusion representation module, used to fuse the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map to obtain a monitoring area multimodal fusion representation feature;

[0013] The regional anomaly detection module is used to determine whether there is an anomaly in the monitoring area based on the multimodal fusion representation features of the monitoring area.

[0014] According to another aspect of the present application, a smart security alarm method based on information fusion technology is provided, which includes:

[0015] Acquire surveillance video and sound signals of the monitored area collected by sound sensors and cameras in a predetermined time period;

[0016] Extracting features from the surveillance video using a video encoder based on a deep neural network model to obtain a video semantic understanding feature map of the surveillance area;

[0017] Performing frame processing on the sound signal to obtain a sequence of sound signal frames;

[0018] Calculating the amplitude spectrum of each sound signal frame in the sequence of sound signal frames to obtain a sequence of sound signal frame amplitude spectrum vectors;

[0019] Performing sound signal frame amplitude spectrum feature correlation analysis on the sequence of the sound signal frame amplitude spectrum vectors to obtain a sound signal semantic understanding feature vector;

[0020] Fusion of the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map to obtain a monitoring area multimodal fusion representation feature;

[0021] Based on the multimodal fusion representation features of the monitoring area, determine whether there is an abnormality in the monitoring area.

[0022] Compared with the prior art, the present application provides a smart security alarm system and method based on information fusion technology, which collects the sound signal and monitoring video of the monitoring area in real time through sound sensors and cameras, and introduces signal analysis and data processing algorithms based on artificial intelligence at the back end to perform semantic analysis of the sound signal and the monitoring video, so as to comprehensively detect and alarm abnormal situations by combining the sound semantics and video semantics of the monitoring area. In this way, it is possible to comprehensively perform smart security monitoring through the analysis and fusion representation of multimodal information of the monitoring area, thereby realizing real-time security monitoring and alarming, and improving the ability to accurately detect and alarm abnormal situations in the monitoring area. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 A block diagram of an intelligent security alarm system based on information fusion technology according to an embodiment of the present application;

[0025] Figure 2 A system architecture diagram of a smart security alarm system based on information fusion technology according to an embodiment of the present application;

[0026] Figure 3 A block diagram of a regional anomaly detection module in a smart security alarm system based on information fusion technology according to an embodiment of the present application;

[0027] Figure 4 A block diagram of a monitoring area multimodal fusion feature optimization unit in a smart security alarm system based on information fusion technology according to an embodiment of the present application;

[0028] Figure 5 This is a flow chart of an intelligent security alarm method based on information fusion technology according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0030] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0031] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0032] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0033] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0034] Traditional security systems mainly rely on cameras to capture and analyze image information in the security monitoring area. However, image information alone may not be able to fully capture some hidden or invisible abnormalities, such as behavior in low-light environments, abnormalities in sound generation, etc. In addition, when using image information for abnormality detection, traditional security systems are easily disturbed by environmental factors such as light, shadows, and weather, resulting in false alarms or false alarms. For example, changes in light or shaking leaves may be misjudged as abnormalities, resulting in unnecessary alarms. Therefore, relying solely on image information for security monitoring may result in monitoring blind spots, and it is impossible to fully and accurately determine whether there are abnormalities in the monitoring area. Therefore, an intelligent security alarm system based on information fusion technology is desired.

[0035] In the technical solution of the present application, an intelligent security alarm system based on information fusion technology is proposed. Figure 1 It is a block diagram of an intelligent security alarm system based on information fusion technology according to an embodiment of the present application. Figure 2 : is a system architecture diagram of a smart security alarm system based on information fusion technology according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the embodiment of the present application, the intelligent security alarm system 300 based on information fusion technology includes: a data acquisition module 310, which is used to obtain the monitoring video and sound signal of the monitored area in a predetermined time period collected by the sound sensor and the camera; a video semantic understanding module 320, which is used to extract features of the monitoring video through a video encoder based on a deep neural network model to obtain a video semantic understanding feature map of the monitoring area; a sound signal framing module 330, which is used to perform frame processing on the sound signal to obtain a sequence of sound signal frames; a sound signal frame amplitude spectrum calculation module 340, which is used to calculate the sound signal frame The amplitude spectrum of each sound signal frame in the sequence is used to obtain a sequence of sound signal frame amplitude spectrum vectors; a sound signal frame amplitude spectrum semantic association analysis module 350 is used to perform sound signal frame amplitude spectrum feature association analysis on the sequence of sound signal frame amplitude spectrum vectors to obtain a sound signal semantic understanding feature vector; a monitoring area multimodal fusion representation module 360 ​​is used to fuse the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map to obtain a monitoring area multimodal fusion representation feature; a regional anomaly detection module 370 is used to determine whether there is an anomaly in the monitoring area based on the monitoring area multimodal fusion representation feature.

[0036] In particular, the data acquisition module 310 is used to obtain the monitoring video and sound signals of the monitored area collected by the sound sensor and the camera in a predetermined time period. The sound sensor is a device used to detect and measure the sound in the environment. It can convert the sound signal into an electrical signal for further processing and analysis.

[0037] In particular, the video semantic understanding module 320 is used to extract features from the surveillance video through a video encoder based on a deep neural network model to obtain a video semantic understanding feature map of the surveillance area. In particular, the deep neural network model is a three-dimensional convolutional neural network model. Considering that traditional image processing methods mainly focus on low-level features of the image of the surveillance area, such as color, texture, and edges. However, these low-level features often cannot capture high-level semantic information in the video, such as the category, action, and scene of the object. In order to better understand the video content, a deep learning algorithm needs to be introduced. In addition, considering that the surveillance video of the monitored area is composed of multiple video frames, in order to be able to more fully extract the semantic information in the surveillance video, so as to better understand the content and behavior of the surveillance area, in the technical solution of the present application, the surveillance video is further subjected to feature mining in a video encoder based on a three-dimensional convolutional neural network model to extract the temporal correlation feature information of the semantic features of each video frame in the surveillance video in the time dimension, thereby obtaining a video semantic understanding feature map of the surveillance area. By using the video encoder based on the three-dimensional convolutional neural network model to process and extract features of the surveillance video, the content and behavior of the surveillance area can be better understood, thereby achieving accurate detection and alarm of abnormal situations. This deep learning-based video encoder can improve the security system's ability to understand video data, making the system more intelligent and reliable. Specifically, the surveillance video is subjected to feature mining in a video encoder based on a three-dimensional convolutional neural network model to obtain the video semantic understanding feature map of the surveillance area, including: using each layer of the video encoder based on the three-dimensional convolutional neural network model to perform convolution processing on the input data in the forward transfer of the layer: performing convolution processing on the input data to obtain a convolution feature map; performing pooling based on a local feature matrix on the convolution feature map to obtain a pooled feature map; and performing nonlinear activation on the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the video encoder based on the three-dimensional convolutional neural network model is the video semantic understanding feature map of the surveillance area, and the input of the first layer of the video encoder based on the three-dimensional convolutional neural network model is the surveillance video.

[0038] It is worth noting that the three-dimensional convolutional neural network (3D CNN) is an extended form of convolutional neural network (CNN) for processing data with spatiotemporal relationships, such as videos, medical images, action recognition, etc. Unlike traditional two-dimensional convolutional neural networks, 3D CNN considers the third dimension, namely the time dimension, in the convolution operation. It can perform convolution operations in space and time at the same time to extract spatiotemporal features. The following are the general steps of 3D CNN: Input data: The input of 3D CNN is usually a five-dimensional tensor of shape [Batch Size, Channels, Depth, Height, Width], where Batch Size represents the batch size, Channels represents the number of channels, Depth represents the time dimension, and Height and Width represent the spatial dimension; Convolution operation: 3D CNN applies convolution operations on the three dimensions of the input data. The convolution kernel is a weight matrix with three dimensions that slides over each position of the input data and performs a convolution operation. Convolution operation can extract spatiotemporal features; nonlinear activation function: after the convolution operation, a nonlinear activation function, such as ReLU, is usually applied to introduce nonlinear features; pooling operation: you can choose to perform pooling operation in the spatial dimension to reduce the size of the feature map and extract more important features. Common pooling operations include maximum pooling and average pooling; flattening: flatten the pooled feature map into a one-dimensional vector for input to the fully connected layer; fully connected layer: perform classification or regression tasks through the fully connected layer to output the final prediction result.

[0039] In particular, the sound signal framing module 330 is used to perform framing processing on the sound signal to obtain a sequence of sound signal frames. Considering that the sound signal is a continuous time signal, in practical applications, it is usually necessary to divide the sound signal into small segments of frames for processing. This is because the sound signal is usually stable in a short period of time, thereby simplifying the processing and analysis of the speech signal. Based on this, in the technical solution of the present application, it is necessary to perform framing processing on the sound signal to obtain a sequence of sound signal frames.

[0040] Accordingly, in a possible implementation, the sound signal can be framed to obtain a sequence of sound signal frames by the following steps, for example: selecting the frame length and the moving step between frames. The frame length determines the duration of each sound signal frame, and the frame shift determines the time interval between adjacent frames; applying a window function to each frame to reduce spectral leakage at the frame boundary; dividing the original sound signal into a series of overlapping frames. Starting from the starting position of the original signal, a time window of a frame length is selected every frame shift time, and the signal within the window is regarded as a frame. A portion of samples are shared between overlapping frames to retain continuity in the time domain; each frame is point-by-point multiplied with the corresponding window function to obtain a windowed frame; the windowed frames are arranged in sequence to obtain the sequence of sound signal frames.

[0041] In particular, the sound signal frame amplitude spectrum calculation module 340 is used to calculate the amplitude spectrum of each sound signal frame in the sequence of the sound signal frames to obtain a sequence of sound signal frame amplitude spectrum vectors. That is, after obtaining the sequence of sound signal frames, the amplitude spectrum of each sound signal frame is calculated to extract the amplitude information of each sound signal frame, thereby obtaining a sequence of sound signal frame amplitude spectrum vectors. It should be understood that the amplitude spectrum represents the energy distribution of the sound signal at different frequencies and can reflect the frequency domain characteristics of the sound signal. Therefore, each element in the amplitude spectrum vector of each sound signal frame represents the amplitude of the speech signal at the corresponding frequency. By performing frame processing and calculating the amplitude spectrum on the sound signal, the system can better capture the time domain and frequency domain characteristics of the sound signal, thereby providing a basis for subsequent semantic encoding of the sound signal and multimodal information fusion. This can enhance the system's ability to understand sound signals and further improve the accuracy and reliability of the smart security alarm system.

[0042] In particular, the sound signal frame amplitude spectrum semantic association analysis module 350 is used to perform sound signal frame amplitude spectrum feature association analysis on the sequence of the sound signal frame amplitude spectrum vector to obtain the sound signal semantic understanding feature vector. It should be understood that the amplitude spectrum vector sequence of the sound signal contains the energy distribution of each sound frame in the frequency domain, but the semantic information of the sound signal cannot be directly obtained only through the amplitude spectrum vector. In order to better understand the meaning and characteristics of the sound signal, it is necessary to introduce an appropriate model for semantic encoding. Therefore, in order to be able to better semantically model and encode the sound signal, so as to extract the semantic association features between each sound frame of the sound signal and the high-level semantic information of the sound signal, in the technical solution of the present application, the sequence of the sound signal frame amplitude spectrum vector is further passed through a sound signal semantic encoder based on the LSTM model to obtain a sound signal semantic understanding feature vector. By using the LSTM model, the sound signal frame amplitude spectrum vector sequence can be temporally modeled and encoded, so as to capture the temporal features and semantic association information between each signal frame of the sound signal.

[0043] It is worth noting that LSTM (Long Short-Term Memory) is a variant of recurrent neural network (RNN) that is specifically designed to process sequence data and capture long-term dependencies. Each unit in LSTM has a memory cell and three gates (input gate, forget gate, and output gate), which can effectively handle information transfer and forgetting in the sequence. The following is the general structure of the LSTM model: Input sequence: The LSTM model accepts a sequence as input, such as time series data, text sequence, etc. The length of the sequence is T, and the input feature dimension of each time step is D; Initialize hidden state and memory unit: Before processing the sequence, the hidden state and memory unit of LSTM need to be initialized. Usually, they can be initialized as all-zero vectors or initialized using a pre-trained model; LSTM unit: LSTM unit is the core component of the LSTM model. Each LSTM unit contains a memory unit and three gates: input gate, forget gate, and output gate; input gate: determines the degree of influence of the input of the current time step on the memory unit; forget gate: determines whether to forget information from the memory unit; output gate: determines the degree of influence of the memory unit state of the current time step on the output; time step processing: the LSTM model processes the input sequence time step by time step. For each time step, the LSTM unit updates the hidden state and memory unit of the current time step based on the current input, the hidden state and memory unit of the previous time step; output: after processing the entire sequence, you can choose to use the hidden state of the last time step as the output of the model according to the task requirements, or you can use the hidden state of all time steps for further processing.

[0044] In particular, the monitoring area multimodal fusion representation module 360 ​​is used to fuse the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map to obtain the monitoring area multimodal fusion representation feature. Considering that in the smart security system, the sound signal and the monitoring video are two different data modes, which respectively provide feature information of the sound and image of the monitoring area. In order to more comprehensively understand the situation of the monitoring area, it is necessary to fuse the semantic feature information of these two modalities to obtain a richer and more accurate feature representation. Specifically, in the technical solution of the present application, the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map are processed using a meta-network-based information fusion module to obtain a monitoring area multimodal fusion representation feature map. In particular, in a specific example of the present application, the meta-network is MetaNet, which is a network structure for multimodal fusion, which can learn the association and weight between different modalities, such as the association and weight information between the sound semantics and image semantics of the monitoring area, thereby achieving effective information fusion. In other words, the meta-network using MetaNet can adaptively fuse the features of different modalities by learning the semantic associations and weights between different modalities in the monitored area to obtain a more representative multimodal feature representation, so that the multimodal feature representation of the monitored area can more comprehensively describe the situation in the monitored area. Specifically, a meta-network-based information fusion module is used to process the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map to obtain a monitoring area multimodal fusion representation feature map as the monitoring area multimodal fusion representation feature, including: processing the monitoring area video semantic understanding feature map in the convolutional neural network model of the parameter-guided cross-modal encoder to obtain a monitoring area video enhancement semantic feature map; passing the sound signal semantic understanding feature vector through the first convolution layer of the parameter-guided cross-modal encoder and then performing linear correction processing through the ReLU function to obtain a linearly corrected sound signal semantic understanding feature vector; passing the linearly corrected sound signal semantic understanding feature vector through the second convolution layer of the parameter-guided cross-modal encoder and then processing it through the Sigmoid function to obtain a normalized sound signal semantic understanding feature vector; and performing weighted fusion along the channel dimension on the monitoring area video enhancement semantic feature map using the normalized sound signal semantic understanding feature vector to obtain the monitoring area multimodal fusion representation feature map.

[0045] In particular, the regional anomaly detection module 370 is used to determine whether there is an anomaly in the monitoring area based on the multimodal fusion representation features of the monitoring area. In particular, in a specific example of the present application, Figure 3As shown, the regional anomaly detection module 370 includes: a monitoring area multimodal fusion feature optimization unit 371, which is used to perform feature optimization on the monitoring area multimodal fusion representation feature map to obtain an optimized monitoring area multimodal fusion representation feature map; an anomaly detection unit 372, which is used to pass the optimized monitoring area multimodal fusion representation feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether there is an abnormality in the monitoring area.

[0046] Specifically, the monitoring area multimodal fusion feature optimization unit 371 is used to optimize the monitoring area multimodal fusion representation feature map to obtain an optimized monitoring area multimodal fusion representation feature map. In particular, in a specific example of the present application, Figure 4 As shown, the monitoring area multimodal fusion feature optimization unit 371 includes: a linear transformation subunit 3711, used to perform a linear transformation on the monitoring area multimodal fusion representation feature map so that the width and height of each feature matrix along the channel dimension in the monitoring area multimodal fusion representation feature map are equal to obtain the converted monitoring area multimodal fusion representation feature map; a channel dimension optimization subunit 3712, used to perform channel dimension optimization on the converted monitoring area multimodal fusion representation feature map to obtain the optimized monitoring area multimodal fusion representation feature map.

[0047] More specifically, the linear transformation subunit 3711 is used to perform a linear transformation on the monitoring area multimodal fusion representation feature map so that the width and height of each feature matrix along the channel dimension in the monitoring area multimodal fusion representation feature map are equal to obtain the converted monitoring area multimodal fusion representation feature map. It should be understood that by performing a linear transformation so that the width and height of each feature matrix are equal, the size difference between different modal feature maps can be eliminated, reducing the computational complexity in subsequent processing and analysis.

[0048] More specifically, the channel dimension optimization subunit 3712 is used to perform channel dimension optimization on the converted monitoring area multimodal fusion representation feature map to obtain the optimized monitoring area multimodal fusion representation feature map. In particular, in the technical solution of the present application, each feature matrix of the monitoring area video semantic understanding feature map expresses the image semantic features of the temporal association of the image frames of the monitoring video, and follows the inter-frame semantic temporal association distribution of the image frame semantics of the monitoring video in the channel dimension, while the sound signal semantic understanding feature vector expresses the short-range and long-range bidirectional context association of the sound signal amplitude spectrum in time. Therefore, after the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map are processed by the meta-network-based information fusion module, the monitoring area video semantic understanding feature map will be constrained in the channel dimension based on the temporal short-range and long-range bidirectional context association of the sound signal amplitude spectrum. However, this will also cause the monitoring area multimodal fusion representation feature map to have a mixed temporal distribution of different temporal association modes based on different feature semantic representations in the channel dimension, thereby affecting the integrity of the channel dimension feature distribution of the monitoring area multimodal fusion representation feature map, causing the monitoring area multimodal fusion representation feature map to have difficulty in converging when the class probability is converged through the classifier. Therefore, the applicant of the present application preferably first performs a linear transformation on the monitoring area multimodal fusion representation feature map so that the width and height of the feature matrix are equal, and then optimizes the channel dimension of the converted monitoring area multimodal fusion representation feature map, which is expressed as: in and are respectively the first and second features of the converted monitoring area multimodal fusion representation feature map along the channel direction. and The feature matrix of the position, and is the scale adjustment hyperparameter, represents matrix multiplication, It means point multiplication by position. It means adding by position. represents the optimized monitoring area multimodal fusion representation feature map along the channel direction Here, the coupling distribution direction of the local feature distribution of the multimodal fusion representation feature map of the monitoring area along the channel is predicted in the high-dimensional feature space through the progressive structured embedding calculation of the feature matrix with channel adjacent distribution of the multimodal fusion representation feature map of the monitoring area, so as to determine the iteratively generated transfer diagram representation based on channel coupling based on the progressive center of the distribution, and reconstruct the contextual relationship of the multimodal fusion representation feature map of the monitoring area based on the scene layout of the feature matrix by refining the projection normalization proposal from bottom to top along the channel dimension of the multimodal fusion representation feature map of the monitoring area, so as to improve the channel dimension integrity of the feature representation of the multimodal fusion representation feature map of the monitoring area, so as to improve the convergence effect of the multimodal fusion representation feature map of the monitoring area when the class probability converges through the classifier, and improve the training efficiency and the accuracy of the classification results. In this way, intelligent security monitoring can be comprehensively carried out through the analysis and fusion representation of the multimodal information of the monitoring area, so as to realize real-time security monitoring and alarm, and improve the ability to accurately detect and alarm abnormal situations in the monitoring area.

[0049] It is worth mentioning that in other specific examples of the present application, the feature optimization of the multimodal fusion representation feature map of the monitoring area can be performed in other ways to obtain the optimized multimodal fusion representation feature map of the monitoring area, for example: input the multimodal fusion representation feature map of the monitoring area; split the multimodal fusion representation feature map of the monitoring area into individual feature maps along the channel dimension; for each feature map, a convolutional neural network can be used to perform a convolution operation to extract a higher level of feature representation; pooling operations such as maximum pooling or average pooling can be used to reduce the size of the feature map; the feature map can be normalized so that the feature value range is within a certain interval; an attention mechanism can be introduced to assign different weights to different areas in the feature map to enhance the representation ability of important features; a reinforcement learning method can be used to optimize the feature map, and by interacting with the environment, the feature map can be adapted to different task requirements; the feature maps that have been feature optimized are re-merged to obtain the optimized multimodal fusion representation feature map of the monitoring area.

[0050] Specifically, the abnormality detection unit 372 is used to pass the optimized multimodal fusion representation feature map of the monitoring area through a classifier to obtain a classification result, and the classification result is used to indicate whether there is an abnormality in the monitoring area. That is, the multimodal semantic fusion representation information about the monitoring area is used for classification processing, so as to comprehensively detect and alarm the abnormal situation in the area by combining the sound semantics and video semantics of the monitoring area. In this way, intelligent security monitoring can be comprehensively performed through the analysis and fusion representation of the multimodal information of the monitoring area, thereby realizing real-time security monitoring and alarming, and improving the ability to accurately detect and alarm the abnormal situation in the monitoring area. More specifically, passing the optimized multimodal fusion representation feature map of the monitoring area through a classifier to obtain a classification result, and the classification result is used to indicate whether there is an abnormality in the monitoring area, including: expanding the optimized multimodal fusion representation feature map of the monitoring area into a classification feature vector based on a row vector or a column vector; using multiple fully connected layers of the classifier to fully connect the classification feature vector to obtain a coded classification feature vector; and passing the coded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0051] That is, in the technical solution of the present application, the label of the classifier includes the presence of anomalies in the monitoring area (first label), and the absence of anomalies in the monitoring area (second label), wherein the classifier determines to which classification label the optimized monitoring area multimodal fusion representation feature map belongs through a soft maximum function. It is worth noting that the first label p1 and the second label p2 here do not contain artificially set concepts. In fact, during the training process, the computer model does not have the concept of "whether there is an anomaly in the monitoring area". It only has two classification labels and the probability of the output feature under these two classification labels, that is, the sum of p1 and p2 is one. Therefore, the classification result of whether there is an anomaly in the monitoring area is actually converted into a binary class probability distribution that conforms to the laws of nature through the classification label. In essence, the physical meaning of the natural probability distribution of the label is used, rather than the linguistic text meaning of "whether there is an anomaly in the monitoring area".

[0052] A classifier is a machine learning model or algorithm that is used to classify input data into different categories or labels. Classifiers are part of supervised learning and perform classification tasks by learning the mapping relationship from input data to output categories.

[0053] A fully connected layer is a common type of layer in neural networks. In a fully connected layer, each neuron is connected to all neurons in the previous layer, and each connection has a weight. This means that each neuron in the fully connected layer receives inputs from all neurons in the previous layer, and performs a weighted sum of these inputs through the weights, and then passes the result to the next layer.

[0054] The Softmax classification function is a commonly used activation function for multi-classification problems. It converts each element of the input vector into a probability value between 0 and 1, and the sum of these probability values ​​is equal to 1. The Softmax function is often used in the output layer of a neural network, especially for multi-classification problems, because it can map the network output to the probability distribution of each category. During the training process, the output of the Softmax function can be used to calculate the loss function and update the network parameters through the back propagation algorithm. It is worth noting that the output of the Softmax function does not change the relative size relationship between the elements, but only normalizes them. Therefore, the Softmax function does not change the characteristics of the input vector, but only converts it into a probability distribution form.

[0055] It is worth mentioning that in other specific examples of the present application, it is also possible to determine whether there is an abnormality in the monitoring area based on the multimodal fusion representation features of the monitoring area in other ways, for example: collect multimodal data of the monitoring area, such as video, image, sound, etc.; extract features for each data modality and convert it into a meaningful representation. For video and image data, convolutional neural networks can be used to extract visual features. For sound data, sound processing technology can be used to extract audio features; the features of different modalities are fused to obtain a more comprehensive and rich feature representation. Common fusion methods include feature-level fusion and decision-level fusion. Feature-level fusion connects or superimposes features of different modalities to form a multimodal feature vector. Decision-level fusion combines the classification results or confidences of different modalities to obtain the final decision result; use a training data set with labeled abnormal samples to train an anomaly detection model; input the multimodal fusion representation features of the monitoring area into the trained anomaly detection model to obtain an anomaly score or probability; according to the anomaly score or probability, set a threshold to determine whether there is an abnormality in the monitoring area. If the anomaly score exceeds the threshold, it is judged as abnormal; otherwise, it is judged as normal.

[0056] As described above, the smart security alarm system 300 based on information fusion technology according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a smart security alarm algorithm based on information fusion technology. In a possible implementation, the smart security alarm system 300 based on information fusion technology according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the smart security alarm system 300 based on information fusion technology can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the smart security alarm system 300 based on information fusion technology can also be one of the many hardware modules of the wireless terminal.

[0057] Alternatively, in another example, the smart security alarm system 300 based on information fusion technology and the wireless terminal may also be separate devices, and the smart security alarm system 300 based on information fusion technology may be connected to the wireless terminal through a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0058] Furthermore, an intelligent security alarm method based on information fusion technology is also provided.

[0059] Figure 5 Flow chart of the intelligent security alarm method based on information fusion technology according to an embodiment of the present application. Figure 5 As shown, according to the embodiment of the present application, the intelligent security alarm method based on information fusion technology includes the following steps: S1, acquiring monitoring video and sound signals of the monitored area collected by the sound sensor and the camera in a predetermined time period; S2, extracting features of the monitoring video through a video encoder based on a deep neural network model to obtain a video semantic understanding feature map of the monitored area; S3, performing frame processing on the sound signal to obtain a sequence of sound signal frames; S4, calculating the amplitude spectrum of each sound signal frame in the sequence of sound signal frames to obtain a sequence of sound signal frame amplitude spectrum vectors; S5, performing sound signal frame amplitude spectrum feature correlation analysis on the sequence of sound signal frame amplitude spectrum vectors to obtain a sound signal semantic understanding feature vector; S6, fusing the sound signal semantic understanding feature vector and the video semantic understanding feature map of the monitored area to obtain a multimodal fusion representation feature of the monitored area; S7, determining whether there is an abnormality in the monitored area based on the multimodal fusion representation feature of the monitored area.

[0060] In summary, the intelligent security alarm method based on information fusion technology according to the embodiment of the present application is explained, which collects the sound signal and monitoring video of the monitoring area in real time through sound sensors and cameras, and introduces signal analysis and data processing algorithms based on artificial intelligence at the back end to perform semantic analysis of the sound signal and the monitoring video, so as to combine the sound semantics and video semantics of the monitoring area to comprehensively detect and alarm abnormal situations. In this way, intelligent security monitoring can be comprehensively performed through the analysis and fusion representation of multimodal information of the monitoring area, thereby realizing real-time security monitoring and alarming, and improving the ability to accurately detect and alarm abnormal situations in the monitoring area.

[0061] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An intelligent security alarm system based on information fusion technology, characterized in that: include: A data acquisition module, used to obtain monitoring video and sound signals of the monitored area in a predetermined time period collected by the sound sensor and the camera; A video semantic understanding module, used for extracting features from the surveillance video through a video encoder based on a deep neural network model to obtain a video semantic understanding feature map of the surveillance area; A sound signal framing module, used for performing frame processing on the sound signal to obtain a sequence of sound signal frames; A sound signal frame amplitude spectrum calculation module, used to calculate the amplitude spectrum of each sound signal frame in the sequence of sound signal frames to obtain a sequence of sound signal frame amplitude spectrum vectors; A sound signal frame amplitude spectrum semantic association analysis module is used to perform sound signal frame amplitude spectrum feature association analysis on the sequence of the sound signal frame amplitude spectrum vectors to obtain a sound signal semantic understanding feature vector; A monitoring area multimodal fusion representation module, used to fuse the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map to obtain a monitoring area multimodal fusion representation feature; A regional anomaly detection module, used to determine whether there is an anomaly in the monitoring area based on the multimodal fusion representation features of the monitoring area; The monitoring area multimodal fusion representation module is used to: use a meta-network-based information fusion module to process the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map to obtain a monitoring area multimodal fusion representation feature map as the monitoring area multimodal fusion representation feature; Wherein, the monitoring area multimodal fusion representation module is used to: The surveillance area video semantic understanding feature map is processed in a convolutional neural network model of a parameter-guided cross-modal encoder to obtain a surveillance area video enhanced semantic feature map; The sound signal semantic understanding feature vector is passed through the first convolutional layer of the parameter-guided cross-modal encoder and then linearly corrected through a ReLU function to obtain a linearly corrected sound signal semantic understanding feature vector; The linearly corrected sound signal semantic understanding feature vector is passed through the second convolutional layer of the parameter-guided cross-modal encoder and then processed through a Sigmoid function to obtain a normalized sound signal semantic understanding feature vector; The normalized sound signal semantic understanding feature vector is used to perform weighted fusion along the channel dimension on the video enhancement semantic feature map of the monitoring area to obtain the multimodal fusion representation feature map of the monitoring area.

2. The intelligent security alarm system based on information fusion technology according to claim 1 is characterized in that: The deep neural network model is a three-dimensional convolutional neural network model.

3. The intelligent security alarm system based on information fusion technology according to claim 2 is characterized in that: The sound signal frame amplitude spectrum semantic association analysis module is used to: pass the sequence of the sound signal frame amplitude spectrum vectors through a sound signal semantic encoder based on an LSTM model to obtain the sound signal semantic understanding feature vector.

4. The intelligent security alarm system based on information fusion technology according to claim 3 is characterized in that: The regional anomaly detection module includes: A monitoring area multimodal fusion feature optimization unit, used for performing feature optimization on the monitoring area multimodal fusion representation feature map to obtain an optimized monitoring area multimodal fusion representation feature map; The anomaly detection unit is used to pass the optimized multimodal fusion representation feature map of the monitoring area through a classifier to obtain a classification result, and the classification result is used to indicate whether there is an anomaly in the monitoring area.

5. The intelligent security alarm system based on information fusion technology according to claim 4 is characterized in that: The monitoring area multimodal fusion feature optimization unit includes: A linear transformation subunit, used for performing a linear transformation on the monitoring area multimodal fusion representation feature map so that the width and height of each feature matrix along the channel dimension in the monitoring area multimodal fusion representation feature map are equal to obtain a converted monitoring area multimodal fusion representation feature map; The channel dimension optimization subunit is used to perform channel dimension optimization on the converted monitoring area multimodal fusion representation feature map to obtain the optimized monitoring area multimodal fusion representation feature map.

6. The intelligent security alarm system based on information fusion technology according to claim 5 is characterized in that: The abnormality detection unit comprises: An expansion subunit, used for expanding the optimized monitoring area multimodal fusion representation feature map into a classification feature vector based on a row vector or a column vector; a fully connected encoding subunit, configured to perform fully connected encoding on the classification feature vector using a plurality of fully connected layers of the classifier to obtain an encoded classification feature vector; and The classification result generating subunit is used to pass the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

7. A smart security alarm method based on information fusion technology according to the smart security alarm system based on information fusion technology according to claim 1, characterized in that: include: Acquire surveillance video and sound signals of the monitored area collected by sound sensors and cameras in a predetermined time period; Extracting features from the surveillance video using a video encoder based on a deep neural network model to obtain a video semantic understanding feature map of the surveillance area; Performing frame processing on the sound signal to obtain a sequence of sound signal frames; Calculating the amplitude spectrum of each sound signal frame in the sequence of sound signal frames to obtain a sequence of sound signal frame amplitude spectrum vectors; Performing sound signal frame amplitude spectrum feature correlation analysis on the sequence of the sound signal frame amplitude spectrum vectors to obtain a sound signal semantic understanding feature vector; Fusing the sound signal semantic understanding feature vector and the monitoring area video semantic understanding feature map to obtain a monitoring area multimodal fusion representation feature; Based on the multimodal fusion representation features of the monitoring area, determine whether there is an abnormality in the monitoring area.

Citation Information

Patent Citations

  • Patient behavior multimodal analysis and prediction system based on statistical learning

    CN111920420A

  • Speech enhancement method based on low-frequency syllable recognition

    CN112233657A

  • Voice activation detection method and device, electronic equipment and storage medium

    CN114333912A

  • Sound signal enhancement method and device based on waveform and frequency domain information fusion network

    CN114783457A

  • Abnormal sound detection method and device, storage medium and electronic equipment

    CN115565548A