Monitoring equipment monitoring video event analysis method, system and device

Abstract

The invention discloses a monitoring video event analysis method, system and device for monitoring equipment, and relates to the technical field of video abstraction. Obtaining a monitoring video recorded by monitoring equipment, and screening out a moving video clip from the monitoring video; extracting a video frame feature and an audio feature from the video clip, determining a plurality of event clips in the video clip according to the video frame feature and the audio feature, and determining an abstract of each event clip; storing the abstract of each event segment and the corresponding timestamp in a database; and determining event segments recorded by the monitoring equipment through the abstracts in the database and the corresponding timestamps. According to the method, the monitoring video of the monitoring equipment can be effectively analyzed, so that the time for finding the related video clip of the target event is shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of video summarization, and in particular to a monitoring video event analysis method, system and device for monitoring equipment. Background Art

[0002] Security monitoring equipment provides real-time monitoring and video playback functions, which can help improve security. With the continuous advancement of science and technology, the types and functions of security equipment are increasing, and they are becoming more and more popular. Whether it is a residence, store, office building or public place, monitoring equipment can capture and record video clips at any time. The video clips recorded by monitoring equipment can be used as evidence to provide important evidence support in the event of disputes or crimes. More and more people choose to install security monitoring equipment to prevent security risks such as intrusion and theft. Security monitoring strengthens the security of property and personnel and reduces the risks brought by security risks.

[0003] In the prior art, the amount of video clip data recorded by monitoring equipment is too large. Usually, most of the video clips are redundant clips that do not contain any valid information, and some scenarios may involve monitoring videos from multiple channels. This requires people to spend a lot of time reviewing massive amounts of monitoring videos to find relevant video clips involving the target event.

[0004] However, the prior art lacks a method for effectively analyzing surveillance videos of surveillance equipment, which results in a long time required to find relevant video clips of the target event. Summary of the invention

[0005] Based on this, it is necessary to provide a method, system and device for analyzing surveillance video events of a monitoring device in response to the above technical problems. The method can effectively analyze the surveillance video of the monitoring device, thereby reducing the time to find relevant video clips of the target event.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a monitoring video event analysis method of a monitoring device, comprising:

[0008] Obtain surveillance video recorded by surveillance equipment and filter out motion video clips from the surveillance video;

[0009] Extracting video frame features and audio features from the video clip, and determining multiple event clips in the video clip based on the video frame features and the audio features;

[0010] Determine the summary of each event segment;

[0011] The summary and corresponding timestamp of each event segment are stored in the database; the event segment recorded by the monitoring device is determined by the summary and corresponding timestamp in the database.

[0012] Preferably, obtaining the surveillance video recorded by the surveillance device includes:

[0013] Detect supported surveillance devices in the local area network through the Open Network Video Interface Forum protocol;

[0014] Bind the monitoring device according to its username and password;

[0015] Obtaining surveillance video recorded by the surveillance device from the surveillance device through a real-time streaming protocol;

[0016] The surveillance video is segmented and packaged into video files and stored in a video pool, and the surveillance video is obtained from the video pool.

[0017] Preferably, extracting audio features from the video clip includes:

[0018] Resample the audio in the video clip to obtain the audio in the video clip;

[0019] According to the preset time interval and time window, the audio is divided into frames to obtain multiple audio frames;

[0020] Perform short-time Fourier transform on each audio frame to obtain multiple frequency spectra;

[0021] Taking the logarithm of the weighted average of multiple frequency spectra, and mapping the logarithm-based frequency spectra to the Mel frequency scale to obtain a Mel frequency spectrum;

[0022] The Mel-spectrogram is input into the audio feature extraction model to obtain audio features.

[0023] Preferably, determining a plurality of event segments in a video segment according to the video frame features and the audio features comprises:

[0024] Determine event segment boundaries in the video segment based on video frame features and audio features;

[0025] Segment the video segments according to the event segment boundaries to obtain segmented video frame features and audio features, as well as event segment timestamps;

[0026] The segmented video frame features, audio features, and event segment timestamps are fused to obtain multiple event segments.

[0027] Preferably, the method further comprises:

[0028] In response to the user's request for video playback, the timeline of the video in the monitoring device currently selected by the user is displayed, and the part with the event clip is highlighted;

[0029] In response to the user's summary playback request, the timestamps and corresponding summaries of all event segments in the monitoring device currently selected by the user are displayed, and when the user clicks on the summary of the event segment, the corresponding event segment is displayed.

[0030] Preferably, a summary of each event segment is determined, including:

[0031] Each event segment is input into the video summary model to determine the summary of each event segment. The construction process of the video summary model includes:

[0032] Get multiple convolutional neural networks of different depths;

[0033] The convolutional neural networks are trained respectively by using sample training data to obtain multiple initial video summary models; the sample training data includes sample event segments and summaries corresponding to the sample event segments;

[0034] Each initial video summary model is evaluated respectively, and the initial video summary model with the best evaluation index is determined as the video summary model.

[0035] The present invention provides a monitoring video event analysis system for monitoring equipment, the system comprising a video management module and an analyzer module;

[0036] A video management module, used to obtain surveillance videos recorded by surveillance equipment;

[0037] The analyzer module is used to filter out moving video clips from surveillance videos; extract video frame features and audio features from the video clips, and determine multiple event clips in the video clips based on the video frame features and audio features; determine the summary of each event clip; save the summary of each event clip and the corresponding timestamp in a database; and determine the event clip recorded by the surveillance device through the summary in the database and the corresponding timestamp.

[0038] Preferably, the system interacts with the user through a Web page, and the system functions include user management, system overview, real-time image viewing, video playback, and system configuration functions.

[0039] Preferably, the system further comprises an evaluation module;

[0040] The evaluation module is used to evaluate the computing performance of the hardware platform to which the system belongs, and determine the video summary model according to the computing performance of the hardware platform and the number of bound monitoring devices.

[0041] The present invention provides a monitoring video event analysis device of a monitoring device, comprising:

[0042] A screening module is used to obtain surveillance videos recorded by surveillance equipment and screen out motion video clips from the surveillance videos;

[0043] A first determination module is used to extract video frame features and audio features from the video segment, and determine multiple event segments in the video segment according to the video frame features and the audio features;

[0044] A second determination module is used to determine a summary of each event segment;

[0045] The third determination module is used to store the summary and corresponding timestamp of each event segment in the database; and determine the event segment recorded by the monitoring device through the summary and corresponding timestamp in the database.

[0046] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the monitoring video event analysis method of the monitoring device is implemented.

[0047] The present invention provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the monitoring video event analysis method of the monitoring device when executing the program.

[0048] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0049] In the present invention, by analyzing the event fragments in the surveillance video and generating a summary of the event fragments, the summary of the event fragments and the corresponding timestamp are stored in a database. In this way, the event fragments in the surveillance video can be obtained more time-savingly and intuitively, and the target event can be found more quickly from a large number of surveillance videos without spending a lot of time viewing surveillance playback. This method can reduce the time to find the relevant video fragments of the target event. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0051] Figure 1 A schematic diagram of a flow chart of a monitoring video event analysis method for a monitoring device provided by the present invention;

[0052] Figure 2 A schematic flow chart of another monitoring video event analysis method for monitoring equipment provided by the present invention;

[0053] Figure 3 A network environment topology diagram provided by the present invention;

[0054] Figure 4 A schematic diagram of a monitoring video event analysis device for a monitoring device provided by the present invention;

[0055] Figure 5 A schematic diagram of a computer device for implementing a monitoring video event analysis method of a monitoring device provided by the present invention.

[0056] Description of reference numerals:

[0057] 101. Gateway; 102. Main router; 103. Switch; 104. Network camera; 105. Wireless router; 106. User terminal; 107. Surveillance video event analysis system of monitoring equipment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] With the development of monitoring technology, a series of technologies have been proposed to try to solve the above problems. A common method is to mark video clips with motion scenes or video clips with people through motion detection or human detection technology. This method enables people to find target clips more quickly from video clips recorded by monitoring equipment. Another method is to deploy edge computing devices in the system. Users can deploy corresponding artificial intelligence (AI) models in edge computing devices as needed to detect monitoring videos through AI. However, these methods still have some disadvantages. First, the types of events that can be detected by adding detection technology methods to the cameras of monitoring equipment are limited. For example, only clips with people or vehicles can be detected. Moreover, this detection method usually depends on the model of the monitoring camera. Different models of monitoring cameras may support different types of events to be detected. This means that in some occasions where there are many monitoring cameras, if there are new needs, it is necessary to replace the monitoring cameras on a large scale, and even the monitoring equipment may need to be replaced. Moreover, the more events the equipment supports to detect, the more expensive the equipment will be. Second, although detection technology can filter out video clips that may contain valid information, it still takes a lot of time to play back these filtered clips and find the target clips. The reason is that the video contains too much redundant information. Third, although edge computing devices are powerful, they are expensive and require users to have a high level of technical skills. The deployment cost is too high for ordinary users.

[0060] Based on this, the present application provides another surveillance video event analysis method of a surveillance device, which first obtains the surveillance video recorded by the surveillance device, and selects the moving video clips from the surveillance video; extracts the video frame features and audio features from the video clips, and determines multiple event clips in the video clips based on the video frame features and audio features, and determines the summary of each event clip; saves the summary of each event clip and the corresponding timestamp in the database; determines the event clip recorded by the surveillance device through the summary in the database and the corresponding timestamp. In this way, the user can quickly find the target event and locate the target clip through the summary in the database. Compared with the method of reviewing the video, this method significantly reduces the time required for the user to retrieve the target event clip.

[0061] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0062] The present invention provides a surveillance video event analysis system of a monitoring device, the system comprising a video management module and an analyzer module; the video management module is used to obtain the surveillance video recorded by the monitoring device; the analyzer module is used to screen out moving video clips from the surveillance video; extract video frame features and audio features from the video clips, and determine multiple event clips in the video clips according to the video frame features and the audio features; determine a summary of each event clip; save the summary of each event clip and the corresponding timestamp in a database; determine the event clip recorded by the monitoring device through the summary in the database and the corresponding timestamp.

[0063] The system can be deployed on a general platform of x86 and Advanced RISC Machines (ARM) architecture. It can discover surveillance cameras in the local area network through the Open Network Video Interface Forum (ONVIF) protocol, and obtain the recording images of the surveillance equipment through the Real-Time Streaming Protocol (RTSP) protocol and record and analyze the images. The system extracts the feature information of the two modalities of image and audio of the surveillance video for feature fusion, summarizes the video clips through deep learning technology, and generates text descriptions. Users can quickly find the target event and locate the target clip through the summary text.

[0064] Specifically, taking the monitoring device as a network camera as an example, after the system is connected to the LAN, it discovers the supported network cameras in the LAN through the ONVIF protocol, lists the network cameras for users to choose to add, and the user enters the network camera username and password. After successful verification, the network camera can be added to the system as a new channel.

[0065] The video management module is responsible for obtaining audio and video streams from network cameras in the local area network through the RTSP protocol and encoding them into video files for storage. Encoding and saving video files includes: encoding and decoding the audio and video streams through FFMpeg, encapsulating the audio and video streams into video files (surveillance videos) and storing them in the video pool. Each video file can be 5 minutes long.

[0066] While the video management module is recording, the analyzer module analyzes the surveillance videos in the video pool and inputs the videos in the video pool into the neural network model. The neural network model outputs the summary and timestamp corresponding to the event fragment, and saves the summary and timestamp corresponding to the event fragment into the database.

[0067] In the neural network model, the neural network model selects moving video clips from the surveillance video; extracts video frame features and audio features from the video clips, and determines multiple event clips in the video clips based on the video frame features and audio features; and then determines the summary of each event clip.

[0068] Optionally, the system interacts with the user through a Web page. The system's functions include user management, system overview, real-time image viewing, video playback, and system configuration functions.

[0069] The system interacts with users through World Wide Web (Web) pages developed by Vue.js+Django.

[0070] The user management function can create users for the system, set user names and passwords, and assign user permissions, making it convenient and flexible to deploy different personnel to use the system.

[0071] The system overview page displays the basic status information of the system, as well as the recent event list and recent event statistics. Furthermore, the basic status information of the system includes the system version number, model version number, system running time, system network information, central processing unit (CPU) load, graphics processing unit (GPU) load, memory load and storage space. The recent event list displays the latest events in all channels, as well as the text summary, timestamp, dynamic image and event importance level of the corresponding event. The recent event statistics show the number of events of different importance levels in the past 24 hours in the form of charts.

[0072] The real-time image viewing function can view the real-time images of network cameras in each channel. Furthermore, the real-time images can be viewed on a single channel, or one channel, four channels, six channels, or nine channels can be displayed on the screen at the same time. When viewing a channel alone, the real-time image of the channel can be displayed and the real-time audio of the channel can be played at the same time (the audio function of the network camera needs to be enabled).

[0073] The video playback function can playback the videos saved in the hard disk, including the traditional playback function and the summary playback function. Furthermore, when using the traditional video playback function, the system will also display the timeline of the video in the currently selected channel and highlight the part where the event occurred. When using the summary playback function, the system will also list the timestamps and text summaries of all events in the currently selected channel. Users can jump to the corresponding video clip by clicking on the text summary of the event.

[0074] The configuration function can change various system configurations, including system network configuration, adding, deleting, and managing network cameras, video configuration, hard disk management, analyzer configuration, software updates, performance testing, and system logs.

[0075] Optionally, the system further includes an evaluation module; the evaluation module is used to evaluate the computing performance of the hardware platform to which the system belongs, and determine the video summary model according to the computing performance of the hardware platform and the number of bound monitoring devices. Specifically, the video summary model with a suitable number of parameters can be selected according to the computing performance of the hardware platform and the number of configured video channels.

[0076] For example, to adapt to platforms with different performance, three video summary models with different parameter amounts are provided: Lite, Base, and Full, with the parameter amounts increasing in sequence. The constructed video summary model is trained and evaluated, and the model weights with the best evaluation indicators are selected from the three models of different sizes; according to the system performance, the model of the appropriate size is selected for loading.

[0077] The surveillance video event analysis system of the monitoring equipment provided by the present invention can be deployed on an independent platform without affecting the work of the original monitoring equipment and video recorder. The system is deployed in a local area network, discovers and obtains the video stream of the monitoring equipment in the local area network through ONVIF and RTSP protocols, inputs the video stream into the system for analysis, and displays the analysis results to the user through a Web interface or application. Compared with traditional monitoring equipment and edge computing devices, the system can more flexibly identify events in the video and can generate corresponding text summaries. Users can obtain information in the surveillance video more time-saving and intuitively, and find target events faster from a large number of monitoring clips without spending a lot of time viewing monitoring playback. The system deployment is flexible and simple, friendly to non-professional users, and can be deployed in the network without changing the original network architecture and equipment. The algorithm model can be updated regularly, and the system has good hardware compatibility and supports most mainstream commercial and professional hardware on the market.

[0078] In an exemplary embodiment, based on the monitoring video event analysis system of the monitoring device, the present invention also provides a monitoring video event analysis method for the monitoring device, which is applied to the monitoring video event analysis system of the monitoring device, such as Figure 1 As shown, this embodiment includes the following steps:

[0079] S101, obtaining a surveillance video recorded by a surveillance device, and filtering out motion video clips from the surveillance video.

[0080] In one embodiment, obtaining surveillance video recorded by a surveillance device includes: detecting supported surveillance devices in a local area network through an open network video interface forum protocol; binding the surveillance device according to a user name and password of the surveillance device; obtaining surveillance video recorded by the surveillance device from the surveillance device through a real-time streaming protocol; packaging the surveillance video into segments and storing them in a video pool, and obtaining the surveillance video from the video pool.

[0081] Among them, the supported monitoring devices in the local area network can include multiple ones, and the required monitoring devices can be bound according to needs; for example, in the user selection page, including multiple monitoring devices supported by the local area network, the user can click on a monitoring device and enter the user name and password of the monitoring device to bind the monitoring device. In this way, the monitoring video recorded by the monitoring device can be obtained from the bound monitoring device through the real-time streaming protocol.

[0082] The monitoring video is segmented and packaged into video files and stored in the video pool, which may include: segmenting the monitoring video according to time intervals, and storing each video file in the video pool. For example, each video file may be a 5-minute video.

[0083] Optionally, when it is necessary to analyze the surveillance video of the surveillance device, the surveillance video can be directly obtained from the video pool.

[0084] Screening out moving video clips from surveillance video, including: using frame difference method to screen out moving video clips from surveillance video.

[0085] The basic idea of ​​the frame difference method is to detect moving objects by comparing the differences between two adjacent frames. When an object moves in a scene, there will be obvious differences in the location of the object between two adjacent frames. A suitable threshold is set to determine whether this difference is sufficient to indicate that motion has occurred. Specifically, for the t-th frame image and the t+1-th frame image in the video, the grayscale values ​​of their corresponding pixels are subtracted (if it is a color image, it is usually converted to a grayscale image before the operation, which can simplify the calculation and highlight the main brightness difference). In the obtained difference image, the pixel value of the area where the moving object is located will be relatively large, while the pixel value of the non-moving area will be close to 0.

[0086] S102: extracting video frame features and audio features from the video segment, and determining a plurality of event segments in the video segment according to the video frame features and the audio features.

[0087] In one embodiment, a deep residual network (ResNet) is used to extract video frame features from video clips, and video summary models of different sizes use ResNet networks of different depths to extract video frame features; for example, the video summary model is a Lite model, using ResNet-18; the video summary model is a Base model, using ResNet-34; the video summary model is a Full model, using ResNet-50. The Lite model represents a lightweight model, the Base model represents a baseline model, and the Full model represents a complete model.

[0088] Taking the video summary model as the Base model as an example, the ResNet network includes an input layer, 5 convolution blocks and an output layer. Specifically, the input layer: the input video frame size is 512*512*3; Convolution block 1: contains a convolution layer with a convolution kernel size of 7*7, 64 channels, and a stride of 2; Convolution block 2: contains a maximum pooling layer with a size of 3*3, a stride of 2, and 3 BasicBlocks with a channel number of 64; Convolution block 3: contains 4 BasicBlocks with a channel number of 128; Convolution block 4: contains 6 BasicBlocks with a channel number of 256; Convolution block 5: contains 3 BasicBlocks with a channel number of 512; Output layer: outputs a feature vector of size 7*7.

[0089] Specifically, the calculation steps of each BasicBlock are as follows: the input x passes through a convolution layer and batch normalization, and then the ReLU activation function is applied to obtain the feature F(x).

[0090] F(x) = ReLU(BN(Conv(x))) (1)

[0091] After the second convolutional layer and batch normalization, F(x) outputs the feature H(x).

[0092] H(x)=BN(Conv(F(x))) (2)

[0093] Finally, H(x) is added to the input x. If the dimensions of the input and output H(x) are the same, they are added directly. If the dimensions are different, they are resized by 1*1 convolution. The ReLU activation function is applied to the added vector to obtain the final output y.

[0094] y=ReLU(H(x)+x) (3)

[0095] Extracting audio features from the video clip, including: extracting the Mel spectrum of the audio in the video clip through a VGGish network, and extracting audio features from the Mel spectrum of the audio.

[0096] Therefore, extracting audio features from a video clip may include: resampling the audio in the video clip to obtain the audio in the video clip; dividing the audio into frames according to preset time intervals and time windows to obtain multiple audio frames; performing short-time Fourier transform on each audio frame to obtain multiple frequency spectra; taking the logarithm of the multiple frequency spectra after weighted averaging, and mapping the logarithm of the frequency spectra to the Mel frequency scale to obtain a Mel spectrum graph; and inputting the Mel spectrum graph into an audio feature extraction model to obtain audio features.

[0097] Specifically, the audio is resampled to 16kHz to obtain the audio in the video clip; the audio is divided into frames using a 33ms window, and the windows overlap every 10ms to obtain multiple audio frames; each audio frame is short-time Fourier transformed to obtain multiple frequency spectra; the multiple frequency spectra are weighted averaged and then logarithmized, mapped to the Mel frequency scale, and standardized to form a 64-dimensional Mel spectrum diagram.

[0098] The audio feature extraction model may be a VGGish network. The Mel spectrum graph is input into the VGGish network to obtain the Mel spectrum of the audio in the video clip, and the audio features are extracted from the Mel spectrum of the audio.

[0099] Among them, the VGGish network consists of 5 layers. The first layer includes a convolution layer with a convolution kernel size of 3*3 and 64 channels, and a maximum pooling layer with a kernel size of 2*2 and a stride of 2; the second layer includes a convolution layer with a convolution kernel size of 3*3 and 128 channels, and a maximum pooling layer with a kernel size of 2*2 and a stride of 2; the third layer includes two convolution layers with a convolution kernel size of 3*3 and 256 channels, and a maximum pooling layer with a kernel size of 2*2 and a stride of 2; the fourth layer includes two convolution layers with a convolution kernel size of 3*3 and 512 channels, and a maximum pooling layer with a kernel size of 2*2 and a stride of 2; the output of the fourth layer is flattened and input into the fifth layer. The fifth layer includes two fully connected layers with a size of 4096 and an activation function of ReLU, and a fully connected layer with a size of 1000, and outputs a 128-dimensional feature vector.

[0100] Optionally, multiple event segments in a video clip are determined based on video frame features and audio features, including: determining event segment boundaries in the video clip based on video frame features and audio features; segmenting the video clip according to the event segment boundaries to obtain segmented video frame features and audio features, and event segment timestamps; and fusing the segmented video frame features and audio features, and event segment timestamps to obtain multiple event segments.

[0101] According to the video frame features and the audio features, the event segment boundary in the video segment is determined, including: inputting the video frame features and the audio features into an event segment boundary detection model, aligning the video features and the audio features on a time scale in the event segment boundary detection model, flattening the video feature vector, and splicing it with the audio feature vector, the spliced ​​vector passes through two layers of fully connected layers with a size of 512 and an activation function of ReLU, and an output layer with a size of 100 and an activation function of Sigmoid, the output vector represents the possibility that the current time frame is an event boundary, the output vector is 0 or 1, 0 represents a non-event segment boundary, and 1 represents an event segment boundary; therefore, the event segment boundary can be determined according to the value of the output vector of each time frame.

[0102] Among them, when training the event segment boundary detection model, a binary cross entropy loss function can be used, specifically:

[0103]

[0104] Among them, n represents the number of training samples, y i represents the predicted label of the i-th training sample, and y represents the true label of the training sample (0 is the non-event segment boundary, and 1 is the event segment boundary).

[0105] Specifically, the segmented video frame features and audio features, as well as the event segment timestamps are fused to obtain multiple event segments, including: constructing three different embedding layers for the segmented video frame features, audio features, and event segment timestamps, respectively, and embedding the segmented video frame features, audio features, and event segment timestamps to obtain corresponding embedding vectors; after fusing the video embedding vector, audio embedding vector, and timestamp embedding vector, the event segment is obtained. Embedding refers to the representation method of mapping high-dimensional discrete data (such as categories, words, etc.) to a low-dimensional continuous vector space, which means converting video frame features, audio features, and event segment event stamps into corresponding embedding vectors.

[0106] S103, determining a summary of each event segment.

[0107] Each event segment may be input into the video summary model to determine a summary of each event segment.

[0108] The construction process of the video summary model includes: obtaining multiple convolutional neural networks of different depths; training each convolutional neural network respectively through sample training data to obtain multiple initial video summary models; the sample training data includes sample event segments and summaries corresponding to the sample event segments; evaluating each initial video summary model respectively, and determining the initial video summary model with the best evaluation index as the video summary model.

[0109] Among them, multiple convolutional neural networks of different depths may include a Lite model, a Base model, and a Full model, and the number of parameters of Lite, Base, and Full increases in sequence. Multiple convolutional neural networks of different depths may all be Transformer structures, and the Transformer includes a 6-layer encoder and a 6-layer decoder.

[0110] Each encoder processes the input data through the self-attention mechanism and feedforward neural network. After the input vector passes through the 6-layer encoder, it finally outputs the encoded sequence vector. Each decoder uses the multi-head attention mechanism and feedforward neural network, combined with the sequence vector encoded by the encoder, to autoregressively generate each element of the target sequence step by step.

[0111] Each encoder and decoder contains a multi-head attention layer and a feedforward neural network layer, and each layer uses residual connections.

[0112] The calculation method of self-attention is:

[0113]

[0114] Among them, Q represents query, K represents key, V represents value, and d k is the vector dimension.

[0115] The feedforward neural network layer contains two fully connected layers with ReLU activation function. The calculation method of the feedforward neural network is:

[0116] FFN(x)=max(0,xW1+b1)W2+b2 (6)

[0117] Where x represents the input of the feedforward neural network layer, W n and b n They represent the weight and bias of the nth fully connected layer respectively.

[0118] The calculation method of residual connection is:

[0119] y=LayerNorm(x+SubLayer(x)) (7)

[0120] Among them, x represents the input of the residual connection, y represents the output of the residual connection, and SubLayer(x) is the attention layer or the feedforward neural network layer.

[0121] After multiple initial video summary models are obtained through training of the above network structure, the quality of the summaries generated by each initial video summary model can be evaluated by two indicators: ROUGE-N score and ROUGE-L score. ROUGE is an evaluation indicator commonly used in NLP tasks, which is used to measure the similarity between the generated summary text and the reference summary text. ROUGE-N and ROUGE-L are variants of the ROUGE indicator. The higher the ROUGE score, the higher the similarity between the generated summary text and the reference summary text.

[0122] The ROUGE-N index is an evaluation index based on n-gram, where N represents the length of n-gram. For example, ROUGE-1 and ROUGE-2 represent 1-gram and 2-gram respectively. The calculation method is:

[0123]

[0124] The denominator represents the number of n-grams in the reference summary text, and the numerator represents the number of n-grams that overlap between the reference summary text and the generated summary text.

[0125] ROUGE-L calculates the similarity between the generated summary text and the reference summary text based on the longest common subsequence. The calculation method is:

[0126]

[0127] Among them, X is the generated summary text, Y is the reference summary text, m is the length of the generated summary text, n is the length of the reference summary text, and the LCS() function means calculating the longest common subsequence length of two sequences.

[0128] The sum of ROUGE-N and ROUGE-L under each model can be calculated, and the initial video summary model with the largest sum is determined as the video summary model.

[0129] S104, saving the summary and corresponding timestamp of each event segment in a database; determining the event segment recorded by the monitoring device through the summary and corresponding timestamp in the database.

[0130] After obtaining the summary and corresponding timestamp of each event clip, it can be saved in the database. In this way, when it is necessary to search for events in the surveillance video of the monitoring device, the corresponding summary and timestamp can be directly searched from the database, so as to locate the video clip of the event according to the summary and timestamp.

[0131] In an exemplary embodiment, the embodiment includes: responding to a user's request for video playback, displaying the timeline of the video in the monitoring device currently selected by the user, and highlighting the portion where event clips exist; responding to a user's request for summary playback, displaying the timestamps and corresponding summaries of all event clips in the monitoring device currently selected by the user, and displaying the corresponding event clip when the user clicks on the summary of the event clip.

[0132] In the present invention, a text summary of a surveillance video is generated by a neural network, so that a user can obtain the information recorded in the surveillance video more intuitively and can find a target segment from a large number of saved surveillance videos more quickly.

[0133] In an exemplary embodiment, the present invention also provides a monitoring video event analysis method of a monitoring device, such as Figure 2 As shown, the embodiment includes: firstly obtaining surveillance video, and filtering motion clips in the surveillance video by using a frame difference method, extracting video feature vectors of the video in the motion clip by using a ResNet network, and extracting audio feature vectors of the audio in the motion clip by using a VGGish network, and inputting the video feature vector and the audio feature vector into an output layer, performing event boundary detection to obtain event boundaries of event clips, and inputting the video features after event boundary slicing, the audio features after event boundary slicing, and the event clip timestamp into an embedding layer for embedding to obtain an embedding vector, and after fusing the video embedding vector, the audio embedding vector, and the timestamp embedding vector, the video embedding vector, and the timestamp embedding vector are input into a video summary model Transformer to obtain a summary of the video clip.

[0134] In an exemplary embodiment, the network architecture of the deployment environment is as follows: Figure 3 As shown, it includes a gateway 101, a main router 102, multiple switches 103, multiple network cameras 104, a wireless router 105, multiple user terminals 106 and a monitoring device surveillance video event analysis system 107. The hardware platform configuration of the monitoring device surveillance video event analysis system is shown in Table 1.

[0135] Motherboard AMDX670 chipset motherboard CPU AMD Ryzen 9 7900X RAM DDR56000MHz16GB*2 GPU NVIDIA RTX3080Ti 12GB GDDR6X System installation hard disk 512GB PCIe 4.0 NVMe SSD Video storage hard disk 8TB SATA 6Gb / s mechanical hard disk

[0136] like Figure 3 As shown, the surveillance video event analysis system of the monitoring equipment is connected to the network from the switch through a Category 6e network cable, and the access network segment is 192.168.31.0 / 24.

[0137] The main router's Internet access mode is Dynamic Host Configuration Protocol (DHCP) mode. In the router's configuration page, a static IP of 192.168.31.240 is manually assigned to this system.

[0138] Burn the installation image of this system to the external storage, connect the external storage to the system, and select to use the external storage to start in the system. The installation image starts the installation wizard and installs the software into the system.

[0139] After the system installation is completed, the user uses the terminal device to access the local area network, enter the system IP address 192.168.31.240, log in to the system background, and create a management account.

[0140] Open system configuration and initialize the storage hard disk.

[0141] Scan the network cameras under the local area network. The system finds 6 network cameras under the same network segment. After entering the camera password, the 6 network cameras are added to the system.

[0142] Configure the recording plan to record 24 / 7 and save the recordings to storage hard disk 1.

[0143] Run the Performance Test Wizard and set the event detection model to use the Base model according to the recommended configuration provided by the Performance Test Wizard.

[0144] After the system has been working for a period of time, you need to find the video clips associated with a certain event from the clips recorded by the network camera. Use a terminal device to access the LAN, visit 192.168.31.240, and log in to the system backend. Enter the video playback function and choose to use summary playback. Select the time period when the event may occur, and you can see the text summary of all events recorded by the network camera during this time period. Filter out the target event based on the summary and jump to the associated video clip. Click "Lock" to lock the target video clip. The locked video clip will not be overwritten by the loop recording. Click "Download" to download the target video clip to an external storage device.

[0145] In an exemplary embodiment, the training process of the event segment boundary detection model and the video summary model is described in detail. The model size trained in this embodiment is the Base model. The specific steps of model training are as follows:

[0146] Collect a dataset for training, which includes surveillance videos (video and audio), event boundary event stamp labels corresponding to the surveillance videos, and summary text of the events corresponding to the surveillance videos.

[0147] The data is preprocessed, the frame difference method is used to select video clips with motion scenes, and a Chinese dictionary is constructed to add special tags to the summary text.

[0148] The video frame size is adjusted to 512*512, a ResNet-34 model is constructed, the ResNet-34 model parameters are initialized, and the video frame features are extracted using ResNet-34.

[0149] The audio is resampled to 16kHz and divided into frames using a 33ms window, with the windows overlapping every 10ms. Each audio frame is short-time Fourier transformed to obtain a frequency spectrum. The frequency spectrum is weighted averaged and then logarithmized, mapped to the Mel frequency scale, and normalized to form a 64-dimensional Mel spectrum.

[0150] Build a VGGish model, initialize the VGGish model parameters, and use the VGGish model to extract audio features from the Mel spectrogram.

[0151] The video frame feature vector is flattened and concatenated with the audio feature vector. The concatenated vector is passed through two fully connected layers with a size of 512 and an activation function of ReLU, and an output layer with a size of 100 and an activation function of Sigmoid to obtain an output vector, which represents the possibility that the current time frame is an event boundary. The event segment boundary detection model loss is calculated using the binary cross entropy loss function.

[0152] Use the Adam optimizer and gradient descent method to train the event segment boundary detection model, perform back propagation based on the loss and gradient graph, set the model training parameters, the learning rate is 1e-5, the number of iterations is 100, and the early stopping strategy is set. When the loss curve tends to be stable, the model is judged to have converged and the training ends.

[0153] According to the event boundaries predicted by the event segment boundary detection model, the video is sliced ​​by event, the video feature vector and the audio feature vector are aligned in the time dimension, and the corresponding timestamps are generated.

[0154] Three different embedding layers are constructed for the video feature vector, audio feature vector and timestamp respectively, and the video feature vector, audio feature vector and timestamp are embedded to obtain the embedding vector.

[0155] After the video embedding vector, audio embedding vector, and timestamp embedding vector are fused, they are input into the encoder module of the video summary model Transformer. After passing through the 6 encoder layers in the encoder module, the encoded sequence is output.

[0156] The event summary text is converted into tokens according to the dictionary and input into the embedding layer to obtain the text embedding vector. The text embedding vector is input into the decoder module, and the sequence vector encoded by the encoder is input into each decoder layer in the decoder module of the video summary model Transformer. After the text embedding vector passes through the 6 decoder layers in the decoder module, it passes through the output layer to output the prediction vector, which represents the predicted word and then outputs the index in the dictionary.

[0157] Each encoder layer and decoder layer contains a multi-head attention layer and a feed-forward neural network layer, and each layer uses residual connections.

[0158] The video summarization model is trained using the Adam optimizer and gradient descent method. Back propagation is performed based on the loss and gradient graph. The model training parameters are set, the learning rate is 1e-3, the number of iterations is 250, and the early stopping strategy is set. When the loss curve tends to be stable, the model is judged to have converged and the training is terminated.

[0159] After the model training is completed, the quality of the summary generated by the model is judged by two indicators: ROUGE-N score and ROUGE-L score. The higher the ROUGE score, the higher the similarity between the generated text and the reference text.

[0160] When applying the monitoring video event analysis method provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0161] The above is a monitoring video event analysis method for a monitoring device provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding monitoring video event analysis device for a monitoring device, such as Figure 4 shown.

[0162] Figure 4 A schematic diagram of a surveillance video event analysis device of a surveillance device provided by the present invention, the device 400 includes:

[0163] The screening module 401 is used to obtain the surveillance video recorded by the surveillance device and screen out the moving video clips from the surveillance video;

[0164] A first determination module 402 is used to extract video frame features and audio features from the video segment, and determine multiple event segments in the video segment according to the video frame features and the audio features;

[0165] A second determination module 403 is used to determine a summary of each event segment;

[0166] The third determination module 404 is used to store the summary and corresponding timestamp of each event segment in the database; and determine the event segment recorded by the monitoring device through the summary and corresponding timestamp in the database.

[0167] The specific definition of the surveillance video event analysis device of the monitoring device can be found in the definition of the surveillance video event analysis method of the monitoring device above, which will not be repeated here. Each module in the surveillance video event analysis device of the monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0168] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A monitoring video event analysis method for a monitoring device is provided.

[0169] The present invention also provides Figure 5 The structural diagram of the computer device shown in FIG. Figure 5 As shown in the figure, at the hardware level, the computer device includes a processor, a graphics processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it. The graphics processor is used to accelerate the calculation of the deep learning model to achieve the above Figure 1 A monitoring video event analysis method for a monitoring device is provided.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0171] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A monitoring video event analysis method for a monitoring device, characterized in that: The method comprises: Obtaining surveillance video recorded by a surveillance device, and filtering out motion video clips from the surveillance video; Extracting video frame features and audio features from the video clip, and determining a plurality of event clips in the video clip according to the video frame features and the audio features; Determine the summary of each event segment; The summary and corresponding timestamp of each event segment are stored in a database; and the event segment recorded by the monitoring device is determined by the summary and corresponding timestamp in the database.

2. The method according to claim 1, characterized in that The obtaining of the surveillance video recorded by the surveillance device includes: Detect supported surveillance devices in the local area network through the Open Network Video Interface Forum protocol; Binding the monitoring device according to the user name and password of the monitoring device; Acquire the surveillance video recorded by the surveillance device from the surveillance device through a real-time streaming protocol; The monitoring video is segmented and packaged into video files and stored in a video pool, and the monitoring video is obtained from the video pool.

3. The method according to claim 1, characterized in that Extracting audio features from the video clip includes: Resampling the audio in the video clip to obtain the audio in the video clip; Divide the audio into frames according to preset time intervals and time windows to obtain multiple audio frames; Performing short-time Fourier transform on each of the audio frames to obtain multiple frequency spectra; Taking the logarithm of the weighted average of multiple frequency spectra, and mapping the logarithm-based frequency spectra to the Mel frequency scale to obtain a Mel frequency spectrum; The mel-spectrogram is input into an audio feature extraction model to obtain the audio feature.

4. The method according to claim 1, characterized in that: The determining, according to the video frame features and the audio features, a plurality of event segments in the video segment comprises: Determining event segment boundaries in the video segment according to the video frame features and the audio features; Segmenting the video segment according to the event segment boundary to obtain segmented video frame features and audio features, as well as event segment timestamps; The segmented video frame features and audio features, as well as event segment timestamps are fused to obtain the multiple event segments.

5. The method according to claim 1, characterized in that The method further comprises: In response to a video playback request from a user, display the timeline of the video in the monitoring device currently selected by the user, and highlight the part where the event clip exists; In response to the user's summary playback request, the timestamps and corresponding summaries of all event segments in the monitoring device currently selected by the user are displayed, and when the user clicks on the summary of an event segment, the corresponding event segment is displayed.

6. The method according to claim 1, characterized in that The summary of each event segment is determined, including: Input each event segment into the video summary model respectively to determine the summary of each event segment; the construction process of the video summary model includes: Get multiple convolutional neural networks of different depths; The convolutional neural networks are trained respectively by using sample training data to obtain multiple initial video summary models; the sample training data includes sample event segments and summaries corresponding to the sample event segments; Each of the initial video summary models is evaluated respectively, and the initial video summary model with the best evaluation index is determined as the video summary model.

7. A surveillance video event analysis system for a surveillance device, characterized in that: The system includes a video management module and an analyzer module; The video recording management module is used to obtain the surveillance video recorded by the surveillance equipment; The analyzer module is used to filter out moving video clips from the surveillance video; Extracting video frame features and audio features from the video clip, and determining multiple event clips in the video clip according to the video frame features and the audio features; determining a summary of each event clip; and storing the summary of each event clip and a corresponding timestamp in a database; The event segments recorded by the monitoring device are determined through the summaries in the database and the corresponding timestamps.

8. The system according to claim 7, characterized in that The system interacts with users through Web pages. The functions of the system include user management, system overview, real-time image viewing, video playback and system configuration functions.

9. The system according to claim 7, characterized in that The system also includes an evaluation module; The evaluation module is used to evaluate the computing performance of the hardware platform to which the system belongs, and determine the video summary model according to the computing performance of the hardware platform and the number of bound monitoring devices.

10. A monitoring video event analysis device for monitoring equipment, characterized in that: The device comprises: A screening module, used to obtain surveillance videos recorded by surveillance equipment and screen out motion video clips from the surveillance videos; A first determination module is used to extract video frame features and audio features from the video segment, and determine multiple event segments in the video segment according to the video frame features and the audio features; A second determination module is used to determine a summary of each event segment; The third determination module is used to store the summary and corresponding timestamp of each event segment in a database; and determine the event segment recorded by the monitoring device through the summary and corresponding timestamp in the database.

Images