Convolutional neural network-based real-time monitoring video and emergency scene fusion method

Abstract

The invention belongs to the technical field of video monitoring and emergency scene fusion, and particularly relates to a real-time monitoring video and emergency scene fusion method based on a convolutional neural network. According to the method, a unified video platform is utilized to integrate multi-source video resources, format conversion and coding are carried out, and compatibility is ensured; the video quality is enhanced by adopting adaptive histogram equalization; an emergency scene database is established, abnormity is analyzed through a convolutional neural network and is matched with the database, and video and emergency scene fusion is realized; and the fused videos are stored in a centralized manner, and key frames are selected based on scene feature changes, so that a playback function is realized. According to the invention, the problems of video data dispersion, poor compatibility, lagging emergency response and the like in the existing power grid monitoring system are solved, and the accuracy and efficiency of emergency disposal are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video surveillance and emergency scene fusion, and specifically relates to a method for fusing real-time surveillance video and emergency scene based on a convolutional neural network. Background Art

[0002] In today's power grid system, video surveillance technology plays a vital role. However, existing video surveillance systems often have many problems. For example, the existing substation videos, power transmission monitoring, individual equipment and other video resources of various professions are scattered in different systems, making it difficult to achieve centralized sharing and unified management of resources. This often leads to "invisible and unclear" situations during emergency response, and it is impossible to obtain accurate on-site information in a timely manner, thus affecting the efficiency and effectiveness of emergency response.

[0003] In order to solve these problems, existing technologies attempt to integrate different video systems, but due to the poor compatibility and interoperability between the systems, the integration effect is not ideal. In addition, the existing video platform has problems such as imperfect functions and low efficiency in terms of device access, resource management, video storage and intelligent analysis, and cannot meet the high requirements of the power grid system for video monitoring.

[0004] Therefore, there is an urgent need for an innovative technical solution that can give full play to the advantages of a unified video platform in all aspects, fully integrate and access video resources from various professions, realize centralized sharing of resources, improve the ability and efficiency of emergency response, and solve the problem of "invisible and unclear" in previous emergency response processes. Summary of the invention

[0005] In view of the above-mentioned technical problems, the present invention proposes a method for fusing real-time monitoring video and emergency scenes based on convolutional neural network.

[0006] In order to achieve the above object, the technical solution adopted by the present invention comprises the following steps:

[0007] S1. First, we take advantage of the unified video platform’s equipment access advantages to fully integrate and access existing substation videos, power transmission monitoring, and individual equipment video resources of various professions, and perform unified format conversion and encoding of video resources from different sources to ensure the compatibility and processability of video data;

[0008] S2, secondly, the integrated video data is subjected to image enhancement operation by using adaptive histogram equalization to improve the video quality;

[0009] S3. Then, an emergency scenario database is established, which includes emergency situations and their characteristic descriptions. When an abnormal situation occurs in the real-time monitoring video, the abnormal type is determined through convolutional neural network analysis and matched with the emergency scenario database. Based on the matching results, the real-time monitoring video is fused with the corresponding emergency scenario to provide more intuitive information for emergency handling.

[0010] S4. Finally, the fused video resources are centrally stored in the video storage system of the unified video platform, and the playback function of the video resources is realized by selecting key frames based on changes in scene features.

[0011] The image enhancement operation performed by using adaptive histogram equalization in step S2 is:

[0012] S21, first of all, image segmentation is to divide the input image into several local regions. Regular grid segmentation can be adopted, or adaptive segmentation can be performed according to the characteristics of the image. The total size of the image is M×N, and the number of local regions divided is k. Then the size of each local region is Where m and n are determined according to the number of local regions and the image size;

[0013] S22. Next, for each local area, calculate its grayscale histogram. The grayscale value range of the pixels in the local area is [0, L-1], where L is 256 representing the grayscale level. The local histogram H is expressed as H = {h0, h1, .., h L-1}, where h i Indicates the number of pixels with gray value i;

[0014] S23, then perform equalization processing according to the local histogram, calculate the local cumulative distribution function cdf0 = h0, cdf i =h i +cdf i-1 , i=1,2,...,L-1, then gray value mapping is performed. For each pixel in the local area, its original gray value is r, and the mapped gray value is s. The calculation formula is Where MN is the total number of pixels in the local area;

[0015] S24, next, when processing adjacent local areas, in order to avoid obvious boundary effects, a weighted average method is used to process boundary pixels;

[0016] S25. Finally, each local area processed by the local histogram equalization is fused to obtain the final enhanced image.

[0017] Preferably, the specific implementation of the unified format conversion and encoding of video data resources from different sources in step S1 is as follows:

[0018] S11, firstly, preliminarily parse the incoming video, identify different video formats, classify the videos according to different source devices, and mark the resolution, frame rate, and encoding format features of the videos;

[0019] S12, then use the format conversion algorithm to convert the video streams of different formats into the same H.264 encoding format, and the implementation formula is V out (t) = f(V in (t),F t ), where V out (t) is the converted video stream, V in (t) is the original input video stream, F t Conversion parameters indicating the target format;

[0020] S13, then perform video encoding and compression, perform unified encoding processing on the format-converted video, and realize video compression through a motion compensation-based encoding algorithm. out (t) = C(V out (t),P), where C out (t) is the compressed video stream, P is the encoding parameter, and C is the encoding function;

[0021] S14. Finally, the video stream that has been format-converted and encoded is stored or transmitted to ensure its compatibility and processability. Video indexing technology is used to generate a unique index tag for each video stream to facilitate subsequent retrieval and analysis.

[0022] Preferably, in step S3, the abnormal situation analysis by convolutional neural network CNN, the matching of emergency scene database and the video fusion are carried out, and the specific implementation steps can be described as follows:

[0023] S31. First, build a database covering a variety of emergency scenarios, including fire, equipment failure, flood, and personal injury emergency situations, and each emergency scenario needs to have a corresponding feature description;

[0024] S32, secondly, for each emergency scene, extract its visual features, and vectorize these features so as to match them with abnormal features in the real-time video;

[0025] S33, followed by abnormal situation detection and classification, when the real-time monitoring video stream enters the system, after preprocessing, it is input into the convolutional neural network for abnormality detection. The convolutional neural network model extracts the features of the video frame and generates a feature map F through multi-layer convolution operations. video =CNN(I video ), where F video is the feature vector extracted from the video frame, I videoThe input image of the video frame is used. The features output by the convolutional layer are used to detect abnormalities in the video. This process further filters out obvious abnormal features through the anomaly detection network, and uses the anomaly classification function to determine the specific anomaly type. Among them C abnormal is the detected anomaly category, P class (cF video ) is the predicted probability of category c;

[0026] S34, then the detected abnormal feature F video and the feature F in the emergency scene database scene Match them and find the closest emergency scenario by calculating the similarity between the two. The similarity calculation can use the cosine similarity formula Where sim(F video ,F scene ) represents the similarity value between the video feature and the scene feature. The higher the value, the closer it is. In addition, by maximizing the similarity metric value, the most matching emergency scene S is selected. match , the formula is Where S represents the set of all emergency scenarios in the database;

[0027] S35. Finally, based on the matching emergency scene, the real-time video is fused with the corresponding emergency scene. The fusion process is V fusion (t) = α·I video (t)+(1-α)·I scene (t), where V fusion (t) is the fused video frame, I video (t) is the real-time monitoring video, I scene (t) is the emergency scene video image, and α is the weight coefficient used to adjust the fusion ratio of the two videos.

[0028] Preferably, in step S4, the video storage system automatically adjusts the distribution of video data in the storage nodes according to the access frequency and importance factors of the video. For frequently accessed videos, they are stored on storage nodes with higher performance to increase the access speed; for infrequently accessed videos, they are stored on storage nodes with lower costs to reduce storage costs.

[0029] Preferably, the key frame selection step based on scene feature changes in step S4 is as follows:

[0030] S41. First, define the metric standard for scene feature changes, and use the color histogram of the image for measurement. The scene feature change measurement formula is: Among them, H t (i, j) is the color histogram of the video frame at time t, D scene(t) represents the scene change measure at time t. If the change is large, it can be regarded as a scene switch and the frame is selected as the key frame;

[0031] S42, secondly, a hierarchical indexing mechanism is used to divide the video into different time periods, the first-level index records key frames with a longer time interval of 10 seconds, and the second-level index records key frames with a shorter time interval of 1 second;

[0032] S43, then the key frame and the common frame are stored. For the common frame, a differential encoding method is adopted to store only the change information relative to the previous key frame, and the key frame needs to be stored completely;

[0033] S44. Finally, the key frame index is managed using a B+ tree structure. The nodes of the B+ tree contain the timestamp information of the key frames, and the leaf nodes point to the specific key frame data addresses. When the video data stream is input in real time, the key frame index is dynamically updated according to the set scene change standard. Whenever a new key frame is detected, it is inserted into the B+ tree to ensure the real-time and consistency of the index.

[0034] Compared with the prior art, the advantages and positive effects of the present invention are that the present invention provides a method for fusing real-time monitoring video and emergency scenes based on convolutional neural networks, which mainly solves the problems of multi-source video data dispersion, poor compatibility, and delayed emergency response in the existing power grid monitoring system. Through a unified video platform, format conversion and encoding of videos from different sources are achieved, which enhances the compatibility and processability of the video. Adaptive histogram equalization technology is used to improve video clarity, and convolutional neural networks are used to intelligently detect abnormal events, combined with emergency scene databases for matching and fusion, which significantly improves the accuracy and efficiency of emergency response. The key frame selection mechanism further enhances the playback speed and processing efficiency of the video, making the handling of emergency events more timely and effective. DETAILED DESCRIPTION

[0035] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0037] Embodiment, the present invention provides a method for fusing real-time monitoring video and emergency scene based on convolutional neural network, which aims at the multi-source video data fusion problem in the power grid monitoring system, and is particularly suitable for real-time detection and response of emergency scene. It integrates various monitoring videos through the device access capability of the unified video platform, uses convolutional neural network to perform video anomaly detection, and combines the real-time monitoring video with a pre-established emergency scene database to ensure that the fused video information can be provided quickly and accurately during the emergency response process.

[0038] First, we take advantage of the device access of the unified video platform to fully integrate and access various video resources, including substation video monitoring, transmission line monitoring, and individual equipment video. Since these video sources are different, the formats, resolutions, and encoding standards may be different, so the following processing is required: convert videos from different sources to the same encoding format H.264. In this embodiment, by parsing the video stream, identifying its resolution, frame rate, encoding method and other characteristics, and using the format conversion algorithm to uniformly format the video, ensure that video data in different formats can be compatible with the processing V out (t) = f(V in (t),F t ), where V out (t) is the converted video stream, V in (t) is the original input video stream, F t Indicates the conversion parameters of the target format. In addition, the formatted video needs to be encoded and compressed, and a motion compensation-based encoding algorithm is used to further reduce the bandwidth occupied by storage and transmission. The compressed video is represented by the following formula C out (t) = C(V out (t),P), where C out (t) is the compressed video stream, P is the encoding parameter, and C is the encoding function. Through this processing step, a unified format of videos from different sources is achieved, ensuring the compatibility and processability of the data.

[0039] Considering the poor image quality of the integrated video data, in order to improve the visualization effect of the video, this embodiment uses adaptive histogram equalization to enhance the video image. The specific steps are: first, image segmentation, dividing the input image into several local areas. A regular grid segmentation method can be used, or adaptive segmentation can be performed based on the characteristics of the image. The total size of the image is M×N, and the number of local areas divided is k. The size of each local area is Where m and n are determined according to the number of local regions and the image size; secondly, for each local region, its grayscale histogram is calculated. The grayscale value range of the pixels in the local region is [0, L-1], where L is 256 representing the grayscale level. The local histogram H is expressed as H = {h0,h1,..,h L-1}, where h i represents the number of pixels with gray value i; then the local histogram is used for equalization, and the local cumulative distribution function cdf0=h0 is calculated, cdf i =h i +cdf i-1 , i=1,2,...,L-1, then gray value mapping is performed. For each pixel in the local area, its original gray value is r, and the mapped gray value is s. The calculation formula is Where MN is the total number of pixels in the local area; next, when processing adjacent local areas, in order to avoid obvious boundary effects, the weighted average method is used to process the boundary pixels; finally, the local areas after local histogram equalization are fused to obtain the final enhanced image. This processing step effectively improves the clarity and visual quality of the video, especially in emergency scenarios, it can effectively enhance the visual performance of abnormal videos, making subsequent anomaly detection more accurate.

[0040] Then, in order to respond quickly in emergency scenarios, the system needs to automatically identify and process abnormal situations in real-time surveillance videos. To this end, this embodiment constructs an emergency scenario database and combines convolutional neural networks to detect and match abnormal situations in videos. The database contains various emergency situations and their feature descriptions, covering a variety of common emergency scenarios such as fires, equipment failures, floods, and personal injuries. Each scene has pre-extracted corresponding visual features, such as color, shape, texture, etc., and stored as feature vectors for subsequent matching. When the system receives a real-time video stream, it first pre-processes the video frames and then inputs them into the convolutional neural network model. The convolutional neural network extracts features from the video frames and generates a feature map F. video =CNN(I video ), where F video is the feature vector extracted from the video frame, I video The input image of the video frame is used. The features output by the convolutional layer are used to detect abnormalities in the video. This process further filters out obvious abnormal features through the anomaly detection network, and uses the anomaly classification function to determine the specific anomaly type. Among them C abnormal is the detected anomaly category, P class (cF video ) is the predicted probability of category c; then the detected abnormal feature F videoand the feature F in the emergency scene database scene Match them and find the closest emergency scenario by calculating the similarity between the two. The similarity calculation can use the cosine similarity formula Where sim(F video ,F scene ) represents the similarity value between the video feature and the scene feature. The higher the value, the closer it is. In addition, by maximizing the similarity metric value, the most matching emergency scene S is selected. match , the formula is Where S represents the set of all emergency scenes in the database; finally, based on the matching emergency scenes, the real-time video is fused with the corresponding emergency scenes. The fusion process is V fusion (t) = α·I video (t)+(1-α)·I scene (t), where V fusion (t) is the fused video frame, I video (t) is the real-time monitoring video, I scene (t) is the emergency scene video image, and α is the weight coefficient used to adjust the fusion ratio of the two videos. Through the above steps, abnormal events in the surveillance video can be detected quickly and accurately, and they can be fused and displayed with the corresponding emergency scene, providing more intuitive and comprehensive information support for emergency handling.

[0041] Finally, in order to achieve efficient video storage and playback, especially for key video frames in emergency scenes, the present invention adopts a key frame selection mechanism based on scene feature changes. First, the metric standard for scene feature changes is defined, and the color histogram of the image is used for measurement. The scene feature change measurement formula is: Among them, H t (i, j) is the color histogram of the video frame at time t, D scene(t) represents the scene change measurement at time t. If the change is large, it can be regarded as a scene switch, and the frame is selected as a key frame. Secondly, a hierarchical indexing mechanism is used to divide the video into different time periods. The first layer index records the key frames with a longer time interval of 10 seconds, and the second layer index records the key frames with a shorter time interval of 1 second. Then the key frames and ordinary frames are stored. For ordinary frames, differential encoding is used to store only the change information relative to the previous key frame, and the key frames need to be stored completely. Finally, the B+ tree structure is used to manage the key frame index. The nodes of the B+ tree contain the timestamp information of the key frame, and the leaf nodes point to the specific key frame data address. When the video data stream is input in real time, the key frame index is dynamically updated according to the set scene change standard. Whenever a new key frame is detected, it is inserted into the B+ tree to ensure the real-time and consistency of the index. Through the key frame indexing mechanism, the system can quickly play back important surveillance video content, especially after an emergency event occurs, it can quickly locate the key scene and provide a reliable basis for emergency decision-making.

[0042] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for integrating real-time monitoring video and emergency scene based on convolutional neural network, characterized in that: The following steps are involved: S1. First, we take advantage of the unified video platform’s equipment access advantages to fully integrate and access existing substation videos, power transmission monitoring, and individual equipment video resources of various professions, and perform unified format conversion and encoding of video resources from different sources to ensure the compatibility and processability of video data; S2, secondly, the integrated video data is subjected to image enhancement operation by using adaptive histogram equalization to improve the video quality; S3. Then, an emergency scenario database is established, which includes emergency situations and their characteristic descriptions. When an abnormal situation occurs in the real-time monitoring video, the abnormal type is determined through convolutional neural network analysis and matched with the emergency scenario database. Based on the matching results, the real-time monitoring video is fused with the corresponding emergency scenario to provide more intuitive information for emergency handling. S4. Finally, the fused video resources are centrally stored in the video storage system of the unified video platform, and the playback function of the video resources is realized by selecting key frames based on changes in scene features; The image enhancement operation performed by using adaptive histogram equalization in step S2 is: S21, first of all, image segmentation is to divide the input image into several local regions. Regular grid segmentation can be adopted, or adaptive segmentation can be performed according to the characteristics of the image. The total size of the image is M×N, and the number of local regions divided is k. Then the size of each local region is Where m and n are determined according to the number of local regions and the image size; S22. Next, for each local area, calculate its grayscale histogram. The grayscale value range of the pixels in the local area is [0, L-1], where L is 256 representing the grayscale level. The local histogram H is expressed as H = {h0, h1, .., h L-1 }, where h i Represents the number of pixels with gray value i; S23, then perform equalization processing according to the local histogram, calculate the local cumulative distribution function cdf0 = h0, cdf i =h i +cdf i-1 , i=1,2,...,L-1, then gray value mapping is performed. For each pixel in the local area, its original gray value is r, and the mapped gray value is s. The calculation formula is Where MN is the total number of pixels in the local area; S24, next, when processing adjacent local areas, in order to avoid obvious boundary effects, a weighted average method is used to process boundary pixels; S25. Finally, each local area processed by the local histogram equalization is fused to obtain the final enhanced image.

2. According to claim 1, a method for integrating real-time monitoring video and emergency scene based on convolutional neural network is characterized in that: The specific implementation of the unified format conversion and encoding of video data resources from different sources in step S1 is as follows: S11, firstly, preliminarily parse the incoming video, identify different video formats, classify the videos according to different source devices, and mark the resolution, frame rate, and encoding format features of the videos; S12, then use the format conversion algorithm to convert the video streams of different formats into the same H.264 encoding format, and the implementation formula is V out (t) = f(V in (t),F t ), where V out (t) is the converted video stream, V in (t) is the original input video stream, F t Conversion parameters indicating the target format; S13, then perform video encoding and compression, perform unified encoding processing on the format-converted video, and realize video compression through a motion compensation-based encoding algorithm. out (t) = C(V out (t),P), where C out (t) is the compressed video stream, P is the encoding parameter, and C is the encoding function; S14. Finally, the video stream that has been format-converted and encoded is stored or transmitted to ensure its compatibility and processability. Video indexing technology is used to generate a unique index tag for each video stream to facilitate subsequent retrieval and analysis.

3. According to claim 1, a method for integrating real-time monitoring video and emergency scene based on convolutional neural network is characterized in that: In step S3, the abnormal situation analysis by the convolutional neural network CNN, the matching of the emergency scene database and the video fusion are specifically implemented as follows: S31. First, build a database covering a variety of emergency scenarios, including fire, equipment failure, flood, and personal injury emergency situations, and each emergency scenario needs to have a corresponding feature description; S32, secondly, for each emergency scene, extract its visual features, and vectorize these features so as to match them with abnormal features in the real-time video; S33, followed by abnormal situation detection and classification, when the real-time monitoring video stream enters the system, after preprocessing, it is input into the convolutional neural network for abnormality detection. The convolutional neural network model extracts the features of the video frame and generates a feature map F through multi-layer convolution operations. video =CNN(I video ), where F video is the feature vector extracted from the video frame, I video The input image of the video frame is used. The features output by the convolutional layer are used to detect abnormalities in the video. This process further filters out obvious abnormal features through the anomaly detection network, and uses the anomaly classification function to determine the specific anomaly type. Among them C abnormal is the detected anomaly category, P class (cF video ) is the predicted probability of category c; S34, then the detected abnormal feature F video and the feature F in the emergency scene database scene Match them and find the closest emergency scenario by calculating the similarity between the two. The similarity calculation can use the cosine similarity formula Where sim(F video ,F scene ) represents the similarity value between the video feature and the scene feature. The higher the value, the closer it is. In addition, by maximizing the similarity metric value, the most matching emergency scene S is selected. match , the formula is Where S represents the set of all emergency scenarios in the database; S35. Finally, based on the matching emergency scene, the real-time video is fused with the corresponding emergency scene. The fusion process is V fusion (t) = α·I video (t)+(1-α)·I scene (t), where V fusion (t) is the fused video frame, I video (t) is the real-time monitoring video, I scene (t) is the emergency scene video image, and α is the weight coefficient used to adjust the fusion ratio of the two videos.

4. According to claim 1, a method for integrating real-time monitoring video and emergency scene based on convolutional neural network is characterized in that: In step S4, the video storage system automatically adjusts the distribution of video data in the storage nodes according to the access frequency and importance factors of the video. For frequently accessed videos, they are stored on storage nodes with higher performance to improve the access speed; for infrequently accessed videos, they are stored on storage nodes with lower costs to reduce storage costs.

5. According to claim 1, a method for integrating real-time monitoring video and emergency scene based on convolutional neural network is characterized in that: The key frame selection steps based on scene feature changes in step S4 are as follows: S41. First, define the metric standard for scene feature changes, and use the color histogram of the image for measurement. The scene feature change measurement formula is: Among them, H t (i, j) is the color histogram of the video frame at time t, D scene (t) represents the scene change measure at time t. If the change is large, it can be regarded as a scene switch and the frame is selected as the key frame; S42, secondly, a hierarchical indexing mechanism is used to divide the video into different time periods, the first-level index records key frames with a longer time interval of 10 seconds, and the second-level index records key frames with a shorter time interval of 1 second; S43, then the key frame and the common frame are stored. For the common frame, a differential encoding method is adopted to store only the change information relative to the previous key frame, and the key frame needs to be stored completely; S44. Finally, the key frame index is managed using a B+ tree structure. The nodes of the B+ tree contain the timestamp information of the key frames, and the leaf nodes point to the specific key frame data addresses. When the video data stream is input in real time, the key frame index is dynamically updated according to the set scene change standard. Whenever a new key frame is detected, it is inserted into the B+ tree to ensure the real-time and consistency of the index.

Images