Deep learning-based split-screen monitoring abnormal event highlighting method

Abstract

The invention provides a split screen monitoring abnormal event highlighting method based on deep learning, which monitors the abnormal event of each split screen video in real time through a deep learning model and highlights the split screen, and comprises the following steps: extracting a single frame image from a monitoring video stream in real time by using FFmpeg, and obtaining the position coordinate of each split screen; detecting a video stream in real time through a YOLOv5 model to obtain an abnormal event area, and analyzing the event area by using a ConvLSTM model to predict an abnormal event; when an abnormal event is detected, mapping the position of an abnormal event bounding box detected by YOLOv5 with the split screen position to position the split screen where the abnormal event is located; and highlighting the split screen in different modes according to the type of the abnormal event. The method has the advantages that the abnormal event is automatically detected, the split screen where the abnormal event is located is highlighted, the burden of manual monitoring is reduced, and the reaction efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video surveillance, and in particular relates to a method for highlighting abnormal events in split-screen surveillance based on deep learning. Background Art

[0002] In large-scale factory monitoring systems, since the production activities of the factory involve a large number of equipment operations, personnel flow, raw material processing and complex automation processes, the monitoring system covers a wide range and requires real-time analysis of a large amount of video data. Usually, there are multiple monitoring split screens to display real-time images of different areas, and abnormal events (such as noise, explosion, fire, smoke, etc.) of each split screen are manually analyzed. Due to the large amount of monitoring content in these areas, manual identification of potential abnormalities will face huge challenges, such as information overload: monitoring personnel need to pay attention to multiple split screens at the same time and process massive video information, which may easily lead to ignoring certain potential abnormalities; slow response: when an emergency event (such as fire or explosion) occurs, the response speed of manual monitoring is slow, which may delay emergency handling; identification difficulty: there are many types of abnormal events in the factory environment, and manual identification has limitations, especially for abnormalities that are uncommon or difficult to directly identify visually (such as heat source changes, smoke, etc.). Summary of the invention

[0003] In view of this, the present invention aims to propose a split-screen monitoring abnormal event highlighting method based on deep learning, so as to utilize deep learning technology to automatically detect abnormal events and highlight the split screen where they are located, thereby reducing the burden of manual monitoring and improving response efficiency.

[0004] To achieve the above object, the technical solution of the present invention is achieved as follows: Abnormal event highlighting method in split-screen monitoring based on deep learning.

[0005] Furthermore, the deep learning model is used to monitor abnormal events of each split-screen video in real time and highlight the split-screen, including: S1. Real-time video stream processing: Use FFmpeg to extract single-frame images from the surveillance video stream in real time and obtain the position coordinates of each split screen; S2. Abnormal event detection: The YOLOv5 model is used to detect the video stream in real time to obtain the abnormal event area, and then the ConvLSTM model is used to analyze the event area to predict abnormal events; S3, split screen positioning: When an abnormal event is detected, the position of the abnormal event bounding box detected by YOLOv5 is mapped to the split screen position to locate the split screen where the abnormal event is located; S4. Split-screen highlighting: Based on the type of abnormal event, the split-screen where the abnormal event is located is highlighted in different ways.

[0006] Furthermore, in S1, if the monitoring screen is a fixed split-screen layout, the position coordinates of each split-screen are directly divided using a predefined grid; If the monitoring screen changes dynamically, the image segmentation algorithm is used to automatically detect the position of each split screen.

[0007] Further, in S2, the abnormal event types include explosion events, smoke events, fire events, noise events, and high temperature events; The YOLOv5 model is trained in advance as a multi-category detection model to detect various abnormal event types and locate abnormal event areas in the image, and outputs the relevant abnormal event categories, confidence levels, and bounding box information detection results to the ConvLSTM model. The ConvLSTM model performs time series analysis on them and predicts abnormal events.

[0008] Furthermore, in S3, when YOLOv5 detects an abnormal event, the event is mapped to a corresponding split screen through its bounding box position.

[0009] Furthermore, for the coordinate mapping of a single abnormal event: the split-screen area where the abnormal event is located is determined by calculating the coordinates of the center point of the bounding box of the abnormal event; Coordinate mapping for multiple abnormal events occurring simultaneously: For multiple abnormal events, set priorities according to the event categories, and events with higher priorities will be mapped to more split-screen areas; For the coordinate mapping of multiple abnormal events on the same split screen: use transparency changes, borders of different colors or icons to distinguish different types of events.

[0010] Furthermore, in said S4, the split screen highlighting method includes: different color boxes represent different categories of abnormal events, and warning text, flashing effects or icons are added in the split screen.

[0011] Furthermore, an electronic device includes a processor and a memory that is communicatively connected to the processor and is used to store executable instructions of the processor, and the processor is used to execute the deep learning-based split-screen monitoring abnormal event highlighting method.

[0012] Furthermore, a server includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the deep learning-based split-screen monitoring abnormal event highlighting method.

[0013] Furthermore, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for highlighting abnormal events in split-screen monitoring based on deep learning is implemented.

[0014] Compared with the prior art, the method for highlighting abnormal events in split-screen monitoring based on deep learning described in the present invention has the following beneficial effects: (1) The deep learning-based split-screen monitoring abnormal event highlighting method described in the present invention can significantly improve the automation and intelligence level of the monitoring system through real-time abnormal event detection and prediction based on YOLOv5 and ConvLSTM; the YOLOv5 model can quickly and accurately detect various abnormal events in the video stream, and use the ConvLSTM model to perform time series analysis and prediction of events, helping monitoring personnel to predict the subsequent development of the event.

[0015] (2) The method for highlighting abnormal events in split-screen monitoring based on deep learning described in the present invention utilizes split-screen mapping technology and different highlighting methods (such as color coding, flashing effects, icon display, etc.). The system can accurately associate abnormal events with specific split-screen areas and highlight them.

[0016] (3) The deep learning-based split-screen monitoring abnormal event highlighting method described in the present invention can flexibly adapt to different monitoring environments for fixed or dynamically changing split-screen layouts. Under a fixed layout, the predefined split-screen coordinates can efficiently process the image of each frame; under a dynamic layout, the system can detect and update the split-screen area in real time through an image segmentation algorithm, ensuring the stability and reliability of the system in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 The figure is a schematic diagram of the abnormal event highlighting process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0019] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0020] This embodiment provides a method for highlighting abnormal events in split-screen monitoring based on deep learning, which monitors abnormal events of each split-screen video in real time through a deep learning model and highlights the split screen, such as Figure 1 As shown, including: S1. Real-time video stream processing: Use FFmpeg to extract single-frame images from the surveillance video stream in real time and obtain the position coordinates of each split screen; S2. Abnormal event detection: The YOLOv5 model is used to detect the video stream in real time to obtain the abnormal event area, and then the ConvLSTM model is used to analyze the event area to predict abnormal events; S3, split screen positioning: When an abnormal event is detected, the position of the abnormal event bounding box detected by YOLOv5 is mapped to the split screen position to locate the split screen where the abnormal event is located; S4. Split-screen highlighting: Based on the type of abnormal event, the split-screen where the abnormal event is located is highlighted in different ways.

[0021] Specifically, in S1, if the monitoring screen is a fixed split-screen layout, the position coordinates of each split screen are directly divided using a predefined grid. Assuming that the resolution of a video frame is (W, H), and the split-screen layout is 3x3, the width of each split screen is W / 3 and the height is H / 3. Each split-screen area is directly divided according to this rule. If the monitoring screen changes dynamically, the position of each split screen is automatically detected using an image segmentation algorithm. For example, a semantic segmentation network based on U-Net automatically detects each split-screen area in the image. This method can automatically identify the position coordinates of each split screen according to image content and layout changes. The pixel coordinates of each area obtained by segmentation can further provide accurate split-screen coordinates for subsequent abnormal event detection.

[0022] Specifically, in S2, the abnormal event types include explosion events, smoke events, fire events, noise events, and high temperature events. Relying solely on video surveillance may have certain limitations, and it is necessary to add sensors for joint monitoring. Noise events can detect abnormal noise through acoustic sensors; high temperature events can monitor temperature changes in real time through temperature sensors or infrared sensors; smoke events can be combined with smoke detectors for early identification; fire events can be more accurately monitored through a combination of flame, smoke and temperature sensors; and explosion events can use pressure sensors or vibration sensors to detect abnormal vibrations or shock waves in advance. These sensors can complement the video surveillance system to improve the system's detection accuracy and response speed to various potential abnormal events.

[0023] Specifically, the YOLOv5 model is pre-trained as a multi-category detection model to detect various abnormal event types and locate abnormal event areas in the image, and output the relevant abnormal event categories, confidence levels and bounding box information detection results to the ConvLSTM model. The ConvLSTM model performs time series analysis and predicts abnormal events. For example, if the event is an explosion, the ConvLSTM can predict the diffusion trend of the explosion; if the event is smoke, the model can predict whether the smoke will diffuse or disappear.

[0024] Specifically, in S3, when YOLOv5 detects an abnormal event, the event is mapped to the corresponding split screen through its bounding box position. First, the center coordinates of the bounding box are obtained: ,Then, the center coordinates are compared with the coordinate range of the split-screen area to determine whether the event is located in a specific split-screen area.

[0025] For the coordinate mapping of a single abnormal event: determine the split-screen area where the abnormal event is located by calculating the center point coordinates of the bounding box of the abnormal event. If the center point coordinates ( , ) falls within the coordinate range of a split-screen area, the event belongs to that split-screen.

[0026] Coordinate mapping for multiple abnormal events occurring simultaneously: For multiple abnormal events, priorities are set according to the categories of the events. Events with higher priorities will be mapped to more split-screen areas, or occupy more display areas.

[0027] For the coordinate mapping of multiple abnormal events on the same split screen: use transparency changes, borders of different colors or icons to distinguish different types of events.

[0028] Specifically, in S4, the split screen highlighting method includes: different color boxes represent different types of abnormal events, and warning text, flashing effects or icons are added in the split screen.

[0029] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0030] In the several embodiments provided in the present application, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the division of the units described above is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The above-mentioned units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0031] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

[0032] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for highlighting abnormal events in split-screen monitoring based on deep learning, characterized by: The deep learning model is used to monitor abnormal events in each split-screen video in real time and highlight the split-screen, including: S1. Real-time video stream processing: Use FFmpeg to extract single-frame images from the surveillance video stream in real time and obtain the position coordinates of each split screen; S2. Abnormal event detection: The YOLOv5 model is used to detect the video stream in real time to obtain the abnormal event area, and then the ConvLSTM model is used to analyze the event area to predict abnormal events; S3, split screen positioning: When an abnormal event is detected, the position of the abnormal event bounding box detected by YOLOv5 is mapped to the split screen position to locate the split screen where the abnormal event is located; S4. Split-screen highlighting: Based on the type of abnormal event, the split-screen where the abnormal event is located is highlighted in different ways.

2. The method for highlighting abnormal events in split-screen monitoring based on deep learning according to claim 1 is characterized in that: In S1, if the monitoring screen is a fixed split-screen layout, the position coordinates of each split-screen are directly divided using a predefined grid; If the monitoring screen changes dynamically, the image segmentation algorithm is used to automatically detect the position of each split screen.

3. The method for highlighting abnormal events in split-screen monitoring based on deep learning according to claim 1 is characterized in that: In S2, the abnormal event types include explosion events, smoke events, fire events, noise events, and high temperature events; The YOLOv5 model is trained in advance as a multi-category detection model to detect various abnormal event types and locate abnormal event areas in the image, and outputs the relevant abnormal event categories, confidence levels, and bounding box information detection results to the ConvLSTM model. The ConvLSTM model performs time series analysis on them and predicts abnormal events.

4. The method for highlighting abnormal events in split-screen monitoring based on deep learning according to claim 1 is characterized in that: In S3, when YOLOv5 detects an abnormal event, the event is mapped to the corresponding split screen through its bounding box position.

5. The method for highlighting abnormal events in split-screen monitoring based on deep learning according to claim 4 is characterized in that: For the coordinate mapping of a single abnormal event: the split-screen area where the abnormal event is located is determined by calculating the coordinates of the center point of the bounding box of the abnormal event; Coordinate mapping for multiple abnormal events occurring simultaneously: For multiple abnormal events, set priorities according to the event categories, and events with higher priorities will be mapped to more split-screen areas; For the coordinate mapping of multiple abnormal events in the same split screen: use transparency changes, borders of different colors or icons to distinguish different types of events.

6. The method for highlighting abnormal events in split-screen monitoring based on deep learning according to claim 1 is characterized in that: In said S4, the split screen highlighting method includes: different color boxes represent different types of abnormal events, and warning text, flashing effects or icons are added in the split screen.

7. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute the deep learning-based split-screen monitoring abnormal event highlighting method described in any one of claims 1-6 above.

8. A server, characterized in that: It includes at least one processor and a memory communicatively connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor so that the at least one processor executes the deep learning-based split-screen monitoring abnormal event highlighting method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for highlighting abnormal events in split-screen monitoring based on deep learning as described in any one of claims 1 to 6 is implemented.

Images