Monitoring video AI analysis system based on computer vision and deep learning technology

Abstract

The invention relates to the technical field of video monitoring, in particular to a monitoring video AI (Artificial Intelligence) analysis system based on computer vision and deep learning technology, which comprises a real-time monitoring module for detecting a target object appearing in a monitoring video and identifying and analyzing human actions; wherein the video compression storage unit is used for extracting and storing monitoring videos with human actions; and the rule judgment module is used for calculating the scene abnormity degree by combining the scene object and the human action in the monitoring video so as to judge whether the abnormal human action exists in the monitoring video or not. The query module is used for retrieving historical monitoring videos similar to the monitoring videos with the abnormal human actions, so that the historical video retrieval efficiency is improved; the early warning module carries out early warning according to the abnormal human action, and the abnormal human action tracking module obtains an area where the abnormal human action is located and calls a camera in the area to carry out continuous monitoring. According to the invention, through the above modules, the analysis and processing efficiency of human actions in the monitoring video is improved.

Description

Technical Field

[0001] The present invention relates to the field of video surveillance technology, and in particular to a surveillance video AI analysis system based on computer vision and deep learning technology. Background Art

[0002] Computer vision aims to enable computers to understand, analyze and interpret information obtained from images or videos. It imitates the human visual system and uses mathematical and statistical methods to convert pixels in images into understandable information, thereby achieving functions such as recognition, detection, and classification. Deep learning is a subfield of machine learning that uses artificial neural networks to simulate the way the human brain learns and is particularly good at extracting useful features from large amounts of data.

[0003] With the rapid development of video surveillance in recent years, the number of cameras has increased rapidly, and a large amount of video data has brought challenges to real-time monitoring alarms and the effective use of video data. A standard video surveillance system often has dozens or even thousands of videos and corresponding digital video data. It becomes very difficult for operators to detect alarm events by observing each video. In addition, when analyzing alarm events afterwards, operators need to find relevant videos in a timely manner; however, traditional video surveillance lacks intelligent analysis of videos, and video data cannot be effectively retrieved. It can only be manually searched based on a rough time period, resulting in a large amount of working time consumed in video data analysis. Statistics show that operators will miss 90% of video information after staring at the screen video wall for more than 10 minutes.

[0004] Through computer vision and deep learning technology, the system can automatically analyze key events in surveillance videos, reduce manpower consumption, and free security operators from the complicated and boring task of "staring at the screen". However, in the current surveillance video analysis process, as time accumulates, there are a large number of surveillance videos to be processed. The recognition, analysis and prediction of human actions in surveillance videos, as well as the retrieval and processing of historical video frames similar to abnormal human actions are inefficient, and there is still room for research.

[0005] To this end, a surveillance video AI analysis system based on computer vision and deep learning technology is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a surveillance video AI analysis system based on computer vision and deep learning technology, which realizes automatic detection and recognition of human actions in surveillance videos through the convolutional attention model in the human action recognition unit; effectively compresses and stores historical surveillance videos through the video compression storage unit; the early warning module performs real-time early warning according to the category of abnormal human actions and the degree of scene abnormality; the abnormal human action tracking module performs real-time monitoring and tracking of abnormal human actions. Through the above operations and methods, the analysis capability of surveillance videos is enhanced, and the recognition and processing efficiency of abnormal human actions is improved.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A surveillance video AI analysis system based on computer vision and deep learning technology, including:

[0009] The real-time monitoring module includes a target detection unit, a human action recognition unit and a video compression storage unit; wherein the target detection unit obtains the monitoring video and processes it using a target detection algorithm to obtain a detected target;

[0010] The human action recognition unit is used to obtain the human monitoring video frame information output by the target detection unit, and recognize the human action in the human monitoring video frame information according to the convolutional attention model;

[0011] The video compression storage unit is used to obtain and process the surveillance video to obtain historical storage video data, and upload the historical storage video data to the historical surveillance video cloud platform;

[0012] Furthermore, the real-time monitoring module includes: the real-time monitoring module obtains and processes the monitoring video captured by the camera, calculates the pixel difference between the monitoring video frames of the monitoring video, and obtains the total pixel difference; if the total pixel difference exceeds a preset pixel difference threshold, there is human activity in the monitoring video.

[0013] Furthermore, the target detection unit includes: if there is human activity in the surveillance video, the target detection unit obtains and processes the surveillance video frame of the surveillance video through a target detection algorithm to obtain a detected target within a target detection frame, and the detected target includes humans and scene objects; according to the target detection frame, the human image is segmented to obtain human surveillance video frame information.

[0014] Furthermore, the human action recognition unit includes: obtaining and processing the human monitoring video frame information output by the target detection unit to obtain a human action data set; training a convolutional attention model according to the human action data set, the convolutional attention model including a channel attention module and a spatial attention module; the convolutional attention model is used to obtain and process the human monitoring video frame information to obtain human action.

[0015] Furthermore, the video compression storage unit includes: obtaining the surveillance video containing human actions and performing segmentation processing to obtain a surveillance video segment set; calculating the clarity index of all surveillance video frames in the surveillance video segment, wherein the clarity index includes picture quality clarity and target object structure clarity; obtaining the surveillance video frame with the highest clarity index according to the clarity index to obtain a first surveillance video frame;

[0016] Further, all the monitoring video segments are obtained and processed to obtain the first monitoring video frame of each monitoring video segment; all the first monitoring video frames are obtained and processed to obtain storage video data;

[0017] Furthermore, the stored video data includes the first surveillance video frame and a surveillance tag; the surveillance tag includes a camera number, time, scene object and human action category.

[0018] The rule determination module is used to obtain and process the data of the scene object in the scene object database and the human action to obtain the scene abnormality degree. If the scene abnormality degree is greater than a preset scene abnormality degree threshold, there is an abnormal human action.

[0019] Further, the rule determination module includes: acquiring the data of the scene object in the scene object database, wherein the data includes the importance weight of the scene object;

[0020] Furthermore, the geometric centers of the target detection frames of the human action and the scene object are obtained, the distance between the geometric centers of the target detection frame of the human action and the geometric centers of the target detection frame of the scene object are calculated, the distance and the importance weight are obtained and processed to obtain a first scene abnormality degree; the overlapping area between the target detection frame of the human action and the target detection frame of the scene object is obtained and processed to obtain a second scene abnormality degree; the first scene abnormality degree and the second scene abnormality degree are obtained and processed, and according to the abnormality degree judgment function, it is judged whether there is abnormal human action in the surveillance video.

[0021] The query module obtains the surveillance video with the abnormal human action and the data in the historical surveillance video cloud platform and calculates the scene similarity, and obtains the abnormal human action video set according to a preset scene similarity threshold.

[0022] Furthermore, the query module includes: a first query unit obtains and processes the camera number to obtain all historical stored video data under the camera number; obtains and processes the stored video data containing abnormal human actions and the monitoring tags of all the historical stored video data to obtain a first scene similarity, and if the first scene similarity exceeds a preset first scene similarity threshold, the historical stored video data is saved to a first data set; a second query unit obtains the stored video data containing the abnormal human action and the historical stored video data in the first data set, calculates the second scene similarity according to the difference in clarity index, and if the second scene similarity is less than the preset scene similarity threshold, the historical stored video data is uploaded to the abnormal human action video set.

[0023] The early warning module is used to issue an early warning according to the abnormal human action.

[0024] Further, the early warning module includes: obtaining the abnormal human action video set, and sending the abnormal human action video set to security personnel;

[0025] Further, the scene abnormality and abnormality judgment function are obtained and processed, and an early warning is issued according to the abnormality and human action category;

[0026] Furthermore, the warning also includes the location of the abnormal human action, and is prompted in the real-time monitoring picture containing the abnormal human action.

[0027] The abnormal human motion tracking module is used to obtain the area where the abnormal human motion is located and call the camera in the area for continuous monitoring.

[0028] Further, the abnormal human motion tracking module includes: obtaining the position of the abnormal human motion, and calculating the influence range of the abnormal human motion according to the position and the scene object, obtaining the surveillance camera within the influence range and performing target tracking;

[0029] Further, the target tracking includes using a target tracking algorithm to continuously track the area where the abnormal human action is located, and updating the location of the abnormal human action in real time;

[0030] Furthermore, the areas of the target detection frames of the abnormal human action in all cameras are calculated and sorted, and the real-time monitoring screen with the largest area is presented on the main control interface.

[0031] Furthermore, the target tracking also includes: acquiring and processing the abnormal human action to obtain the motion trajectory of the abnormal human action and issuing a trajectory warning; acquiring the data of the scene object in the scene object database, the data including the importance weight of the scene object; monitoring the scene object whose importance weight exceeds a preset importance threshold in the real-time monitoring video where the abnormal human action exists.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. First, by calculating the pixel difference between two consecutive frames, the target detection unit is activated only when a moving object is detected, which optimizes resource mobilization, reduces computing load, and improves the system's operating efficiency. The target detection unit can enable a variety of intelligent algorithms, and after deployment, the target detection algorithm can be flexibly adjusted according to user needs. The human action recognition unit processes human actions in surveillance video frames through a convolutional attention model to improve the accuracy of human action recognition. The video compression storage unit calculates the clarity index to ensure that only high-quality surveillance video frames are stored, saving storage space while ensuring the validity and integrity of the surveillance data. Through the above operations, the real-time monitoring module improves the monitoring efficiency and the intelligence level of the monitoring system.

[0034] 2. The first scene abnormality takes into account the importance of the scene objects and the distance between the scene objects and the detected traces of human activities. The first scene abnormality is obtained through weighted calculation. The distance change is monitored in real time, which can quickly respond to emergencies and identify potential abnormal human actions approaching the scene objects. By calculating the overlapping area of ​​the human target detection frame and the scene object target detection frame, and then combining the time the human action stays, the second scene abnormality is calculated, which can accurately identify the direct threat of human activities to important items and enhance the recognition ability of abnormal human actions. According to the abnormality judgment function, normal activities and abnormal behaviors can be effectively distinguished, the false alarm rate can be reduced, and the accuracy of monitoring can be improved.

[0035] 3. The first query unit can quickly obtain all historical video data under a specific camera, which is convenient for subsequent abnormal behavior detection and analysis and improves query efficiency. By calculating the similarity of the first scene, it can quickly match similar scenes and improve the search efficiency of historical videos. The second query unit calculates the similarity of the second scene and uses the clarity index to filter video frames to ensure that only high-quality monitoring data is focused on, thereby improving the availability and effectiveness of the video. The early warning module can send abnormal human action videos to security personnel in real time to ensure rapid response and improve the efficiency of handling security incidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1A schematic diagram of the structure of a surveillance video AI analysis system based on computer vision and deep learning technology provided in Example 1 of the present invention;

[0037] Figure 2 A schematic diagram of the structure of a real-time monitoring module provided in Example 1 of the present invention;

[0038] Figure 3 A schematic diagram of the structure of the convolutional attention model provided in Example 1 of the present invention;

[0039] Figure 4 A structural diagram of data processing of the channel attention mechanism and the spatial attention mechanism provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Example 1

[0042] In order to improve the overall safety and monitoring efficiency of a chemical plant and build a smart chemical plant, a bee-eye video monitoring AI intelligent service system was built on the basis of a monitoring video AI analysis system based on computer vision and deep learning technology provided by the present invention for monitoring the smart chemical plant. The system structure is as follows Figure 1 As shown, the specific implementation steps are as follows:

[0043] The real-time monitoring module in the Bee Eye video surveillance AI intelligent service system includes a target detection unit, a human action recognition unit, and a video compression storage unit. The structure of the real-time monitoring module is as follows: Figure 2 As shown; wherein, the target detection unit obtains the monitoring video and processes it using a target detection algorithm to obtain a detected target;

[0044] The human action recognition unit is used to obtain the human monitoring video frame information output by the target detection unit, and recognize the human action in the human monitoring video frame information according to the convolutional attention model;

[0045] The video compression storage unit is used to obtain and process the surveillance video to obtain historical storage video data, and upload the historical storage video data to the historical surveillance video cloud platform;

[0046] Furthermore, the real-time monitoring module obtains and processes the monitoring video captured by the camera, calculates the pixel difference between the monitoring video frames of the monitoring video, and obtains the total pixel difference; if the total pixel difference exceeds a preset pixel difference threshold, there is human activity in the monitoring video.

[0047] Furthermore, two surveillance video frame images I1 and I2 separated by a preset interval time T0 in the surveillance video are obtained, and the total pixel difference between the two surveillance video frame images is calculated. The calculation formula is:

[0048]

[0049] Wherein, S represents the total pixel difference, W and H represent the width and height of the image respectively, I1(w,h) represents the pixel value of the first monitoring video frame image at the position (w,h), and I2(w,h) represents the pixel value of the second monitoring video frame image at the position (w,h);

[0050] If the total pixel difference S is greater than the preset pixel difference threshold S0, it can be determined that there is human activity in the surveillance video. Once motion is detected, the real-time monitoring module will activate the target detection unit to perform a more in-depth analysis of the human activity area on the surveillance video.

[0051] The real-time monitoring module can quickly determine whether there is human activity in the monitoring scene by calculating the pixel differences between different frames of monitoring video frame images. The use of simple but effective pixel difference method and threshold judgment not only realizes effective motion detection, but also optimizes resource mobilization and improves the overall performance of the monitoring system.

[0052] Furthermore, if there is human activity in the surveillance video, the target detection unit obtains and processes the surveillance video frame of the surveillance video through the target detection algorithm to obtain the detected target within the target detection frame, and the detected target includes humans and scene objects; the human is image segmented according to the target detection frame to obtain human surveillance video frame information.

[0053] Furthermore, the core of the target detection unit is to use existing target detection algorithms such as YOLO, SSD and FasterR-CNN to analyze the surveillance video frames to identify the humans and scene objects therein. The input is the surveillance video frame and the output is the target detection box and category label.

[0054] Furthermore, the target detection process first extracts features, using a convolutional neural network to extract features from the input surveillance video frame I to obtain a feature map. The calculation formula is:

[0055] F = f(I);

[0056] Among them, f() represents the convolution operation and F represents the feature map.

[0057] Furthermore, the algorithm is used to classify and regress each candidate box to obtain the target detection box set B = {(x, y, w, h)} and the category set C, where (x, y) represents the center coordinates of the target detection box, and (w, h) represents the width and height of the target detection box.

[0058] Furthermore, in order to avoid the redundancy of overlapping frames, the non-maximum suppression algorithm is applied to remove redundant candidate frames, and finally the detection frame B is determined. final =NMS(B,IoU threshold ), where NMS() represents the non-maximum suppression algorithm, B represents the target detection box set, and IoU threshold Represents an indicator that measures the degree of overlap of candidate boxes.

[0059] Furthermore, after obtaining the final detection frame, image segmentation is performed on each final detection frame containing human activities to obtain human monitoring video frame information.

[0060] The target detection unit can effectively identify humans and scene objects in surveillance video frames through steps such as feature extraction, region proposal generation, target classification and positioning, and non-maximum suppression, and perform accurate image segmentation on target detection frames containing human actions, providing key information for subsequent action recognition and scene analysis.

[0061] The human action recognition unit includes: obtaining and processing the human monitoring video frame information output by the target detection unit to obtain a human action data set; training a convolutional attention model according to the human action data set, the convolutional attention model including a channel attention module and a spatial attention module; the convolutional attention model is used to obtain and process the human monitoring video frame information to recognize human actions.

[0062] In order to further identify human actions, a convolutional attention model is proposed, including a channel attention mechanism and a spatial attention mechanism; among them, the channel attention mechanism focuses on the features existing on the feature map, and the spatial attention mechanism focuses on the spatial structure of the feature map.

[0063] Furthermore, the segmented human action images are of different sizes, so the preprocessing step is particularly important. First, the images are normalized and each image is adjusted to a uniform size to ensure the consistency of the model input. Subsequently, data augmentation methods such as rotation, scaling, flipping, and cropping can be implemented to improve the robustness of the model. Filters such as Gaussian filtering or median filtering are used to remove noise from the image to improve image clarity and the accuracy of feature extraction. Finally, the preprocessed images are combined with the existing action dataset and organized into a standard dataset for subsequent model training and verification.

[0064] Furthermore, the structure of the convolutional attention model is as follows Figure 3 As shown, it includes: input layer, batch normalization layer, convolution layer, maximum pooling layer, channel attention module, spatial attention module, convolution layer, fully connected layer, normalization layer and output layer.

[0065] Furthermore, the channel attention module and the spatial attention module are implemented as follows: Figure 4 As shown in the figure, first, the input features are globally pooled and average pooled, and then sent to the channel attention module and the spatial attention module respectively; in the channel attention module, the output feature is a one-dimensional vector, which is fused with the original input feature in the feature fusion layer to obtain the first fused feature, focusing on the features existing on the feature map; in the spatial attention module, the output feature is a three-dimensional vector, which is fused with the original input feature in the feature fusion layer to obtain the second fused feature, focusing on the spatial structure on the feature map; finally, the first fused feature and the second fused feature are obtained and processed to obtain the overall fused feature.

[0066] Table 1 shows an example of the training results of the convolutional attention model for human action recognition. The data shows that the accuracy of action recognition after training is above 91%.

[0067] Furthermore, the recognition result of each surveillance video frame in the surveillance video is obtained, and the action with the highest frequency is selected as the judgment result. This can be achieved through probability statistics or weighted voting. Such operations can further compensate for the misjudgment caused by the model in action recognition errors.

[0068] Furthermore, all types of actions that appear in the determined surveillance video are recorded, and if action A has the highest frequency of appearance, the action is determined to be the main action.

[0069] Table 1. Key indicators of model training

[0070] round Training set accuracy (%) Validation set accuracy (%) Training set loss value Validation set loss value 1 65.34 61.72 1.25 1.31 2 72.41 70.16 0.92 1.03 3 78.92 75.34 0.76 0.85 4 83.18 80.26 0.62 0.73 5 87.27 84.63 0.52 0.67 6 89.83 86.92 0.46 0.58 7 91.48 88.36 0.41 0.54 8 92.65 89.78 0.36 0.49 9 93.57 90.86 0.32 0.46 10 94.24 91.47 0.29 0.43

[0071] The human action recognition unit can effectively identify and analyze all human actions appearing in the surveillance video by extracting human images within the target detection frame and performing feature processing, and output detailed human action categories, providing an important basis for subsequent security monitoring and anomaly detection.

[0072] The video compression storage unit includes: obtaining the surveillance video containing human actions and performing segment processing to obtain a surveillance video segment set; calculating the clarity index of all surveillance video frames in the surveillance video segment, wherein the clarity index includes picture quality clarity and target object structure clarity; obtaining the surveillance video frame with the highest clarity index according to the clarity index to obtain a first surveillance video frame;

[0073] Acquire and process all the surveillance video segments to obtain the first surveillance video frame of each surveillance video segment; acquire and process all the first surveillance video frames to obtain stored video data;

[0074] The stored video data includes the first surveillance video frame and a surveillance tag; the surveillance tag includes a camera number, time, scene object, and human action category.

[0075] Furthermore, firstly, a surveillance video containing human actions is obtained and segmented. The video can be segmented at fixed time intervals or based on certain events, such as recognized action changes, to form a plurality of surveillance video segment sets SP = {SP i}, where SP i It represents the i-th surveillance video segment, which contains surveillance video frames within a period of time.

[0076] Furthermore, for each surveillance video segment, the clarity index of each surveillance video frame is calculated, which mainly includes two aspects: picture quality clarity, which is used to measure the overall visual quality of the surveillance video frame, and can usually be calculated using indicators based on aspects such as image sharpness and contrast; target object structure clarity, which focuses on the details of the target object in the surveillance video frame, such as the outline and structure clarity of human actions and scene objects in the surveillance video frame, which can be evaluated using technologies such as edge detection and texture analysis.

[0077] Furthermore, the calculation of the clarity index can be represented by a matrix. Assuming that the jth frame F in the i-th video segment is ij In the calculation, the picture quality clarity matrix C is q (F ij ) and the target structure clarity matrix C s (F ij ), the calculation formula of the clarity index matrix is:

[0078] C(F ij )=ωq ·C q (F ij )+ω s ·C s (F ij );

[0079] Among them, C(F ij ) represents the clarity index matrix of the j-th monitoring video frame in the i-th video segment, ω q Represents the weight coefficient of the picture quality clarity matrix, ω s Represents the weight coefficient of the target object structure clarity matrix;

[0080] Furthermore, the calculation formula of the clarity index is:

[0081]

[0082] Where QXD represents the clarity index of the jth surveillance video frame in the i-th video segment, M and N are C(F ij ) the number of rows and columns, C mn (F ij ) represents the matrix C(F ij ) at the (m,n) position.

[0083] Furthermore, after calculating the clarity indexes of all surveillance video frames in the i-th video segment, the surveillance video frame with the highest clarity index is selected for storage to obtain the first surveillance video frame.

[0084] The above operation is performed on all surveillance video segments to obtain a compressed storage video frame set SP′={SP i ′}, where SP i ′ represents the surveillance video frame with the highest clarity index in the i-th surveillance video segment, that is, the first surveillance video frame.

[0085] Furthermore, the surveillance video data is extracted to obtain surveillance labels, including camera number, timestamp, scene object category, and human action type.

[0086] Furthermore, the compressed stored video frames and monitoring tags are obtained and processed to obtain stored video data.

[0087] The video compression storage unit selects the surveillance video frames with the highest clarity index for storage, greatly reducing the storage of redundant data and optimizing the system's storage resources. The rich label data enables the system to quickly locate specific surveillance video segments without searching frame by frame, greatly improving retrieval efficiency and facilitating subsequent queries and analysis.

[0088] A rule determination module is used to obtain and process the data of the scene object in the scene object database and the human action to obtain a scene abnormality degree. If the scene abnormality degree is greater than a preset scene abnormality degree threshold, an abnormal human action exists;

[0089] Further, the rule determination module includes: acquiring the data of the scene object in the scene object database, wherein the data includes the importance weight of the scene object;

[0090] Furthermore, in a smart factory, the scene objects mainly include the factory's automated production equipment, dangerous equipment, material storage areas, and confidential areas, and their importance weights are shown in Table 2.

[0091] Table 2. Scene objects and their importance weights

[0092] Scene object category describe Importance Weight Automated production equipment Mainly used for daily automated production in factories 0.7 Dangerous equipment Including high temperature and high pressure equipment, etc., improper operation may lead to safety accidents 0.9 Material storage area Storage of important production materials and raw materials 0.6 Confidential Area Areas that store patents, sensitive information, or factory secrets 1.0 Personnel entrance and exit Areas where factory workers enter and exit with high turnover 0.3 Emergency exit Emergency exits must be kept clear 0.5 Energy control system Involving the supply and management of energy in factories, such as electricity or gas 0.8 Monitoring Center Real-time monitoring of safety and production conditions in all areas of the factory 0.9 Fire Fighting Equipment Important facilities for emergency situations such as fires 0.8 High pressure gas storage area Areas where hazardous gases are stored and there is a risk of explosion 0.9

[0093] Obtain the geometric centers of the target detection frames of the human action and the scene object, calculate the distance between the geometric centers of the target detection frames of the human action and the scene object, obtain the distance and the importance weight and process them to obtain a first scene abnormality; obtain the overlapping area between the target detection frame of the human action and the target detection frame of the scene object and process them to obtain a second scene abnormality; obtain the first scene abnormality and the second scene abnormality and process them, and judge whether there is abnormal human action in the surveillance video according to the abnormality judgment function.

[0094] Furthermore, the distance between the geometric center of the detection frame of the human action and the geometric center of the detection frame of the scene object is calculated, and the distance correction based on scene calibration and perspective transformation correction are combined to obtain the corrected distance between the human action and the scene object, which is represented by the real distance set d = {d z}, where d z represents the distance from the human action to the zth scene object. Then the importance weight of the scene object in the scene object database is obtained, and the first scene abnormality is calculated using the following formula:

[0095]

[0096] Among them, DAN1 represents the abnormality of the first scene, Z represents the number of scene objects, β z Represents the importance weight of the z-th scene object.

[0097] Furthermore, the calculation formula of the abnormality degree of the second scene is:

[0098]

[0099] Among them, DAN2 represents the abnormality of the second scene, s z represents the overlapping area between the human action detection frame and the z-th scene object detection frame, t z Represents the overlap time between the human action detection frame and the z-th scene object detection frame.

[0100] Furthermore, the calculation formula of the comprehensive abnormality is:

[0101] DAN=γ1·DAN1+γ2·DAN2;

[0102] Among them, DAN represents the comprehensive abnormality, γ1 represents the weight coefficient of the abnormality of the first scene, and γ2 represents the weight coefficient of the abnormality of the second scene.

[0103] Furthermore, according to the abnormality judgment function, it is judged whether there is abnormal human action in the surveillance video. The formula of the abnormality judgment function is:

[0104]

[0105] Wherein, E represents the warning degree, DAN0, DAN1 and DAN2 represent the first, second and third preset abnormality thresholds;

[0106] Furthermore, if the comprehensive abnormality is greater than a first preset abnormality threshold, abnormal human action exists in the surveillance video.

[0107] By calculating the actual distance between human actions and scene objects and combining the importance weights of scene objects, the threat level of human actions to target objects can be judged more accurately. In addition to calculating the distance between geometric centers, the second scene abnormality degree is introduced, which takes into account the overlapping area and overlapping time of human actions and target objects. This can more comprehensively reflect the actual impact of human actions on scene objects. Through the calculation of comprehensive abnormality, the system can intelligently determine whether there are abnormal human actions in the surveillance video.

[0108] The query module includes: a first query unit obtains and processes the camera number to obtain all historical stored video data under the camera number; obtains and processes the stored video data containing abnormal human actions and the monitoring tags of all the historical stored video data to obtain a first scene similarity, and if the first scene similarity exceeds a preset first scene similarity threshold, the historical stored video data is saved to a first data set; a second query unit obtains the stored video data containing the abnormal human action and the historical stored video data in the first data set, calculates a second scene similarity according to the difference in clarity index, and if the second scene similarity is less than the second scene similarity threshold, the historical stored video data is uploaded to the abnormal human action video set.

[0109] Furthermore, the calculation formula of the first scene similarity is:

[0110]

[0111] Among them, XSD1 represents the first scene similarity, G represents the total number of monitoring tags, η g represents the weight coefficient of the g-th monitoring label, sim() represents the similarity function, A g The g-th surveillance label of the surveillance video with abnormal human actions, A′ g Represents the g-th surveillance video label in the historical surveillance video data.

[0112] Furthermore, the similarity calculation of timestamps can be measured according to the closeness of the time of the two videos, usually using the inverse of the time difference or an exponential decay function; the scene object category can use the classification similarity method, for example: if the scene object category is the same, the similarity is 1, if the scene object category is different, the similarity is 0; the similarity of human action types can be calculated based on the category label of the human action identified by the convolutional attention model, for example: if the action type is the same, the similarity is 1; if the action type is different, the similarity is 0.

[0113] Furthermore, the second query unit is used to obtain historical stored video data that is most similar to the surveillance video scene and the surveillance video with abnormal human actions among the historical stored video data in the first data set.

[0114] The first query unit obtains the camera number in the video and calculates the first scene similarity based on label data such as time, scene objects and human actions, thereby improving the retrieval efficiency of historical video data; the second scene similarity is calculated based on the clarity index, and the stored data is extracted to provide data support for subsequent warnings.

[0115] The early warning module includes: obtaining the abnormal human action video set, and sending the abnormal human action video set to the main control interface and security personnel; obtaining and processing the scene abnormality and abnormality judgment function, and issuing an early warning according to the abnormality and human action category;

[0116] Furthermore, the warning also includes the location of the abnormal human action, and is prompted in the real-time monitoring picture containing the abnormal human action.

[0117] Furthermore, the system obtains the abnormal human action video set and sends the abnormal human action video set to the security personnel to ensure that the security personnel can take timely action.

[0118] Further obtaining the warning degree obtained in the rule determination module for early warning, and also combining the category of the abnormal human action existing in the monitoring video for comprehensive early warning;

[0119] Furthermore, for example, if a person is identified in the surveillance video as standing still for a long time in a confidential area such as the door of an office, an early warning will be issued and attention will be paid to the situation in the surveillance video; if a person is identified as wandering around and picking the lock in a confidential area such as the door of an office, a high-risk early warning will be issued immediately.

[0120] By automatically collecting videos of abnormal human movements and sending them to security personnel, the system can significantly reduce the response time to abnormal human movements and ensure that security personnel receive and respond in a timely manner. By combining the abnormality of the scene and the category of abnormal human movements for early warning, the system can provide more accurate risk assessments and provide reference data for the actions taken by security personnel.

[0121] The abnormal human motion tracking module includes: obtaining the position of the abnormal human motion, calculating the influence range of the abnormal human motion according to the position and the scene object, obtaining the surveillance camera within the influence range and performing target tracking.

[0122] Furthermore, the target tracking includes using a target tracking algorithm to continuously track the area where the abnormal human action is located, and updating the location of the abnormal human action in real time.

[0123] Furthermore, the areas of the target detection frames of the abnormal human action in all cameras are calculated and sorted, and the real-time monitoring screen with the largest area is presented on the main control interface.

[0124] Furthermore, the position of the abnormal human action is obtained, and the radius r of the spatial area where the abnormal human action occurs is calculated in combination with the position of the scene object and the abnormal human action in the surveillance video, and the surveillance cameras within the spatial area radius are obtained and tracked.

[0125] Furthermore, a target tracking algorithm is used to continuously track abnormal persons in the area, and the position is updated in real time to ensure the continuity of monitoring; when the target leaves the field of view of the current camera, the module automatically switches to the camera in the adjacent area to ensure that the target person is always under monitoring.

[0126] By calculating the spatial area radius of abnormal human movements, the system can quickly identify and lock relevant cameras to achieve continuous monitoring. This not only improves real-time response capabilities, but also reduces monitoring blind spots and ensures that the target is always under monitoring. In addition, by sorting the target detection frame area, the most relevant images can be displayed first, helping security personnel make quick judgments.

[0127] Furthermore, the target tracking also includes: acquiring and processing the abnormal human action to obtain the motion trajectory of the abnormal human action and issuing a trajectory warning; acquiring the data of the scene object in the scene object database, the data including the importance weight of the scene object; monitoring the scene object whose importance weight exceeds a preset importance threshold in the real-time monitoring video where the abnormal human action exists.

[0128] Furthermore, a surveillance video of the abnormal human action is obtained, and the motion trajectory of the abnormal human action is predicted according to the moving direction of the geometric center of the target detection frame of the abnormal human action in the surveillance video, and a trajectory warning is issued to prompt security personnel to take preventive measures and rush to the scene in time.

[0129] Furthermore, considering the high-value objects in the scene, the camera should also monitor the important objects around the abnormal movement and issue early warnings to security personnel in time to protect the important objects.

[0130] By analyzing the moving direction of the target detection frame in the surveillance video in real time, the system can accurately predict the movement trajectory of abnormal human actions, thereby issuing trajectory warnings in advance, helping security personnel take preventive measures in time to avoid potential risks. At the same time, monitoring important scenes around abnormal movements can ensure that these valuable assets are effectively protected and prevent theft or destruction.

[0131] The Bee-Eye video surveillance AI intelligent service system provided in this embodiment realizes automatic detection and recognition of human actions in surveillance videos through the convolutional attention model in the human action recognition unit, reduces the need for manual monitoring, and greatly improves monitoring efficiency; through the video compression storage unit, historical surveillance videos are effectively compressed and stored in the cloud platform, which improves the storage efficiency of video data and facilitates subsequent query and backtracking; the early warning module provides real-time early warning based on the category of abnormal human actions and the degree of scene abnormality, informs security personnel of security risks in advance, enables quick response and prevention of potential threats, and improves security response speed.

[0132] Example 2

[0133] The security system of a certain school applies a surveillance video AI analysis system based on computer vision and deep learning technology provided by the present invention to improve the ability to detect abnormal behaviors, including abnormal behaviors of students and possible abnormal behaviors of outsiders, and effectively improve the school's safety management level. The following is a specific application of this method in school security monitoring:

[0134] Furthermore, adding a uniform recognition unit in the real-time monitoring module can effectively improve the ability to identify and track personnel behavior. Uniform recognition can be used as an additional screening condition, especially in a campus environment, to distinguish between students, faculty and staff, and outsiders.

[0135] Furthermore, in the rule judgment module, the school scene objects and their importance weights are obtained, and the comprehensive scene abnormality is calculated in combination with the identified human actions, and the abnormality judgment function is used to determine whether there are abnormal human actions in the surveillance video. The school scene objects and their importance weights are shown in Table 3.

[0136] Table 3. School scene objects and their importance weights

[0137] Scenery describe Importance Weight Access control area Monitor campus entrances and exits and important passages 1.0 Security cameras Sources of video data in monitoring systems 0.95 vehicle Monitor the vehicles entering and leaving the school to prevent vehicle violations 0.75 Electronic devices Campus surveillance, computer equipment, etc. 0.5 Campus Fence Detect if someone is climbing or climbing over 0.85

[0138] Furthermore, in the rule determination module, if there are multiple human actions in the surveillance video, the comprehensive abnormality is calculated for each of the existing human actions, and then all the comprehensive abnormalities are obtained to calculate the overall abnormality of the surveillance video containing multiple human actions. The calculation formula of the overall abnormality is:

[0139]

[0140] in, represents the overall abnormality, K represents the number of human actions in the surveillance video, and ρ k DAN represents the comprehensive abnormality of the kth human action k The weight coefficient of .

[0141] Furthermore, the overall abnormality of the surveillance video containing multiple human actions is obtained, and whether there is abnormal human action in the surveillance video is determined according to the abnormality judgment function.

[0142] Furthermore, in the query module, when abnormal human movements are detected, the system can quickly query historical videos using parameters such as camera number, time, and scene objects.

[0143] Furthermore, when calculating the similarity of the second scene, the clarity index matrix of the surveillance video frame and the clarity index matrix of the surveillance video frame in the first data set are obtained and difference processing is performed to obtain the similarity of the second scene. The calculation formula is:

[0144]

[0145] Among them, XSD2 represents the similarity of the second scene, C mn (F) represents the clarity score of the surveillance video frame’s clarity index matrix at position (m,n), C mn (F e) represents the clarity score at the (m,n) position of the clarity index matrix of the surveillance video frame in the first data set.

[0146] Furthermore, after calculating all the second scene similarities, the surveillance video frame in the first data set corresponding to the smallest second scene similarity is selected and uploaded to the abnormal human action video frame set.

[0147] Furthermore, in the early warning module, after detecting abnormal human movements, the system will send warning information to relevant departments such as school police and teachers on duty according to the severity of the warning. At the same time, if dangerous behaviors such as carrying suspicious items into the campus are detected, the system can combine the importance weights of the scene objects and issue an emergency warning.

[0148] Furthermore, in the abnormal human action tracking module, the target is continuously monitored through the target tracking algorithm. If the abnormal person leaves the coverage of the current camera, the system automatically switches to the adjacent camera to ensure that the person is always within the monitoring range; the system predicts the movement trajectory of the abnormal human action based on the moving direction of the geometric center of the target detection frame of the abnormal human action in the monitoring video, and issues a trajectory warning to prompt security personnel to take preventive measures and rush to the scene in time.

[0149] By applying this system to school security monitoring, all-round intelligent monitoring of the campus can be achieved, helping security personnel to respond to emergencies in a timely manner and ensure the safety of students and faculty.

[0150] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A surveillance video AI analysis system based on computer vision and deep learning technology, characterized in that: include: The real-time monitoring module includes a target detection unit, a human action recognition unit and a video compression storage unit; wherein the target detection unit obtains the monitoring video and processes it using a target detection algorithm to obtain a detected target; The human action recognition unit is used to obtain the human monitoring video frame information output by the target detection unit, and recognize the human action in the human monitoring video frame information according to the convolutional attention model; The video compression storage unit is used to obtain and process the surveillance video to obtain historical storage video data, and upload the historical storage video data to the historical surveillance video cloud platform; A rule determination module is used to obtain and process the data of the scene object in the scene object database and the human action to obtain a scene abnormality degree. If the scene abnormality degree is greater than a preset scene abnormality degree threshold, an abnormal human action exists; A query module, which obtains the surveillance video with the abnormal human action and the data in the historical surveillance video cloud platform and calculates the scene similarity, and obtains a set of abnormal human action videos according to a preset scene similarity threshold; An early warning module, used for issuing an early warning according to the abnormal human action; The abnormal human motion tracking module is used to obtain the area where the abnormal human motion is located and call the camera in the area for continuous monitoring.

2. According to claim 1, a surveillance video AI analysis system based on computer vision and deep learning technology is characterized in that: The real-time monitoring module includes: The real-time monitoring module obtains and processes the monitoring video captured by the camera, calculates the pixel difference between the monitoring video frames of the monitoring video, and obtains the total pixel difference; if the total pixel difference exceeds a preset pixel difference threshold, human activities exist in the monitoring video.

3. According to claim 1, a surveillance video AI analysis system based on computer vision and deep learning technology is characterized in that: The target detection unit comprises: If there is human activity in the surveillance video, the target detection unit obtains and processes the surveillance video frame of the surveillance video through the target detection algorithm to obtain the detected target in the target detection frame, and the detected target includes humans and scene objects; the human image is segmented according to the target detection frame to obtain human surveillance video frame information.

4. According to claim 1, a surveillance video AI analysis system based on computer vision and deep learning technology is characterized in that: The human action recognition unit comprises: The human monitoring video frame information output by the target detection unit is obtained and processed to obtain a human action data set; a convolutional attention model is trained according to the human action data set, and the convolutional attention model includes a channel attention module and a spatial attention module; the convolutional attention model is used to obtain and process the human monitoring video frame information to obtain human action.

5. The surveillance video AI analysis system based on computer vision and deep learning technology according to claim 1 is characterized in that: The video compression storage unit comprises: The surveillance video containing human actions is obtained and segmented to obtain a surveillance video segment set; the clarity index of all surveillance video frames in the surveillance video segment is calculated, and the clarity index includes picture quality clarity and target object structure clarity; according to the clarity index, the surveillance video frame with the highest clarity index is obtained to obtain a first surveillance video frame; Acquire and process all the surveillance video segments to obtain the first surveillance video frame of each surveillance video segment; acquire and process all the first surveillance video frames to obtain stored video data; The stored video data includes the first surveillance video frame and a surveillance tag; the surveillance tag includes a camera number, time, scene object, and human action category.

6. The surveillance video AI analysis system based on computer vision and deep learning technology according to claim 1 is characterized in that: The rule determination module comprises: Acquire the data of the scene object in the scene object database, wherein the data includes the importance weight of the scene object; Obtain the geometric centers of the target detection frames of the human action and the scene object, calculate the distance between the geometric centers of the target detection frames of the human action and the scene object, obtain the distance and the importance weight and process them to obtain a first scene abnormality; obtain the overlapping area between the target detection frame of the human action and the target detection frame of the scene object and process them to obtain a second scene abnormality; obtain the first scene abnormality and the second scene abnormality and process them, and judge whether there is abnormal human action in the surveillance video according to the abnormality judgment function.

7. The surveillance video AI analysis system based on computer vision and deep learning technology according to claim 1, characterized in that: The query module includes: The first query unit obtains the camera number and processes it to obtain the historical stored video data under the camera number; obtains the stored video data containing the abnormal human action and the monitoring tags of all the historical stored video data and processes them to obtain the first scene similarity. If the first scene similarity exceeds the preset first scene similarity threshold, the historical stored video data is saved to the first data set; the second query unit obtains the stored video data containing the abnormal human action and the historical stored video data in the first data set, calculates the second scene similarity according to the clarity index, and if the second scene similarity is less than the preset scene similarity threshold, the historical stored video data is uploaded to the abnormal human action video set.

8. The surveillance video AI analysis system based on computer vision and deep learning technology according to claim 1, characterized in that: The early warning module comprises: Acquire the abnormal human action video set, and send the abnormal human action video set to the main control interface; Obtaining and processing the scene abnormality and abnormality judgment function, and issuing an early warning according to the scene abnormality and human action category; The early warning also includes providing a location prompt of the abnormal human action in a real-time monitoring picture including the abnormal human action.

9. The surveillance video AI analysis system based on computer vision and deep learning technology according to claim 1, characterized in that: The abnormal human motion tracking module includes: Obtaining the position of the abnormal human action, and calculating the influence range of the abnormal human action according to the position and the scene object, obtaining the surveillance cameras within the influence range and performing target tracking; The target tracking includes using a target tracking algorithm to continuously track the area where the abnormal human action is located, and updating the location of the abnormal human action in real time; The areas of the target detection frames of the abnormal human actions in the target detection algorithm are calculated and sorted, and the real-time monitoring screen with the largest area is presented on the main control interface.

10. The surveillance video AI analysis system based on computer vision and deep learning technology according to claim 9, characterized in that: The target tracking also includes: Acquire and process the abnormal human action, obtain the motion trajectory of the abnormal human action, and issue a trajectory warning; acquire the data of the scene object in the scene object database, wherein the data includes the importance weight of the scene object; and monitor the scene object whose importance weight exceeds a preset importance threshold in the real-time monitoring video in which the abnormal human action exists.

Images