Video image processing method and device based on deep learning

Abstract

The invention provides a video image processing method and device based on deep learning. The method comprises the following steps: acquiring monitoring video streams of the same monitoring device in different time periods and respectively decoding the monitoring video streams into sequence frame images; calculating information entropy, and training a depth target frame selection model; collecting human body features and object features of the target frame image data, and training a target detection model; detecting a human body feature image and an object feature image of the target frame image, respectively graying and calculating information entropy; and training the anomaly recognition model. According to the invention, the abnormal scene monitored by the monitoring video can be intelligently and accurately identified under the condition of saving manpower and system resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of video image processing, and in particular relates to a video image processing method and device based on deep learning. Background Art

[0002] Currently, more and more communities, schools and other areas have installed and used video surveillance systems. One of the main purposes of installing video surveillance systems is for safety. However, the current video surveillance systems cannot respond in real time. Even if a duty officer is assigned to view the video, it is impossible for the duty officer to observe all the time. Therefore, the video surveillance system is more for providing video recordings. However, for some special situations that endanger safety, such as someone breaking into a door, picking a lock, climbing over a wall, etc., it is too late to look for the video recordings afterwards. Therefore, it is necessary to be able to detect abnormal scenes in real time so as to respond quickly, which puts forward the demand for the intelligence of the video surveillance system.

[0003] There are intelligent algorithms for image analysis and recognition in the prior art, but they are not suitable for video surveillance systems. Video has 24 frames per second, and video surveillance systems generate a large amount of frame image data. Directly using existing image analysis and recognition models will take up a lot of resources during analysis and recognition due to the excessive frame image data, but may not necessarily obtain accurate analysis results. Summary of the invention

[0004] The present invention proposes a video image processing method and device based on deep learning, which can intelligently and relatively accurately identify abnormal scenes detected by surveillance videos while saving manpower and system resources.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] A video image processing method based on deep learning, comprising:

[0007] S1. Obtain monitoring video streams of the same monitoring device at different time periods and decode them into sequence frame images respectively; mark a target frame image in the sequence frame images of each time period and select the first n frame images and the last n frame images of the target frame image;

[0008] S2, graying out 2n+1 frame images at different time periods and calculating information entropy respectively; using 2n+1 frame image data at different time periods and corresponding information entropy, training a deep target frame selection model;

[0009] S3, collecting human features and object features of each target frame image data, and training the target detection model;

[0010] S4, respectively detecting the human feature image and the object feature image of the target frame image through the target detection model, graying them respectively and calculating the information entropy; selecting the abnormal frame image for which an alarm needs to be issued from all the target frame images; using the human feature image, the object feature image, the information entropy and the abnormal frame image to train the abnormal recognition model;

[0011] S5. Decode the monitoring video stream data into sequence frame images in real time, select the target frame through the deep target frame selection model, detect the human feature image and the object feature image through the target detection model, and determine whether an abnormality requires an alarm through the abnormality recognition model.

[0012] Furthermore, in step S2 or S4, the method for calculating information entropy includes:

[0013] S201, the pixel grayscale of the frame image and the neighborhood grayscale are combined to form a binary feature (i, j), where i represents the grayscale value of the pixel and j represents the neighborhood grayscale;

[0014] S202: Count the occurrence probability P of the binary feature (i, j) in the frame image ij ;

[0015] S203, calculating information entropy E,

[0016] Furthermore, in step S2, the training process of the depth target frame selection model includes:

[0017] S211, splicing the frame image data and the corresponding information entropy into feature vector data;

[0018] S212, forming vector group data with feature vector data of 2n+1 frame images, and dividing vector groups in different time periods into training sets, verification sets, and test sets;

[0019] S213. Use the vector group data of the training set as the input of the convolutional neural network, and use the target frame image corresponding to the vector group data as the target of the convolutional neural network for training to obtain a deep target frame selection model.

[0020] Furthermore, step S3 specifically includes:

[0021] S301, marking the human body feature image and the object feature image in the target frame image of each time period;

[0022] S302, constructing a data set based on the target frame image data of each time period, and dividing it into a training set, a validation set, and a test set;

[0023] S303, select the yolov5 model, take the target frame image data as the input of the yolov5 model, take the human body feature image and the object feature image as the target of the yolov5 model, perform training, and obtain the target detection model.

[0024] Furthermore, the training of the abnormality recognition model in step S4 includes:

[0025] S401, splicing the human body feature image, the object feature image, and the corresponding information entropy into feature vector data; dividing into a training set, a validation set, and a test set;

[0026] S402, using the feature vector data of the training set as the input of the convolutional neural network, using the abnormal target frame image as the target of the convolutional neural network, and performing training to obtain the abnormality recognition model.

[0027] On the other hand, the present invention also proposes a video image processing device based on deep learning, comprising:

[0028] Acquisition module: acquires surveillance video streams of the same surveillance device at different time periods and decodes them into sequence frame images respectively; marks a target frame image in the sequence frame images of each time period and selects the first n frame images and the last n frame images of the target frame image;

[0029] Deep target frame selection model module: grayscale the 2n+1 frame images at different time periods and calculate the information entropy; use the 2n+1 frame image data at different time periods and the corresponding information entropy to train the deep target frame selection model;

[0030] Target detection model module: collects human features and object features of each target frame image data and trains the target detection model;

[0031] Abnormal identification model module: detect the human feature image and object feature image of the target frame image respectively through the target detection model, grayscale them respectively and calculate the information entropy; select the abnormal frame image for which an alarm needs to be issued from all target frame images; use the human feature image, object feature image, information entropy and abnormal frame image to train the abnormal identification model;

[0032] Execution module: decodes the monitoring video stream data into sequence frame images in real time, selects the target frame through the deep target frame selection model, detects the human feature image and the object feature image through the target detection model, and determines whether an abnormality requires an alarm through the abnormality recognition model.

[0033] Furthermore, in the deep target frame selection model module or the anomaly recognition model module, calculating the information entropy includes:

[0034] Feature unit: The pixel grayscale of the frame image and the neighborhood grayscale form a binary feature (i, j), where i represents the grayscale value of the pixel and j represents the neighborhood grayscale;

[0035] Probability unit: Statistical binary feature (i, j) in the frame image probability P ij ;

[0036] Entropy calculation unit: calculate information entropy E,

[0037] Furthermore, the deep target frame selection model module includes:

[0038] Splicing unit: splicing frame image data and corresponding information entropy into feature vector data;

[0039] Division unit: The feature vector data of 2n+1 frame images are used to form vector group data, and the vector groups in different time periods are divided into training set, verification set and test set;

[0040] Training unit: The vector group data of the training set is used as the input of the convolutional neural network, and the target frame image corresponding to the vector group data is used as the target of the convolutional neural network for training to obtain a deep target frame selection model.

[0041] Furthermore, the target detection model module includes:

[0042] Annotation unit: annotating the human feature images and object feature images in the target frame images of each time period;

[0043] The second division unit: constructs a data set based on the target frame image data of each time period, and divides it into a training set, a validation set, and a test set;

[0044] The second training unit: select the yolov5 model, use the target frame image data as the input of the yolov5 model, use the human feature image and the object feature image as the target of the yolov5 model, and train it to obtain the target detection model.

[0045] Furthermore, the anomaly recognition model module includes:

[0046] The third division unit: splices the human feature image, the object feature image, and the corresponding information entropy into feature vector data; divides the data into a training set, a verification set, and a test set;

[0047] The third training unit: uses the feature vector data of the training set as the input of the convolutional neural network, uses the abnormal target frame image as the target of the convolutional neural network, performs training, and obtains the abnormal recognition model.

[0048] Compared with the prior art, the method and system for automatically updating planning vector data in the field of urban planning proposed by the present invention have the following beneficial effects:

[0049] 1. By constructing a deep target frame selection model, the present invention can intelligently select target frame images for image recognition from video data in real time without manual intervention, thus saving system computing resources;

[0050] 2. The present invention constructs a target detection model to perform target detection on the intelligently selected target frame image, and can intelligently detect human feature images and object feature images for abnormal scene recognition;

[0051] 3. By constructing an abnormal recognition model, the present invention can identify abnormal scenes through the human feature images and object feature images intelligently detected by the target detection model, thereby realizing automatic and intelligent analysis and processing of video images. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the overall process of Example 1 of the present invention;

[0053] Figure 2 is a schematic diagram of a flow chart of constructing a depth target frame selection model according to Embodiment 1 of the present invention;

[0054] Figure 3 1 is a schematic diagram of the process of constructing a target detection model according to Embodiment 1 of the present invention;

[0055] Figure 4 It is a schematic diagram of the process of constructing an abnormality recognition model in Example 1 of the present invention. DETAILED DESCRIPTION

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0057] In order to make the purpose and features of the present invention more obvious and understandable, the specific implementation methods of the present invention are further described below in conjunction with the accompanying drawings.

[0058] The present invention mainly focuses on monitored video images and can automatically identify abnormal images in the monitoring through an intelligent model, so that processing methods such as issuing alarm information can be carried out in a timely manner.

[0059] The design concept of the present invention is mainly divided into three parts. The first part is to intelligently select the target frame image that is most suitable for intelligent identification in the monitored video stream. The second part is to perform target detection on the selected target frame image to detect the human image and object image in the image. The third part is to intelligently identify whether it is an abnormal scene that requires alarm or alarm based on the image features of the human body and the object. The abnormal scene refers to the illegal behavior of the person in the image, such as breaking the door, picking the lock, climbing over the wall, etc.

[0060] Embodiment 1:

[0061] Based on the above design concept, the video image processing method based on deep learning implemented in this embodiment is as follows: Figure 1 As shown, including:

[0062] 1. Obtain video image data;

[0063] The purpose of this step is to collect image data for intelligent model training. The method is as follows:

[0064] The monitoring video streams of the same monitoring device at different time periods are obtained and decoded into sequence frame images respectively; that is, the historical monitoring video of the monitoring device is cut into multiple video streams at the same time interval, and each video stream is decoded to obtain its own sequence frame image.

[0065] Mark a target frame image that is the clearest and most recognizable in the sequence frame images of each time period (that is, the clearest image that can reflect whether the current scene is normal or abnormal), and then select the first n frame images and the last n frame images of the target frame image; for example, select the first 12 frame images and the last 12 frame images of the target frame image.

[0066] 2. Build a deep target frame selection model;

[0067] 2.1 grayscale the 2n+1 frame images of different time periods respectively;

[0068] In this step, basic grayscale conversion is performed, for example, by using the average value method to average the brightness values ​​of the three components R, G, and B of each pixel in the image to obtain a value as the grayscale value of the pixel.

[0069] 2.2 Calculate the information entropy of the frame image;

[0070] Based on the grayscale in the previous step, calculate the information entropy.

[0071] In the present invention, information entropy is used for abnormal image recognition and needs to reflect the spatial characteristics of the image. Therefore, in this embodiment, the two-dimensional entropy value of the image is calculated.

[0072] First, establish the binary features of each pixel of the image, represented by (i, j), where i represents the grayscale value of the pixel obtained in step 2.1, 0<=i<=255; j represents the neighborhood grayscale of the pixel, which can be represented by the average grayscale value of each neighboring pixel of the pixel, 0<=j<=255;

[0073] Secondly, the probability P of the binary feature (i, j) appearing in the frame image is counted ij ;P ij =f(i,j) / N 2; f(i,j) represents the frequency of occurrence of binary feature (i,j) in the image, and N represents the scale of the image.

[0074] Then the calculation is performed according to the two-dimensional entropy calculation formula; E is information entropy.

[0075] 2.3 Deep target frame selection model training:

[0076] 2.3.1. Splice the frame image data and the corresponding information entropy into feature vector data;

[0077] 2.3.2. The feature vector data of 2n+1 frame images in each time period are combined to form vector group data. All vector group data are divided into training set, validation set and test set according to the ratio of 6:2:2;

[0078] 2.3.3. The vector group data of the training set is used as the input of the convolutional neural network. The target frame images annotated in the 2n+1 frame images corresponding to each set of vector group data are used as the targets of the convolutional neural network for training to obtain a deep target frame selection model. The model parameters are determined through the validation set, and the model performance is tested through the test set.

[0079] 3. Build target detection model:

[0080] Since the present invention mainly recognizes abnormal scenes based on the image features of human bodies and objects, the detection of human body features and object features in frame images is very important. The purpose of this step is to train a target detection model that can detect human body features and object features.

[0081] 3.1 Collect the human features and object features of each target frame image data.

[0082] The collection of human features and object features in this step can be completed by marking on the target frame image.

[0083] 3.2 Build a dataset with all target frame image data and divide it into training set, validation set and test set in a ratio of 6:2:2;

[0084] 3.3 Select the yolov5 model, which sets an attention mechanism module and a classification module;

[0085] The target frame image data is used as the input of the yolov5 model, and the human feature image and object feature image are used as the target of the yolov5 model. The target frame image data is convolved and globally averaged pooled through the attention mechanism module. The output of the attention mechanism module is used as the input of the classification module. The human feature image and object feature image are output by the classification module as the output of the yolov5 model. After the above training process, the target detection model is obtained. The model parameters are determined through the validation set, and the model performance is tested through the test set.

[0086] 4. Build an anomaly recognition model:

[0087] 4.1 Detect and obtain human feature images and object feature images;

[0088] After the target detection model is obtained through the above steps, the target frame images of each time period are detected through the target detection model to obtain human feature images and object feature images.

[0089] 4.2 Calculate information entropy;

[0090] The human feature image and object feature image are grayed out respectively and the information entropy is calculated; the information entropy calculation method is the same as the calculation method described in the above step 2.2, and this step will not be repeated.

[0091] 4.3 Select the images with abnormal scenes from all target frame images;

[0092] The abnormal scene refers to the illegal actions of the people in the image, such as breaking a door, picking a lock, climbing over a wall, etc.

[0093] 4.4 Constructing the dataset;

[0094] The human body feature image, the object feature image, and the corresponding information entropy are spliced ​​into feature vector data; all feature vector data are divided into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0095] 4.5 Model training;

[0096] The feature vector data of the training set is used as the input of the convolutional neural network, and the abnormal target frame image is used as the target of the convolutional neural network for training to obtain the abnormal recognition model. The model parameters are determined through the validation set, and the model performance is tested through the test set.

[0097] 5. Perform video image processing:

[0098] The monitoring video stream data is decoded into a sequence of frame images in real time, the real-time target frame is selected by the deep target frame selection model, the human feature image and the object feature image of the real-time target frame are detected by the target detection model, and the abnormal recognition model is used to determine whether it is an abnormal scene that requires an alarm or an alarm message.

[0099] The above method can be used to intelligently and relatively accurately identify abnormal scenes detected by surveillance videos, avoiding the need for manual intervention.

[0100] Embodiment 2:

[0101] Embodiment 2 proposes a video image processing device based on deep learning, including:

[0102] Acquisition module: acquires surveillance video streams of the same surveillance device at different time periods and decodes them into sequence frame images respectively; marks a target frame image in the sequence frame images of each time period and selects the first n frame images and the last n frame images of the target frame image;

[0103] Deep target frame selection model module: grayscale 2n+1 frame images of different time periods and calculate information entropy; use 2n+1 frame image data of different time periods and corresponding information entropy to train the deep target frame selection model; the calculation of information entropy includes: feature unit: the pixel grayscale of the frame image and the neighborhood grayscale form a binary feature (i, j), where i represents the grayscale value of the pixel and j represents the neighborhood grayscale; probability unit: statistically calculate the probability P of the binary feature (i, j) appearing in the frame image ij ; Entropy calculation unit: calculate information entropy E, It also includes a splicing unit for model training: splicing frame image data and corresponding information entropy into feature vector data; a division unit: using feature vector data of 2n+1 frame images to form vector group data, and dividing vector groups in different time periods into training set, verification set, and test set; a training unit: using the vector group data of the training set as the input of the convolutional neural network, and the target frame image corresponding to the vector group data as the target of the convolutional neural network, and training is performed to obtain a deep target frame selection model.

[0104] Target detection model module: collects human features and object features of each target frame image data, and trains the target detection model; includes: annotating unit: annotating the human feature images and object feature images in the target frame images of each time period; the second division unit: constructs a data set with the target frame image data of each time period, and divides it into training set, verification set and test set; the second training unit: selects the yolov5 model, takes the target frame image data as the input of the yolov5 model, takes the human feature images and object feature images as the targets of the yolov5 model, and conducts training to obtain the target detection model.

[0105] Abnormal identification model module: detect the human feature image and object feature image of the target frame image respectively through the target detection model, grayscale them respectively and calculate the information entropy; select the abnormal frame image for which an alarm needs to be issued from all target frame images; use the human feature image, object feature image, information entropy and abnormal frame image to train the abnormal identification model; including: a third division unit: splicing the human feature image, object feature image and corresponding information entropy into feature vector data; dividing into training set, verification set and test set; a third training unit: using the feature vector data of the training set as the input of the convolutional neural network, and using the abnormal target frame image as the target of the convolutional neural network, training is performed to obtain the abnormal identification model.

[0106] Execution module: decodes the monitoring video stream data into sequence frame images in real time, selects the target frame through the deep target frame selection model, detects the human feature image and the object feature image through the target detection model, and determines whether an abnormality requires an alarm through the abnormality recognition model.

[0107] The deep learning-based video image processing device described in this embodiment can implement the deep learning-based video image processing method described in Example 1, and the technical effect achieved is the same as that of Example 1.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video image processing method based on deep learning, characterized in that: include: S1. Obtain monitoring video streams of the same monitoring device at different time periods and decode them into sequential frame images respectively; Marking a target frame image in the sequence frame images of each time period and selecting the first n frame images and the last n frame images of the target frame image; S2, graying out 2n+1 frame images at different time periods and calculating information entropy respectively; using 2n+1 frame image data at different time periods and corresponding information entropy, training a deep target frame selection model; S3, collecting human features and object features of each target frame image data, and training the target detection model; S4, respectively detecting the human feature image and the object feature image of the target frame image through the target detection model, graying them respectively and calculating the information entropy; selecting the abnormal frame image for which an alarm needs to be issued from all the target frame images; using the human feature image, the object feature image, the information entropy and the abnormal frame image to train the abnormal recognition model; S5. Decode the monitoring video stream data into sequence frame images in real time, select the target frame through the deep target frame selection model, detect the human feature image and the object feature image through the target detection model, and determine whether an abnormality requires an alarm through the abnormality recognition model.

2. The video image processing method based on deep learning according to claim 1, characterized in that: In step S2 or S4, the method for calculating information entropy includes: S201, the pixel grayscale of the frame image and the neighborhood grayscale are combined to form a binary feature (i, j), where i represents the grayscale value of the pixel and j represents the neighborhood grayscale; S202: Count the occurrence probability P of the binary feature (i, j) in the frame image ij ; S203, calculating information entropy E, 3. The video image processing method based on deep learning according to claim 1, characterized in that: In step S2, the training process of the depth target frame selection model includes: S211, splicing the frame image data and the corresponding information entropy into feature vector data; S212, forming vector group data with feature vector data of 2n+1 frame images, and dividing vector groups in different time periods into training sets, verification sets, and test sets; S213. Use the vector group data of the training set as the input of the convolutional neural network, and use the target frame image corresponding to the vector group data as the target of the convolutional neural network for training to obtain a deep target frame selection model.

4. The video image processing method based on deep learning according to claim 1, characterized in that: Step S3 specifically includes: S301, marking the human body feature image and the object feature image in the target frame image of each time period; S302, constructing a data set based on the target frame image data of each time period, and dividing it into a training set, a validation set, and a test set; S303, select the yolov5 model, take the target frame image data as the input of the yolov5 model, take the human body feature image and the object feature image as the target of the yolov5 model, perform training, and obtain the target detection model.

5. The video image processing method based on deep learning according to claim 1, characterized in that: The training of the abnormality recognition model in step S4 includes: S401, splicing the human body feature image, the object feature image, and the corresponding information entropy into feature vector data; dividing into a training set, a validation set, and a test set; S402, using the feature vector data of the training set as the input of the convolutional neural network, using the abnormal target frame image as the target of the convolutional neural network, and performing training to obtain the abnormality recognition model.

6. A video image processing device based on deep learning, characterized in that: include: Acquisition module: acquires the surveillance video streams of the same surveillance device at different time periods and decodes them into sequence frame images respectively; Marking a target frame image in the sequence frame images of each time period and selecting the first n frame images and the last n frame images of the target frame image; Deep target frame selection model module: grayscale the 2n+1 frame images at different time periods and calculate the information entropy; use the 2n+1 frame image data at different time periods and the corresponding information entropy to train the deep target frame selection model; Target detection model module: collects human features and object features of each target frame image data and trains the target detection model; Abnormal identification model module: detect the human feature image and object feature image of the target frame image respectively through the target detection model, grayscale them respectively and calculate the information entropy; select the abnormal frame image for which an alarm needs to be issued from all target frame images; use the human feature image, object feature image, information entropy and abnormal frame image to train the abnormal identification model; Execution module: decodes the monitoring video stream data into sequence frame images in real time, selects the target frame through the deep target frame selection model, detects the human feature image and the object feature image through the target detection model, and determines whether an abnormality requires an alarm through the abnormality recognition model.

7. The video image processing device based on deep learning according to claim 6, characterized in that: In the deep target frame selection model module or the anomaly recognition model module, calculating information entropy includes: Feature unit: The pixel grayscale of the frame image and the neighborhood grayscale form a binary feature (i, j), where i represents the grayscale value of the pixel and j represents the neighborhood grayscale; Probability unit: Statistical binary feature (i, j) in the frame image probability P ij ; Entropy calculation unit: calculate information entropy E, 8. The video image processing device based on deep learning according to claim 6, characterized in that: The deep target frame selection model module includes: Splicing unit: splicing frame image data and corresponding information entropy into feature vector data; Division unit: The feature vector data of 2n+1 frame images are used to form vector group data, and the vector groups in different time periods are divided into training set, verification set and test set; Training unit: The vector group data of the training set is used as the input of the convolutional neural network, and the target frame image corresponding to the vector group data is used as the target of the convolutional neural network for training to obtain a deep target frame selection model.

9. The video image processing device based on deep learning according to claim 6, characterized in that: The target detection model module includes: Annotation unit: annotating the human feature images and object feature images in the target frame images of each time period; The second division unit: constructs a data set based on the target frame image data of each time period, and divides it into a training set, a validation set, and a test set; The second training unit: select the yolov5 model, use the target frame image data as the input of the yolov5 model, use the human feature image and the object feature image as the target of the yolov5 model, and train it to obtain the target detection model.

10. The video image processing device based on deep learning according to claim 6, characterized in that: The anomaly recognition model module includes: The third division unit: splices the human feature image, the object feature image, and the corresponding information entropy into feature vector data; divides the data into a training set, a verification set, and a test set; The third training unit: uses the feature vector data of the training set as the input of the convolutional neural network, uses the abnormal target frame image as the target of the convolutional neural network, performs training, and obtains the abnormal recognition model.

Images