A CRNN combined network video privacy protection evaluation method for home scene violent behavior monitoring

Abstract

A CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes, which performs multi-layer dimensionality reduction processing on the input video through compressed sensing to obtain privacy-protected videos; uses a transferred convolutional neural network to extract spatial domain features from the video processed by multi-layer compressed sensing, and uses a gated recurrent unit to process the extracted spatial domain features to obtain spatiotemporal domain features; uses a regression function to obtain the privacy protection score of the video for the spatiotemporal domain features; uses a classification network to obtain the recognition rate of violent behavior in the video to characterize the practicality score of the video; and fits the privacy protection score and practicality score of the multi-layer compressed sensing video to obtain a correlation model between the two. This method can not only adjust the number of compressed sensing layers of the video according to the demand for privacy protection, but also approximately estimate the practicality score of the video when the privacy protection score of the video is known, thereby eliminating data sources that do not meet the performance standards.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes. Background Art

[0002] For middle-aged people working in today's society, they are not only busy with work, but also face the situation of having children and the elderly at home without anyone to take care of them, which has led to an increase in the demand for nanny housekeeping services. As a result, incidents of nanny abuse of children and the elderly have occurred frequently in recent years. By installing cameras to monitor the every move of children, the elderly and the nanny at home in a timely manner, the possibility of abuse can be reduced. However, working and living under surveillance has brought privacy violations to children, the elderly and the nanny at home. Therefore, the privacy protection of videos has become increasingly important, and various video privacy protection methods have emerged. Existing video privacy protection strategy methods can be roughly divided into three categories: the first category is to identify and determine the privacy area and directly replace the privacy area; the second category uses image scrambling and encryption technology to blur the sensitive area information and perform scrambling processing such as blurring; the third category uses watermarks to hide the privacy information of the video or directly erases the privacy information as a watermark embedded in the background after repairing the video.

[0003] While protecting the privacy of the video, it is also necessary to measure the practical value of the video. Otherwise, excessive visual abstraction will make the information in the video meaningless. Therefore, an evaluation method is needed to ensure that the video has privacy protection and practicality after multi-layer compressed sensing. The existing methods have certain limitations and lack a unified evaluation standard. The evaluation methods are mainly divided into two categories. One is to analyze from the perspective of coding security, and the security of the evaluation algorithm is evaluated to evaluate its privacy protection ability. The other is to start from the visual perspective, that is, to consider the amount of visual information that the human eye can know. The privacy protection security of the video image is evaluated through subjective and objective evaluation. Among them, the subjective evaluation index relies on the human visual system (HVS), but as an important indicator of visual security, it requires a large number of experimental personnel and time to evaluate the video information, and is affected by factors such as individual differences, and the results are not reproducible. The existing objective video privacy protection evaluation mainly starts from the strong correlation between the video image quality structure and the video privacy protection, uses the means of evaluating the video image quality to evaluate the video privacy protection score, manually selects the features suitable for the video, and then uses a series of clustering and mapping methods to obtain the final video image privacy protection score. Summary of the invention

[0004] In view of the problems existing in the prior art, the irreversibility brought by the random observation matrix of multi-layer compressed sensing ensures extremely high security. The present invention provides a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes, and proposes to use a combination of CNN and RNN networks to replace the manual acquisition of the temporal and spatial features of the video. The combination of classification and regression is used to achieve end-to-end evaluation of the video privacy protection from the two aspects of practicality and privacy protection.

[0005] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0006] A CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes includes the following steps:

[0007] Step 1: Obtain a home behavior video dataset and perform multi-layer compressed sensing on the video to obtain a video set processed by compressed sensing;

[0008] Step 2: Use the CRNN structure composed of the migrated convolutional neural network Inceptionv3 and the gated recurrent unit GRU to obtain the spatiotemporal features of each video in the dataset obtained in step 1;

[0009] Step 3: Using the subjective privacy protection score of the input video in step 1 and the spatiotemporal features of the video extracted by the network in step 2, regression training is used to obtain a neural network structure model for the video privacy protection score;

[0010] Step 4: Using the subjective privacy protection score of the input video in step 1 and the spatiotemporal features of the video extracted by the network in step 2, classification training is used to obtain a neural network structure model for the detection and recognition rate of violent behavior in the video;

[0011] Step 5: Perform curve fitting on the video privacy protection scores obtained in steps 3 and 4 and the practicality scores of violent behavior detection to obtain an associated statistical model structure, and use the associated statistical model to monitor violent behavior in home scenes based on real-time collected video data.

[0012] Preferably, step 1 further comprises:

[0013] Step 1.1: Using the observation dimension reduction in compressed sensing theory, the observation matrix is ​​used to directly sample the video behaviors in the dataset;

[0014] Step 1.2: Divide the sampled video frame image and the observation matrix into multiple 2×2 matrix blocks in sequence, and then calculate the inner product of the corresponding matrix blocks of the two to achieve CS dimensionality reduction;

[0015] Step 1.3: Use bicubic interpolation to resize the reduced image to 299×299;

[0016] Step 1.4: Repeat the above three steps 1.1-1.3 to achieve multi-layer CS dimension reduction and image size standardization.

[0017] Preferably, step 2 further comprises:

[0018] Step 2.1: Input the standardized multi-layer compressed sensing video as the Inceptionv3 network input, and output N×2048 video spatial domain features. N is the number of video frames contained in a single video, and 2048 is the spatial domain features output by a video frame;

[0019] Step 2.2: Input N×2048 video spatial domain features into GRU gating, fuse N 2048 spatial domain features in the time domain, and finally output a 1×2048 spatiotemporal feature vector.

[0020] Preferably, step 3 further comprises:

[0021] Step 3.1: Use the normalization formula to normalize the subjective video privacy protection score input in step 1, and give the normalized label value and training set video to the neural network training;

[0022]

[0023] Where X is the subjective privacy protection score, X min is the minimum subjective privacy protection score in the entire video set, X max The maximum subjective privacy protection score in the entire video set;

[0024] Step 3.2: After obtaining the spatiotemporal features of the video, concatenate the fully connected layers to reduce the dimensionality of the 1×2048 feature vectors into 1 feature vector;

[0025] Step 3.3: Use the sigmoid function to transform the feature vector into the video privacy protection score, where the sigmoid function is:

[0026]

[0027] Step 3.4: Denormalize the normalized video privacy protection score to obtain the final video privacy protection score.

[0028] Preferably, step 4 further comprises:

[0029] Step 4.1: In step 1, the behavior category labels in the video set are fed into the network along with the training videos;

[0030] Step 4.2: After obtaining the spatiotemporal features of the video, concatenate the fully connected layers to reduce the 1×2048 feature vectors into M feature vectors; where M is the number of behavior categories in the video set;

[0031] Step 4.3: Convert the M feature vectors into M probability values ​​through the SoftMax function, and convert the largest probability value into a category through Classification;

[0032] Step 4.4: After the test set in step 1 passes through the above steps, the accuracy of the category label obtained compared with the actual category label is the practicality score of the video.

[0033] Preferably, step 5 further comprises:

[0034] Linear fitting, Gaussian fitting, exponential fitting and polynomial fitting methods are used to perform statistical modeling on the association between the video privacy protection score and the video practicality score (the recognition rate of violent behavior in the video).

[0035] Preferably, the best fitting curve is selected according to three evaluation indicators of the fitting sum of square errors SSE, determination coefficient R-square and root mean square RMSE, wherein when SSE and RMSE are close to 0, and R-square is close to 1, the fitting effect is better.

[0036] The beneficial effects of the present invention are:

[0037] 1. The present invention combines bicubic interpolation with multi-layer compressed sensing technology to standardize the frame image whose size is reduced due to compressed sensing technology to the size required for neural network input, while reducing the mosaic phenomenon caused by image enlargement, which causes the image features to be unable to be better learned by the neural network.

[0038] 2. The present invention combines CNN neural network with RNN network, and uses the migrated inceptionv3 network to greatly shorten the training time. At the same time, it replaces the traditional manual extraction of video spatiotemporal features with GRU gated splicing.

[0039] 3. The present invention evaluates the video privacy protection method from two perspectives: the video privacy protection score and the practicality score of violent behavior recognition in a home environment.

[0040] 4. The present invention associates the video privacy protection score and the practicality score with statistical modeling, providing guidance for its practical application. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the method flow in an embodiment of the present invention.

[0042] Figure 2This is a visual schematic diagram of multi-layer compressed sensing and normalized video images in an embodiment of the present invention.

[0043] Figure 3 Schematic diagram of the CRNN structure in an embodiment of the present invention.

[0044] Figure 4 Graph showing video privacy protection score predictions for compressed sensing videos at different layers in an embodiment of the present invention.

[0045] Figure 5 It is the recognition rate (video practicality score) of violent behavior detection in a home environment of compressed sensing videos of different layers in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0047] The present invention provides a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes. The method evaluates the privacy protection method of multi-layer compressed sensing videos from two aspects: video privacy protection score and video practicality score. By performing correlation statistical modeling on the video privacy protection score and the video practicality score represented by the violent behavior recognition rate, guidance is provided for its practical application.

[0048] like Figure 1 As shown, the present invention is a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes. The specific steps are:

[0049] Step 1: Obtain a home behavior video dataset, perform multi-layer compressed sensing on the video and normalize its size to obtain a video set processed by compressed sensing.

[0050] Step 1.1: This example selects the HockeyFights dataset, which consists of clips from the National Hockey League. The dataset contains 1,000 videos, 500 of which contain violent behavior and 500 of which contain non-violent behavior. The video resolution is 360×228, the duration is about 1.6 seconds, and the frame rate is 25 frames per second. For each of the 25 frames of a single video, the same random observation matrix is ​​used for compressed sensing.

[0051] Step 1.2: Use the bicubic interpolation method to normalize the image size of the compressed sensed video to the standard size of 299×299 for the transferred neural network input. In the bicubic method, the value of the function f at the pixel point B coordinate (x, y) can be obtained by the weighted average of the nearest sixteen pixels in the rectangular grid. Construct the BiCubic function as shown in formula (1) to solve the weight W(x) corresponding to x to 16 sampling points, where the calculation coefficient a is -0.5. Because the BiCubic basis function is one-dimensional, we need to calculate the rows and columns of the pixel points separately twice to obtain the value of the pixel point coordinate B(x, y) as shown in formula (2), where a ij To calculate the coefficients, the pixel point (x, y) to be interpolated is taken from the 4×4 neighborhood points (x i ,y i ),i=0,1,2,3. The interpolation calculation formula is (3):

[0052]

[0053]

[0054]

[0055] Step 1.3: Repeat the above three steps 1.1-1.2 to achieve multi-layer CS dimensionality reduction and image size standardization.

[0056] Step 1.4: After steps 1.1-1.3, the original HockeyFights dataset is compressed and normalized by 1-5 layers, as shown in Figure 2 As shown in the figure, 6,000 videos with standardized sizes of the original videos are generated. 20 professional researchers are asked to subjectively judge the visual privacy protection score of each video in this dataset, with a range of 0-5. The higher the grade, the better the visual privacy protection of the video. The average value is then taken as the final subjective video privacy protection score. This forms the HockeyFights video privacy protection score dataset. According to the behavioral classification of each video in the original HockeyFights dataset, the generated 6,000 videos are annotated to form the HockeyFights video category dataset.

[0057] Step 2: Use the CRNN structure of the migrated Inceptionv3 network and GRU gating combination to obtain the spatiotemporal features of each video in the dataset obtained in step 1. The CRNN structure is as follows: Figure 3 shown.

[0058] Step 2.1: This example chooses to migrate the Inceptionv3 network to obtain the spatial features of each frame in the video. By decomposing convolution and regularization, efficient calculation is achieved with a lower error rate. At the same time, the network has been trained using ImageNet, with strong generalization ability and better robustness, which improves the training accuracy of the designed CRNN network structure and avoids the training from scratch, saving time. The size of the network input image is 299×299. Expand each video in step 1 frame by frame and send it to the network, and the output two-dimensional feature vector size is: N×2048, where N is the number of frames in each video.

[0059] Step 2.2: After splicing the GRU gated recurrent unit with Inceptionv3, it is easier to calculate than LSTM, which greatly improves the training efficiency. The spatial domain salient features extracted by CNN are used as the input of the RNN layer. The GRU network in the RNN layer not only considers the input features of the current time t, but also the time sequence information before and after the current time t, and finally outputs the spatial and temporal features of the video. The experiment sets the GRU network to output at the last time node after learning all the features of the video sequence. The expression of GRU is:

[0060] r t =σ(W r ·[h t-1 ,x t ]) (4)

[0061] z t =σ(W z ·[h t-1 ,x t ]) (5)

[0062]

[0063]

[0064] Where: σ is the sigmoid function, which is used to transform the data into a value in the range of 0-1. t is the current input, z t and r t The update gate and reset gate of GRU are respectively. The update gate is used to control the extent to which the state information of the previous moment is brought into the current state. The larger the value of the update gate, the more state information of the previous moment is brought into the current state. The reset gate is used to control the extent to which the state information of the previous moment is ignored. The smaller the value of the reset gate, the more it is ignored. W is the weight of the output of the feature vector of the previous moment and the input of the current feature vector through each control gate, where W z To update the weight of gate z, W r is to reset the weight of gate r. It mainly contains the x input at the current moment t Targeted Add to the current hidden state. This is equivalent to memorizing the current state. GRU can use the same gate Z to forget and select memory at the same time. (1-z t )*h t-1 Indicates the selective forgetting of the original hidden state. t-1 Some unimportant information in the dimension. Indicates the current node information Do some selective memorization.

[0065] Step 3: Use the HockeyFights video privacy protection score dataset generated in step 1 and the spatiotemporal features of the video extracted by the network in step 2 to obtain a neural network structure model for the video privacy protection score using regression training.

[0066] Step 3.1: Use the normalization formula to accelerate the convergence of the subjective video privacy protection score input in step 1, and randomly divide each layer in the video set into training set, validation set, and test set according to 6:2:2. Submit the normalized label value and the training set video to the neural network for training.

[0067]

[0068] The normalized video privacy protection score formula is (8), X is the subjective privacy protection score, X min is the minimum subjective privacy protection score in the entire video set, X max is the maximum subjective privacy protection score in the entire video set.

[0069] Step 3.2: After obtaining the spatiotemporal features of the video, concatenate the fully connected layers to reduce the dimensionality of the 1×2048 feature vectors into 1 feature vector.

[0070] Step 3.3: Use the sigmoid function as the activation function to transform the feature vector into the video privacy protection score, and obtain the normalized privacy protection prediction score VPP' of the video. pre . The sigmoid function is formula (9):

[0071]

[0072] Step 3.4: VPP' pre Denormalize to get the final video privacy protection score VPP pre .like Figure 4 As shown in the figure, it can be seen from the evaluation results that when the number of compression layers reaches 4, the video privacy protection score is around 4 or above, which meets the requirements of video privacy protection.

[0073] Step 4: Use the HockeyFights video category dataset generated in step 1 and the spatiotemporal features of the video extracted by the network in step 2 to obtain a neural network structure model for the detection and recognition rate of violent behavior in the video using classification training.

[0074] Step 4.1: Randomly divide each layer of the video set into training set, validation set, and test set according to the ratio of 6:2:2. Send the training set and category labels of each layer of videos into the network to train the video practicality score model for different layers. The practicality score is represented by the recognition rate of violent behavior.

[0075] Step 4.2: After obtaining the spatiotemporal features of the video, concatenate the fully connected layers to reduce the 1×2048 feature vectors into M feature vectors; where M is the number of behavior categories in the video set.

[0076] Step 4.3: Use the SoftMax layer after the fully connected layer and concatenate a classifier. The SoftMax function can convert the output values ​​of multiple categories into a probability distribution in the range of [0, 1] and 1. In this experiment, the classifier is truncated to change the final output to the maximum probability of the SoftMax layer output. The SoftMax function formula is formula (10):

[0077]

[0078] Where 0<y r <1, where a r (x) is the output value of the rth node, k is the number of output nodes, that is, the number of classification categories. r (x) is the probability value of the category output.

[0079] Step 4.4: The accuracy of the category labels obtained after the test set passes the above steps and the actual category labels is the practicality score of the video layer. The results are as follows Figure 5 As shown, when the number of compression layers is less than 4 (including 4 layers), the video as a whole maintains a high recognition rate of violent behavior. When compressed to the 5th layer, the recognition rate drops sharply and the practicality of the video decreases.

[0080] Step 5: Perform curve fitting on the video privacy protection scores obtained in steps 3 and 4 and the practicality scores of violent behavior detection to obtain the associated statistical model structure.

[0081] In order to further quantify the correlation between the visual privacy protection score and the domestic violence behavior recognition results, and to more intuitively evaluate the multi-layer compressed sensing video privacy protection method, the video privacy protection score and the practicality score are curve fitted. The fitting methods selected in this embodiment are: linear fitting, Gaussian fitting, exponential fitting and polynomial fitting.

[0082] The best fitting curve is selected according to the three evaluation indicators of fitting: sum of squared errors (SSE), coefficient of determination (R-square) and root mean square (RMSE), wherein the fitting effect is better when SSE and RMSE are close to 0 and R-square is close to 1. In this embodiment, the present invention selects a polynomial function from various fitting results as a statistical model for the association between visual privacy protection and recognition rate.

[0083] The following analysis and application guidance can be obtained from the association statistical model: When more than four layers of compressed sensing are used, that is, when the visual privacy protection score is greater than 4, the recognition rate of domestic violence begins to show a rapid downward trend. When the compressed sensing video has five layers, the recognition rate drops to a too low level and cannot meet the needs of intelligent applications; in addition, when the number of compressed sensing layers is less than four, the visual privacy protection score is lower than 4, which does not meet the needs of visual privacy protection. Therefore, a more appropriate fourth layer is selected as the multi-layer compressed sensing practicality layer of the video set to achieve a balance between the visual privacy protection score and practicality. At the same time, the association model can not only effectively find the appropriate number of compression layers for the data set, but also estimate the approximate practicality score when the video privacy protection score is known, thereby eliminating data that does not meet the performance requirements.

[0084] The association model obtained by the present invention intuitively demonstrates the practical scope and degree of video privacy protection of multi-layer compressed sensing video privacy protection technology from the perspectives of practicality and privacy protection. At the same time, the approximate recognition rate can be estimated based on the video privacy protection score of the new data set, thereby eliminating data sources that do not meet the standards.

[0085] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes, characterized by: The method comprises the following steps: Step 1: Obtain a home behavior video dataset and perform multi-layer compressed sensing on the video to obtain a video set processed by compressed sensing; Step 2: Use the CRNN structure composed of the migrated convolutional neural network Inceptionv3 and the gated recurrent unit GRU to obtain the spatiotemporal features of each video in the dataset obtained in step 1; Step 3: Using the subjective privacy protection score of the input video in step 1 and the spatiotemporal features of the video extracted by the network in step 2, regression training is used to obtain a neural network structure model for the video privacy protection score; The privacy protection score training model process of the multi-layer compressed sensing video in step 3 is as follows: Step 3.1: Use the normalization formula to normalize the subjective video privacy protection score input in step 1, and give the normalized label value and training set video to the neural network training; Where X is the subjective privacy protection score, X min is the minimum subjective privacy protection score in the entire video set, X max The maximum subjective privacy protection score in the entire video set; Step 3.2: After obtaining the spatiotemporal features of the video, concatenate the fully connected layers to reduce the 1×2048 feature vectors into 1 feature vector; Step 3.3: Use the sigmoid function to transform the feature vector into the video privacy protection score, and obtain the normalized privacy protection prediction score of the video. The sigmoid function is: Step 3.4: Denormalize the normalized video privacy protection score to obtain the final video privacy protection score; Step 4: Using the subjective privacy protection score of the input video in step 1 and the spatiotemporal features of the video extracted by the network in step 2, classification training is used to obtain a neural network structure model for the detection and recognition rate of violent behavior in the video; Step 5: Perform curve fitting on the video privacy protection scores obtained in steps 3 and 4 and the practicality scores of violent behavior detection to obtain an associated statistical model structure, and use the associated statistical model to monitor violent behavior in home scenes based on real-time collected video data.

2. According to claim 1, a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes is characterized by: Step 1: Acquisition of multi-layer compressed sensing video sets includes the following steps: Step 1.1: Using the observation dimension reduction in compressed sensing theory, the observation matrix is ​​used to directly sample the video behaviors in the dataset; Step 1.2: Divide the sampled video frame image and the observation matrix into multiple 2×2 matrix blocks in sequence, and then calculate the inner product of the corresponding matrix blocks of the two to achieve CS dimensionality reduction; Step 1.3: Use bicubic interpolation to resize the reduced image to 299×299; Step 1.4: Repeat the above three steps 1.1-1.3 to achieve multi-layer CS dimension reduction and image size standardization.

3. According to claim 1, a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes is characterized by: The spatiotemporal feature extraction process of the multi-layer compressed sensing video in step 2 is as follows: Step 2.1: Input the standardized multi-layer compressed sensing video as the Inceptionv3 network input, and output N×2048 video spatial domain features; where N is the number of video frames contained in a single video, and 2048 is the spatial domain features output by a video frame; Step 2.2: Input N×2048 video spatial domain features into GRU gating, fuse N 2048 spatial domain features in the time domain, and finally output a 1×2048 spatiotemporal feature vector.

4. According to claim 1, a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes is characterized by: The process of the violent behavior recognition model of the multi-layer compressed sensing video in step 3 is as follows: Step 4.1: In step 1, the behavior category labels in the video set are fed into the network along with the training videos; Step 4.2: After obtaining the spatiotemporal features of the video, concatenate the fully connected layers to reduce the 1×2048 feature vectors into M feature vectors; where M is the number of behavior categories in the video set; Step 4.3: Convert the M feature vectors into M probability values ​​through the SoftMax function, and convert the largest probability value into a category through Classification; Step 4.4: After the test set in step 1 passes through the above steps, the accuracy of the category label obtained compared with the actual category label is the practicality score of the video.

5. According to claim 1, a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes is characterized by: The fitting in step 5 is linear fitting, Gaussian fitting, exponential fitting or polynomial fitting.

6. According to claim 5, a CRNN combined network video privacy protection evaluation method for monitoring violent behavior in home scenes is characterized by: The best fitting curve is selected based on the three evaluation indicators of the sum of squared errors (SSE), the coefficient of determination (R-square) and the root mean square (RMSE). When SSE and RMSE are close to 0 and R-square is close to 1, the fitting effect is better.

Images