Encrypted image blind vision safety evaluation method based on depth feature extractor

Abstract

The invention provides an encrypted image blind vision safety evaluation method based on a depth feature extractor. The visual security evaluation not only reflects the degree of information leakage of the encrypted image, but also can further promote the improvement of the encryption algorithm performance. The invention provides a blind vision safety evaluation method on the premise of lacking reference original image information. Specifically, the problems that an existing encrypted image data set is limited, and calculation time based on a deep learning method is long are considered. On the premise that the performance is guaranteed, a VGG19 network trained by ImageNet is used as a deep feature extractor in a transfer learning mode, and the deep feature extractor is used for extracting semantic features of an encrypted image. Meanwhile, a parallel double-flow prediction network is designed to realize mapping from the features to the quality evaluation score and the visual safety evaluation score of the encrypted image. Experimental results show that the blind vision safety evaluation method is superior to an existing blind vision safety evaluation method in performance and calculation speed.

Description

Technical Field

[0001] The present invention relates to the technical problem of blind visual security evaluation of encrypted images in the field of information security. In particular, an existing network model is used as a deep feature extractor (DEF) through transfer learning, and then a parallel dual-stream prediction network (PDSP-Net) is designed to achieve blind visual security evaluation of encrypted images. Background Art

[0002] Images, as a vivid multimedia information, are widely used in the digital multimedia era due to their convenience and visualization characteristics. As the scope of image use gradually expands, the information they contain is particularly important for individuals, society and the country. However, due to the vulnerability to theft and tampering by criminals and organizations during transmission. Therefore, the issue of image security has attracted extensive attention from researchers, enterprises and countries, and also reflects the general awareness of the protection of these valuable multimedia contents.

[0003] After long-term exploration and research by experts in the field of information security, a variety of multimedia information security technologies have been proposed, including digital image watermarking, information hiding, image forensics, image encryption, etc. These solutions have different effects on the image content. Digital image watermarking achieves copyright authentication by embedding identity marker information in the original image, etc.; information hiding mainly hides important information in the host image to achieve the purpose of hidden information transmission; image forensics mainly extracts effective information from the existing image information. The above solutions all use images to transmit and extract secret information, and only the image encryption technology provides the most direct confidentiality protection for the image itself. Encryption is one of the simplest and most widely used methods, which can not only effectively protect image information, but also maintain the transparency and integrity of image information. However, the encryption operation on the image may cause image distortion. Therefore, in order to further evaluate the performance of the encryption algorithm and the image quality, it is crucial to conduct a visual security evaluation of encrypted images.

[0004] The ultimate perceiver of encrypted images is the human visual perception system. Therefore, subjective evaluation is accurate and reliable. However, subjective evaluation has great limitations in actual application scenarios. Therefore, there is an urgent need to design an objective visual security evaluation method focusing on encrypted images. Visual security assessment / evaluation (VSA / VSE) aims to analyze the security of selectively encrypted images from the perspective of the human visual system (HVS) and measure the amount of meaningful information captured by the human visual system from encrypted images.

[0005] According to whether it involves the information of the original reference image, the current objective visual security evaluation methods are mainly divided into reference-based visual security evaluation and non-reference / blind visual security evaluation. However, in real life, the original reference image (perfect image) does not exist. Therefore, non-reference visual security evaluation has more practical research significance and value. Due to the great difficulty of non-reference visual security evaluation methods, there are relatively few studies on non-reference visual security evaluation methods. Currently, the biggest challenge in blind visual security evaluation research is to select appropriate features from encrypted images. However, due to the low visual quality of encrypted images, most of the understandable visual information may be lost. Therefore, it is not easy to select manual features that fit the human visual system from images. In addition, there are significant differences in quality and information between the original reference image and the encrypted image, and there are also great differences in feature extraction.

[0006] With the wide application of deep learning and its excellent performance in the field of image / video quality evaluation, it is feasible to introduce deep learning into the blind visual security evaluation algorithm of encrypted images. However, since deep learning requires a large number of training samples, and the quantity and data volume contained in existing encrypted image databases cannot meet the large-scale training data required by deep learning. Therefore, in order to improve the performance of the evaluation method based on deep learning, it is necessary to generate more training samples in various ways. One of the most popular methods currently is to divide the image into image patches and use the subjective quality score of the entire image or the quality score obtained by the full-reference method as the subjective score of each image patch. However, this method cannot accurately represent the true subjective score of the image patch. At the same time, due to the large number of network parameters and complex network layers of the evaluation method based on deep learning, it requires a long calculation time. Although lightweight networks can be used to speed up the calculation, the reduction of network model complexity may lead to a decline in calculation performance, and a large amount of sample data is still required to train network parameters. Summary of the Invention

[0007] In view of the problem that there is still room for improvement in the performance of the blind visual security evaluation method in the case of a small existing database dataset and no reference image, and the calculation accuracy and speed can be further improved, the present invention aims to propose a blind visual security evaluation method based on a deep feature extractor.

[0008] The basic idea of the present invention is to use the feature extraction part of the existing trained network model as a deep feature extractor, to solve the problem of the small amount of data in the existing encrypted image database and the calculation speed problem of the encrypted image method based on deep learning, and to propose a blind visual security evaluation method based on a deep feature extractor. This method mainly includes three steps, namely:

[0009] Step 1: At the beginning of the training process, in order to adapt to the contrast masking process in human visual perception and highlight the individual differences of image information, local mean removal processing is performed on the input encrypted image. At the same time, in order to ensure the stable convergence of model training, we perform normalization operations on the image and adjust it to a similar range;

[0010] Step 2: After completing the data preprocessing, in order to extract features from the encrypted image, load the network model parameters for feature extraction of the pre-trained model, which is called a deep feature extractor. The main reason for using this method is that the amount of data in the existing encrypted image database is small and it is difficult to meet the parameter training requirements of large network models. In addition, the underlying layer of the network model extracts low-level semantic features through convolution operations, and these features remain unchanged in different tasks. Therefore, we can use the deep feature extractor to extract the visual semantic feature map of the encrypted image, and then design a parallel two-stream prediction network, including two sub-networks, which are respectively used to implement the quality evaluation and visual security evaluation of the encrypted image;

[0011] Step 3: Take the encrypted image as the input, and successively pass through the preprocessing in Step 1, the deep feature extractor in Step 2, and the trained parallel two-stream prediction network, and finally obtain the quality score and visual security score of the encrypted image;

[0012] The specific process is as follows:

[0013] (1) The overall framework diagram of the present invention is as shown in the appendix Figure 1 First, preprocess the input encrypted image data to meet the subsequent encrypted image operations. Mainly to adapt to the contrast masking process in human visual perception and ensure the convergence of the model, perform local mean removal processing and normalization operations on the input encrypted image;

[0014] (2) In this invention, in order to extract the features of encrypted images, the feature extraction part of the existing network model VGG19 is used as a deep feature extractor by means of transfer learning to extract the feature maps of encrypted images. Its structural information is as shown in the appendix Figure 2 It mainly includes 16 convolutional layers; then a parallel dual-stream prediction network is designed to realize the prediction of features to quality scores and visual security scores. PDSP-Net consists of two subtasks. Sub-network-I predicts the quality score in the form of visual semantic feature mapping by extracting image quality features. Sub-network-II analyzes the statistical features of encrypted images in the form of visual semantic feature mapping and takes the image quality features of sub-network-I as part of the features, aiming to predict the visual security score of encrypted images;

[0015] In process (1), first, the input encrypted image is preprocessed. First, in order to adapt to the contrast masking process in human visual perception and highlight the individual differences of image information, we perform local mean removal on the input encrypted image E:

[0016]

[0017] where \(i\in1,2,\cdots,M\), \(j\in1,2,\cdots,N\), and \(M\) and \(N\) respectively represent the length and width of the encrypted image. \(\mu(\cdot)\) is the average value of the encrypted image.

[0018] To ensure the stable convergence of model training, we perform normalization on the image and adjust it to a similar range:

[0019]

[0020] where \(min(\cdot)\) and \(max(\cdot)\) respectively represent the maximum and minimum values of image pixels.

[0021] In process (2), after completing data preprocessing, in order to extract features from encrypted images, we load the network model parameters for feature extraction of the pre-trained model, which is called a deep feature extractor. The main reason for adopting this method is that the amount of data in the existing encrypted image database is small and it is difficult to meet the parameter training requirements of large network models. In addition, the underlying layer of the network model extracts low-level semantic features through convolutional operations, and these features remain unchanged in different tasks. Therefore, we can use the pre-trained network model to extract the visual semantic feature maps of encrypted images, that is, VSFea_map.

[0022]

[0023] where DFE represents the deep feature extractor that loads the VGG network, and its architecture of DFE is as shown in Figure 2As shown, there are a total of 16 convolutional layers. VSFea_map is a feature map obtained by layer-by-layer mapping of the encrypted image through the multi-layer convolutional layers of the deep feature extractor. These mappings learn feature patterns from simple to complex, and finally 512 VSFea_maps are obtained. Using DEF as the deep feature extractor not only greatly reduces the number and complexity of network model parameters, but also improves the computational efficiency. During the entire experiment, DFE selected the VGG19 network model. It is also possible to load the feature extraction part of other network model architectures as DFE.

[0024] To predict the quality score and visual security score of the encrypted image, we designed a parallel two-stream prediction network, as shown in the appendix. Figure 3 As shown. This network consists of two sub-networks: Sub-network I maps the feature vector to a quality evaluation score, and Sub-network II maps the feature vector to a visual security evaluation score. On the one hand, these two tasks are related because the image quality score reflects the visual subjective clarity, and the image clarity also affects the visual security of the image. On the other hand, these two tasks are also different because the amount of leaked information of encrypted images with the same quality score may be different, that is, they have different visual security.

[0025] Subtask-I: This network uses a traditional model structure. After extracting the main features using principal component analysis, feature fusion is used to fuse different features to obtain a column vector of size 1×4096. Then four fully connected layers are connected, and each fully connected layer contains 4096, 2048, 1024, and 512 nodes respectively. The last layer is a linear prediction that outputs the quality evaluation score of the image.

[0026] According to the operation of the traditional network model, the feature map is flattened into a row vector of size 4096 through the fully connected layer. In the traditional network architecture, the number of parameters generated by the first fully connected layer accounts for 71.52% of the number of parameters of the entire VGG19 network model. Therefore, a large amount of computing time will also be consumed here.

[0027] In further research, we found that the focus is mainly on the effective features of VSFea_map. Since there is a large amount of redundant data in VSFea_map, we can reduce the amount of data by removing redundant information to speed up the calculation. To remove the redundant information of the feature map and retain the main information of the feature map to the greatest extent, we use principal component analysis to perform eigenvalue decomposition on the covariance matrix of the feature map, where the feature map covariance matrix:

[0028]

[0029] where \(i\in(1,2,\ldots,512)\) represents the number of VSFea_map, and VSFea_map i represents the covariance matrix of VSFea_map i .

[0030] Then, perform eigen - decomposition on the covariance matrix of the feature map VSFea i :

[0031] (V, D)=\(Eig\)(VSFea i ) (5) where \(Eig(\cdot)\) is eigen - decomposition, V is the eigenvalue, and D is the eigenvector. The eigenvector is represented as V=(v1, v2, …, v L ), and the eigenvalue is represented as D = \(diag\)(d1, d2, …, d L ), where L is the number of principal components.

[0032] When L = 8, the performance reaches the best. Use the principal component analysis method to extract the main features of 512 VSFea_map, and 4096 features are obtained. We collect all the main features of the feature mapping to obtain a column vector of size 4096 as the final feature of the image. This implementation converts 512 VSFea_maps into a column vector of size 4096. Finally, use four fully - connected layers to map the features to the quality score.

[0033] Subtask - II: This network is designed to calculate the visual security score of the image. Since the local details of the encrypted image may leak information, based on this, we perform texture analysis on the feature map VSFea_map obtained by the deep feature extractor. VSFea_map represents the local feature map of the ciphertext image, and the Local Binary Patterns (LBP), compared with other local feature operators (such as Scale - Invariant Feature Transform, Weber Local Descriptor, etc.), has great advantages in terms of computational complexity, feature dimension, etc.

[0034] As shown in Appendix Figure 4 (a), the traditional LBP pattern selects a neighborhood M of size \(n\times n\), and compares the pixel value of the neighborhood center M c with the surrounding pixels M j . If the surrounding pixel is greater than or equal to the center pixel value, it is encoded as 1, otherwise it is encoded as 0. Arrange the comparison results in a clockwise direction into a binary value of length \(n 2 - 1\), and this binary value is the value obtained by LBP. Its calculation method is as follows:

[0035]

[0036] where f(·) represents the activation function, greater than or equal to M c is set to 1, otherwise set to 0.

[0037] However, using the traditional LBP pattern will result in an n×n neighborhood M, and finally the encoding becomes 2 n2-1 types of results. Since VSFea_map has a relatively small scale, this method will increase the feature dimension. To reduce the feature dimension, Ojala et al. designed a rotation-invariant equivalent LBP pattern, regarding the binary encoding as a ring, traversing one circle and calculating the number of jumps between 0 and 1, as shown in Appendix Figure 4 (b). Its calculation method is as follows:

[0038]

[0039] where |f(M n-1 -M c )-f(M0 - M c )| represents the judgment of the head and tail of the binary encoding.

[0040] Ojala et al. found through statistical research that in actual images, the number of jump transformations of the vast majority of LBP patterns is less than or equal to 2, which is called the equivalent pattern, otherwise it is set to the mixed pattern, and the calculation method is as follows:

[0041]

[0042] where the calculation process of J_m i is shown in formula (7).

[0043] In this way, for each pixel in VSFea_map, an n×n region M is selected, and then 10 kinds of feature information are obtained through the rotation-invariant equivalent LBP pattern. The number of 1s in each kind is different, so the LBP value can be simply calculated by counting the number of occurrences of 1 in the rotation-invariant equivalent LBP pattern. During the experiment, n = 3, and this feature is not limited to the neighborhood of n = 3 and can also be extended to a higher neighborhood range.

[0044] The histogram is a statistical feature of an image and also a method for traditional cryptographic security analysis. Finally, the histogram statistical information is used to count the LBP values of VSFea_map. Each VSFea_map obtains 10 features, and there are a total of 512 VSFea_maps, obtaining 5120 features. Finally, the 512 features of the quality evaluation features are fused to predict the visual security evaluation score;

[0045] (3) In the training stage of the present invention, the input encrypted image is first preprocessed to complete the operations of mean removal and normalization. Subsequently, the trained deep feature extractor is used to extract the feature map, and a parallel two-stream prediction network is designed to predict the quality score and visual security score of the encrypted image.

[0046] Advantages and beneficial technical effects of the present invention compared with the prior art:

[0047] (1) The present invention proposes a blind visual security evaluation method based on a deep feature extractor through transfer learning. By adopting the method of transfer learning, the need for a large amount of training data is avoided, which not only improves the blind visual security evaluation performance of encrypted images but also speeds up the calculation speed.

[0048] (2) For the evaluation of encrypted images, the present invention not only considers the quality of the encrypted image but also emphasizes the evaluation of its visual security. Therefore, the present invention proposes a parallel two-stream network model structure that can simultaneously achieve the quality evaluation and visual security analysis of encrypted images.

[0049] (3) In the parallel two-stream prediction network, the two sub-network tasks are mutually related, where the quality affects visual security. However, they are also independent of each other, and the quality evaluation method cannot be directly applied to visual security evaluation simply. By implementing feature sharing between sub-network layers, the number of parameters and complexity of the network model are further reduced.

[0050] (4) The present invention proposes a blind visual security evaluation method for encrypted images based on a deep feature extractor. Compared with the existing quality evaluation and visual security evaluation rating methods, the method proposed by us has significant advantages in terms of performance and calculation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is the overall framework diagram of the proposed method.

[0052] Figure 2 It is the architecture of the deep feature extractor VGG19.

[0053] Figure 3 It is the structure of the parallel two-stream prediction network PDSP-Net.

[0054] Figure 4 It is the calculation process of LBP. (a) The calculation process of traditional LBP, (b) The calculation process of rotation-invariant equivalent LBP. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The present invention will be further described in detail below in conjunction with embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the protection scope of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above invention content and still fall within the protection scope of the present invention.

[0056] (1) The method proposed in the present invention was verified experimentally on four databases: the IVC and PEID databases;

[0057] (2) To evaluate the performance of the algorithm, we used the commonly used Pearson linear correlation coefficient (PLCC), Spearman rank correlation coefficient (SRCC), Kendall rank correlation coefficient (KRCC), and root mean square error (RMSE) as evaluation criteria. PLCC represents the correlation between MOS and the predicted score, and SROCC and KRCC represent the monotonicity between the two quantities of MOS and the predicted score. RMSE is used to evaluate the consistency of the evaluation algorithm prediction. Among them, the closer the absolute values of PLCC, SRCC, and KRCC are to 1, and the closer the value of RMSE is to 0, the better the linear rank correlation of the model and the higher the accuracy;

[0058] (3) The method of the present invention was used to calculate the performance under the above conditions;

[0059] (4) Table 1 shows the comparison of the evaluation methods of the present invention on the IVC and PEID databases. The comparison methods include FR-IQA: SSIM; RVSA methods: ESS, LSS, LFBVS, VSI-Canny, IIBVSI, CNNVSI; BIQA methods: BRISQUE and NIQE; BVSA: QEPE. To ensure the fairness of the experiment, the experimental process was carried out under the same settings. Table 1 lists the results of existing IQA methods and VSA methods in PLCC, SRCC, KRCC, and RMSE. As shown in Table 1, the proposed TL-BVSA method of the present invention is superior to the existing BVSA method QEPE in the encrypted image database. Compared with the BIQA method, the present invention shows advantages in the encrypted database, which also indicates that it is not appropriate to directly use the BIQA method in encrypted images. In addition, compared with the FR-IQA and RVSA methods, the present invention also shows excellent performance;

[0060] (5) Table 2 shows the encryption image performance of different encryption algorithms of the present invention in the PEID database. To further study the robustness of the present invention, the SRCC results of encrypted images of different methods with different encryption algorithms on the PEID database were tested. As shown in Table 2, compared with the blind visual security evaluation method, the present invention ranks first in six of the ten encryption algorithms in the PEID database. It is worth noting that among these ten encryption algorithms, most methods perform poorly in two encryption types, Enc08 and Enc09. These two encryption types are Arnold's Cat Map (Enc08) and Chaos Pseudorandom Generator (Enc09). Since these two encryption methods completely change the content information structure and position of the image, the encrypted image information has lost the original image information structure. The existing full-reference visual security evaluation method calculates the similarity of corresponding information between the original image and the encrypted image. Using this full-reference evaluation method, more accurate results cannot be obtained. Although QEFE is a blind visual security evaluation method, the statistical features of the image extracted by manual production cannot reflect the features of the encrypted image. The present invention uses the method of transfer learning to extract deeper features of the image, and for the visual security evaluation of encrypted images, not only uses rotation-invariant LBP to extract the local features of the ciphertext, but also fuses the image quality evaluation features, which can better realize the visual security evaluation of encrypted images;

[0061] (6) Table 3 shows the influence of training data on the results in the present invention. Since DFE is trained by an existing network model, no additional training data is required. At the same time, to verify the effectiveness of the model of the present invention under the premise of limited training data set, we divided the PEID data set into a training set and a test set, and conducted experiments on different proportions of the training set and the test set. The training data set was reduced from 90% to 10%, each time reducing by 10%, and the corresponding test set increased from 10% to 90%, each time increasing by 10%. Each result was executed 10 times on the data set, and the median of the final PLCC, SRCC, KRCC, and RMSE results was used as the final result. As shown in Table 3, when the training data set increased from 60% to 90%, the results of PLCC, SRCC, and KRCC only increased by 0.024, 0.043, and 0.025 respectively, while the RMSE result decreased by 0.204. Therefore, TL-BVSA can also achieve good results when the number of training data sets is small. Further verifying the effectiveness of TL-BVSA shows that the requirements for training data are not high on the premise of ensuring performance;

[0062] (7)A series of ablation experiments were carried out on Table 4 to evaluate the contribution of key components to visual security evaluation in this invention. PDSP-Net consists of two subtasks: Subtask-I is for quality evaluation, mainly including PCA operation and four fully connected layers. In the design process of Subtask-II, not only the quality evaluation features of Subtask-I are used as part of the VSA features, but the texture features of each feature map are extracted by using LBP, and the histogram statistical results of each feature map are used as its supplementary features. Finally, these two features are fused to achieve BVSA of encrypted images. In order to evaluate the rationality and effectiveness of the VSA task design in this invention, three groups of ablation experiments were carried out on BVSA, as shown in Table 4. The first group of experiments only contains the image quality evaluation features of Subtask-I, which represent the depth features of encrypted images. The second group of experiments only contains LBP features, which represent the manually extracted features of encrypted images. The third group of experiments is the fusion of the two groups of features. As shown in Table 4, the results of the first group of experiments are significantly better than those of the second group, which also reflects that the manually made features are less representative than the features based on the depth feature extractor. The combination of these two methods can not only solve the problem of insufficient data sets by means of the depth feature extractor, but also improve the prediction accuracy by combining local texture statistical features. Therefore, when the two groups of feature items are fused, the visual security evaluation effect of the third group is the best;

[0063] (8)Table 5 shows the computational efficiency of this invention and other quality evaluation methods and visual security evaluation methods. By running the algorithms on the same computer and then comparing the execution times, the complexity differences between them can be roughly understood. In order to compare the computational times of different methods, we selected 10 images with a size of 515×515 in the PEID database for measurement, and took the average value as the final result. As shown in Table 5, the method of manually extracting features is significantly faster than the method of extracting features by deep learning methods. Compared with the existing visual security evaluation methods, the computational speed of this invention is only second to LSS and VSI-Canny, but these two methods use the manual method to extract the features of encrypted images, and their performance is limited. Compared with the visual security evaluation method, although this invention is not as good as CNNVSI in performance, it performs very well in computational speed, and CNNVSI belongs to the RVSA method.

[0064] The comparison method of this invention is:

[0065] Method 1: The method proposed by Wang et al., reference "Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Trans. on Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004, doi: 10.1109 / TIP.2003.819861."

[0066] Method 2: The method proposed by Mao et al., reference "Yinian Mao and Min Wu, “Security evaluation for communication-friendly encryption of multimedia,” in 2004 International Conference on Image Processing, 2004. ICIP’04., Singapore: IEEE, 2004, pp. 569–572. doi: 10.1109 / ICIP.2004.1418818."

[0067] Method 3: The method proposed by Mao et al., reference "Yinian Mao and Min Wu, “Security evaluation for communication-friendly encryption of multimedia,” in 2004 International Conference on Image Processing, 2004. ICIP’04., Singapore: IEEE, 2004, pp. 569–572. doi: 10.1109 / ICIP.2004.1418818."

[0068] Method 4: The method proposed by Tong et al., reference "L. Tong, F. Dai, Y. Zhang, and J. Li, “Visual security evaluation for video encryption,” in Proceedings of the 18th ACM international conference on Multimedia, Firenze Italy: ACM, Oct. 2010, pp. 835–838. doi: 10.1145 / 1873951.1874091."

[0069] Method 5: The method proposed by Xiang et al., reference "T. Xiang, S. Guo, and X. Li, “Perceptual Visual Security Index Based on Edge and Texture Similarities,” IEEE Trans. Inform. Forensic Secur., vol. 11, no. 5, pp. 951–963, May 2016, doi: 10.1109 / TIFS.2016.2515503."

[0070] Method 6: The method proposed by Xiang et al., reference "T. Xiang, Y. Yang, H. Liu, and S. Guo, “Visual Security Evaluation of Perceptually Encrypted Images Based on Image Importance,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 11, pp. 4129–4142, Nov. 2020, doi: 10.1109 / TCSVT.2019.2955298."

[0071] Method 7: The method proposed by Yang et al., reference "Y. Yang, T. Xiang, H. Liu, and X. Liao, “Convolutional Neural Network for Visual Security Evaluation,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 8, pp. 3293–3307, Aug. 2021, doi: 10.1109 / TCSVT.2020.3036854."

[0072] Method 8: The method proposed by Mittal et al., reference: “A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-Reference Image Quality Assessment in the Spatial Domain,” IEEE Trans. on Image Process., vol. 21, no. 12, pp. 4695–4708, Dec. 2012, doi: 10.1109 / TIP.2012.2214050.”

[0073] Method 9: The method proposed by Mittal et al., reference: “A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a ‘Completely Blind’ Image Quality Analyzer,” IEEE Signal Process. Lett., vol. 20, no. 3, pp. 209–212, Mar. 2013, doi: 10.1109 / LSP.2012.2227726.”

[0074] Method 10: The method proposed by Yue et al., reference: “G. Yue, C. Hou, K. Gu, T. Zhou, and H. Liu, “No-Reference Quality Evaluator of Transparently Encrypted Images,” IEEE Trans. Multimedia, vol. 21, no. 9, pp. 2184–2194, Sep. 2019, doi: 10.1109 / TMM.2019.2913315.”

[0075] Table 1 Results of PLCC, SRCC, KRCC, and RMSE in the IVC and PEID databases

[0076]

[0077] Table 2 SRCC results of 10 different encryption algorithms in the PEID database

[0078]

[0079] Table 3 Results of PLCC, SRCC, KRCC, and RMSE for different proportions of training data

[0080] METRIC 10-90 20-80 30-70 40-60 50-50 60-40 70-30 80-20 90-10 PLCC 0.415 0.664 0.758 0.804 0.854 0.875 0.881 0.898 0.899 SRCC 0.389 0.621 0.762 0.829 0.841 0.877 0.890 0.918 0.920 KRCC 0.345 0.574 0.654 0.689 0.72 0.741 0.747 0.753 0.766 RMSE 1.457 1.241 0.896 0.784 0.547 0.401 0.345 0.210 0.197

[0081] Results of the ten medians of the ablation experiment

[0082] Groups DEF LBP PLCC SRCC KRCC RMSE 1 √ × 0.704 0.694 0.512 0.845 2 × √ 0.574 0.521 0.457 1.241 3 √ √ 0.898 0.918 0.753 0.714

[0083] Table 5 Computational times of the IQA and VSA methods

[0084]

Claims

1. A blind visual security assessment method for encrypted images based on deep feature extractor, characterized by The following steps are involved: Step 1: A deep feature extractor for extracting features is obtained through the feature extraction part of the network model trained by visual tasks. Based on this, it is possible to quickly extract encrypted image features and solve problems such as the small number of encrypted images and long training time. Finally, a parallel dual-stream prediction network (PDSP-Net) is designed to implement blind visual security evaluation of encrypted images. The specific process is as follows: The deep feature extractor (DEF) is obtained by intercepting the feature extraction part of the existing network model. At the beginning of the training process, in order to adapt to the contrast masking process in human visual perception and highlight the individual differences in image information, the input encrypted image is locally de-averaged. At the same time, in order to ensure smooth convergence of the model training, the image is normalized and adjusted to a similar range. Step 2: After completing data preprocessing, in order to extract features from the encrypted image, the network model parameters of the pre-trained model for feature extraction are loaded, which is called a deep feature extractor. The main reason for adopting this method is that the amount of data in the existing encrypted image database is small, which is difficult to meet the parameter training requirements of large network models. In addition, the bottom layer of the network model extracts low-level semantic features through convolution operations. These features remain unchanged in different tasks. Therefore, we can use the pre-trained network model to extract the visual semantic feature map of the encrypted image. Using DEF as a deep feature extractor not only greatly reduces the number and complexity of the network model parameters, but also improves the computational efficiency. At the same time, in order to predict the quality score and visual security score of the encrypted image, a parallel two-stream prediction network is designed. It consists of two sub-networks: Sub-network I maps feature vectors to quality assessment scores. This network adopts a traditional model structure. After extracting the main features through principal component analysis, it uses feature fusion to combine different features to obtain a column vector of size 1×4096. It then connects four fully connected layers, each containing 4096, 2048, 1024 and 512 nodes respectively. The last layer is a linear prediction that outputs the quality assessment score of the image. Sub-network II maps feature vectors to visual security assessment scores. Because local details of encrypted images may leak information, texture analysis is performed on the feature map obtained by the deep feature extractor. Local binary patterns are used to extract texture information of the feature map, and the features of sub-network I are fused to achieve visual security assessment. Step 3: Take the encrypted image as input and go through the preprocessing in step 1, the deep feature extractor in step 2, and the trained parallel two-stream prediction network in sequence, and finally obtain the encrypted image quality score and visual security score.

Images