Picture privacy risk prediction method and system based on key region dynamic analysis, model training method and system

Abstract

The application discloses a picture privacy risk prediction model training method based on key region dynamic analysis, comprising the following steps: extracting the preliminary features of pictures in a training data set; clustering the preliminary features to obtain multiple key regions of the pictures and features of the multiple key regions; learning a correlation matrix of the multiple key regions of the pictures by using a neural network based on the features of the multiple key regions; fusing the features of the multiple key regions according to the correlation matrix to obtain fused features of the pictures; and predicting the privacy risk of the pictures based on the fused features.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning, in particular to a picture privacy risk prediction method and system based on key region dynamic analysis, and a model training method and system. BACKGROUND

[0002] Social networks have become a part of daily life, and people have become accustomed to posting a large number of pictures on social networks every day for recording and sharing life. According to statistics, more than 100 million and 300 million pictures are uploaded on Instagram and Facebook every day, respectively. Since 2014, pictures have replaced pure text and become the most popular sharing form for users.

[0003] However, a large amount of information is contained in the shared pictures, which may leak the privacy of the user. Therefore, convenient picture sharing also brings problems about privacy leakage. The shared pictures may be maliciously used, and even lead to some serious consequences. In addition, more implicit information may be dug out by using data mining, information retrieval and other technologies.

[0004] Social network platforms allow users to set the visibility of content to protect user privacy, but there is still a possibility that users may unintentionally share pictures that leak privacy. Although people have some consensus on what pictures may leak privacy, there is a difference between the actual privacy settings of the user for the pictures and their expectations, because the user often underestimates the privacy risk of the pictures to be shared when facing the pictures rather than specific private content. Therefore, the problem of picture privacy is attracting more and more attention, and for the above problem, a method is needed to help users reduce the risk of privacy leakage when sharing pictures.

[0005] In the prior art, an effective method is to infer the privacy risk in the pictures shared by the user, identify the pictures that may leak privacy, and give a warning. Some existing methods use machine learning methods to classify using the overall information of the picture, which lacks explainability and pertinence to the task. Another part of the existing methods finds out the objects related to privacy based on object detection, and further integrates the features of these objects to give a warning, but is limited by the categories of pre-defined objects, and it is difficult to deal with the image privacy leakage caused by objects outside the pre-defined list. SUMMARY

[0006] In view of the above problems, the main purpose of the present application is to provide a picture privacy risk prediction method and system based on key region dynamic analysis, and a model training method and system, which can automatically focus on key elements in the picture, such as object, scene, texture and other information, dynamically build the association between the key elements, predict the privacy risk of the picture, and give a warning to the picture with high privacy risk, reminding the user to upload it carefully.

[0007] To achieve the above objectives, this invention proposes a training method for an image privacy risk prediction model based on dynamic analysis of key regions, comprising: Step 1, extracting preliminary features of images in a training dataset; Step 2, clustering the preliminary features to obtain multiple key regions of the image and their features; Step 3, using a neural network to learn the correlation matrix of the multiple key regions of the image based on the features of the multiple key regions; Step 4, fusing the features of the multiple key regions according to the correlation matrix to obtain the fused features of the image; Step 5, predicting the privacy risk of the image based on the fused features.

[0008] The above-mentioned training method for the image privacy risk prediction model based on dynamic analysis of key regions, wherein the images in the training dataset are classified into public images and private images, wherein public images are labeled as 0 and private images are labeled as 1.

[0009] The above-mentioned image privacy risk prediction model training method based on key region dynamic analysis includes, in step 2: step 21, pre-training a privacy prediction classification model using the training dataset; step 22, clustering the last feature map of the image before it is used for classification by the privacy prediction classification model in the channel dimension to obtain the multiple key regions; step 23, performing a weighted average of the preliminary features corresponding to the multiple key regions to obtain the features of the multiple key regions.

[0010] The above-mentioned training method for image privacy risk prediction model based on dynamic analysis of key regions includes step 4, which includes: step 41, initializing multiple nodes of the graph neural network using the features of the multiple key regions, and initializing the correlation matrix of the graph neural network using the correlation matrix; step 42, fusing the features of the multiple key regions using the graph neural network to obtain the fused features of the image.

[0011] The above-mentioned training method for image privacy risk prediction model based on dynamic analysis of key regions includes step 5, which comprises: step 51, concatenating the fusion features of the image with the overall features of the image to obtain the concatenated features of the image; and step 52, predicting the privacy risk of the image based on the concatenated features.

[0012] In order to achieve the above object, the application provides a picture privacy risk prediction model training system based on key region dynamic analysis, comprising: a feature extraction module for extracting initial features of pictures in a training data set; a key region capturing module for clustering the initial features to obtain a plurality of key regions of the pictures and features of the key regions; a correlation learning module for learning a correlation matrix of the plurality of key regions of the pictures based on the features of the plurality of key regions by using a neural network; a feature fusion module for fusing the features of the plurality of key regions according to the correlation matrix to obtain fused features of the pictures; and a privacy risk prediction module for predicting a privacy risk of the pictures based on the fused features.

[0013] The picture privacy risk prediction model training system based on key region dynamic analysis, wherein the pictures in the training data set are classified into public pictures and privacy pictures, wherein the public pictures are labeled as 0 and the privacy pictures are labeled as 1.

[0014] The picture privacy risk prediction model training system based on key region dynamic analysis, wherein the key region capturing module comprises: a model pre-training module for pre-training a privacy prediction classification model by using the training data set; a key region obtaining module for clustering a last feature map of the pictures in a channel dimension before the privacy prediction classification model is used for classification to obtain the plurality of key regions; and a key feature obtaining module for performing weighted average on the initial features corresponding to the plurality of key regions to obtain features of the plurality of key regions.

[0015] The picture privacy risk prediction model training system based on key region dynamic analysis, wherein the feature fusion module is specifically configured to: initialize a plurality of nodes of a graph neural network by using the features of the plurality of key regions, and initialize a correlation matrix of the graph neural network by using the correlation matrix; and fuse the features of the plurality of key regions by using the graph neural network to obtain the fused features of the pictures.

[0016] The picture privacy risk prediction model training system based on key region dynamic analysis, wherein the privacy risk prediction module is specifically configured to: splice the fused features of the pictures with overall features of the pictures to obtain spliced features of the pictures; and predict the privacy risk of the pictures based on the spliced features.

[0017] In order to achieve the above object, the application provides a picture privacy risk prediction method based on key region dynamic analysis, comprising: inputting a target picture into the picture privacy risk prediction model training method based on key region dynamic analysis to obtain a picture privacy risk prediction model to output a privacy risk value of the picture; and judging a privacy risk of the target picture according to the privacy risk value.

[0018] To achieve the above objectives, this invention proposes an image privacy risk prediction system based on dynamic analysis of key regions, characterized by comprising: a privacy risk value prediction module, used to train an image privacy risk prediction model by inputting a target image into the aforementioned image privacy risk prediction model training method based on dynamic analysis of key regions to output the privacy risk value of the image; and a privacy risk judgment module, used to judge the privacy risk of the target image based on the privacy risk value.

[0019] To achieve the above objectives, the present invention proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described image privacy risk prediction model training method based on dynamic analysis of key regions and / or the above-described image privacy risk prediction method based on dynamic analysis of key regions.

[0020] To achieve the above objectives, the present invention proposes an electronic device, comprising: a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements the above-described image privacy risk prediction model training method based on key region dynamic analysis and / or the above-described image privacy risk prediction method based on key region dynamic analysis.

[0021] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the present invention. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a method for training an image privacy risk prediction model based on dynamic analysis of key regions, according to an embodiment of the present invention.

[0023] Figure 2 This is an architecture diagram of a training method for an image privacy risk prediction model based on dynamic analysis of key regions, according to an embodiment of the present invention.

[0024] Figure 3 This is a block diagram of a training system for an image privacy risk prediction model based on dynamic analysis of key regions, according to an embodiment of the present invention. Detailed Implementation

[0025] The structural and working principles of the present invention will be described in detail below with reference to the accompanying drawings:

[0026] The purpose of this invention is to predict the privacy risks of images that social network users are about to upload, and to issue warnings for images with high privacy risks, reminding users to upload with caution.

[0027] The technical solution of this invention has the following key points:

[0028] First, dynamic key region capture. To break away from the dependence of existing methods on object detection frameworks and to obtain information about regions in images that play a crucial role in privacy risk assessment, such as key objects, scene features, and background textures, a key region capture method based on feature map clustering was designed.

[0029] Second, dynamic relationship construction. The key regions obtained above are derived individually for each image. Existing methods based on dataset statistics cannot be used to construct the relationships between key regions in each image individually. Therefore, this invention designs a dynamic relationship construction method based on a self-attention mechanism to obtain the degree of correlation between key regions.

[0030] Third, dynamic region-aware graph network feature fusion. Existing graph neural networks can only fuse node features using a fixed graph structure. Based on the first and second key points mentioned above, this invention proposes a dynamic region-aware graph convolutional network for dynamically fusing features of key regions in each image.

[0031] Fourth, privacy risk prediction and early warning. Based on the first three key points, fusion features based on key regions of each image are obtained, and finally concatenated with the overall representation of the image to predict the privacy risks of the images, identify images with high privacy risks, and issue early warnings.

[0032] See Figure 1 and Figure 2 As shown, an embodiment of the present invention proposes a training method 100 for an image privacy risk prediction model based on dynamic analysis of key regions, which includes steps S110 to S150. Wherein:

[0033] Step S110: Extract preliminary features from the images in the training dataset.

[0034] In this embodiment, the training dataset is constructed, for example, by collecting and constructing images on a platform for predicting image privacy risks. The collected images are then categorized into public and private images. Furthermore, the collected images are labeled accordingly, with public images labeled as 0 and private images labeled as 1. However, this invention does not limit the method of obtaining the training dataset; it can also be obtained in other ways, such as using other existing publicly available labeled image privacy datasets.

[0035] For the images in the acquired training dataset, neural networks can be used to extract their preliminary features, such as using ResNet to obtain the convolutional features of the images.

[0036] Step S120: Cluster the preliminary features to obtain multiple key regions and their features of the image.

[0037] In this embodiment, to obtain key regions in an image and eliminate the reliance on object detection frameworks found in other works, this invention designs a key region capture method: using the obtained training dataset, the privacy / public labels of the image, and a pre-trained predictive privacy classification model (e.g., a ResNet neural network), this pre-trained predictive privacy classification model is used to cluster the last feature map before classification along the channel dimension, aggregating channels with similar peak response regions into N classes, resulting in N key regions. Specifically, to use a neural network to fit this clustering result for end-to-end training, the initial features (Fb) of the input image are used, and a multi-layer fully connected network (e.g., ...) is employed. Figure 2 The CGL (Clustered Geometric Array) is trained to approximate the clustering results of the feature maps. Its value indicates which of the N regions each channel belongs to, thus determining which channels in the initial features (Fb) represent the same region. A weighted average of the initial features of these channels yields the features of that region. This same process is applied to all N channels: a weighted sum of the end-to-end trained CGL results and the initial features (Fb) of the image is obtained, resulting in the features (Fw) of the N key regions.

[0038] Step S130: Based on the features of the multiple key regions, use a neural network to learn the correlation matrix of the multiple key regions of the image.

[0039] The key regions obtained in step S120 above are dynamic and therefore cannot be predefined using methods based on dataset statistics. Therefore, this invention proposes the following method: based on the features (Fw) of the N key regions obtained in step S120 above, a neural network is used to learn the correlation matrix A of the N key regions (e.g., ...). Figure 2 The diagram shows an example of the self-attention mechanism, where A is an N x N matrix, and each value in the matrix represents the degree of correlation between two key regions.

[0040] Step S140: Based on the correlation matrix, fuse the features of the multiple key regions to obtain the fused features of the image.

[0041] Existing graph neural networks can only fuse node features using a fixed graph structure. Based on the above steps, this invention designs a method for dynamically initializing graph neural networks: initializing the N nodes of the graph neural network using the features (Fw) of the obtained N key regions, and initializing the correlation matrix of the graph neural network using the correlation matrix A of the N key regions; thus, conventional graph neural networks (such as...) can be used... Figure 2 (As shown in the example of GCN) Feature fusion of N key regions is performed to obtain the fused feature (Fp) of the image.

[0042] Step S150: Predict the privacy risks of the image based on the fused features.

[0043] In this embodiment, preferably, to prevent the omission of other information in the image, the fusion feature (Fp) and the overall image feature have the same size but different number of channels. The fusion feature (Fp) and the overall image feature are concatenated along the channel dimension to obtain the image's concatenated feature. Then, based on the concatenated feature, a binary classification judgment is given using a fully connected neural network layer to predict whether the image has the risk of privacy leakage.

[0044] The following are system embodiments corresponding to the above method embodiments. This implementation system can be implemented in conjunction with the above implementation methods. The relevant technical details mentioned in the above embodiments are still valid in this implementation system, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this implementation system can also be applied to the above implementation methods.

[0045] See Figure 3 As shown, based on the same inventive concept, embodiments of the present invention propose a training system 200 for an image privacy risk prediction model based on dynamic analysis of key regions, comprising:

[0046] The feature extraction module 210 is used to extract preliminary features from the images in the training dataset;

[0047] The key region capture module 220 is used to cluster the preliminary features to obtain multiple key regions and their features of the image;

[0048] The correlation learning module 230 is used to learn the correlation matrix of multiple key regions of the image based on the features of the multiple key regions using a neural network.

[0049] The feature fusion module 240 is used to fuse the features of the multiple key regions according to the correlation matrix to obtain the fused features of the image;

[0050] Privacy risk prediction module 250 is used to predict the privacy risk of the image based on the fused features.

[0051] In one embodiment of the present invention, the images in the training dataset are classified into public images and private images, wherein public images are labeled as 0 and private images are labeled as 1.

[0052] In one embodiment of the present invention, the key area capture module includes:

[0053] The model pre-training module is used to pre-train a privacy prediction classification model using the training dataset;

[0054] The key region acquisition module is used to cluster the last feature map of the image before it is classified by the privacy prediction classification model in the channel dimension to obtain the multiple key regions.

[0055] The key feature acquisition module is used to perform a weighted average of the preliminary features corresponding to the multiple key regions to obtain the features of the multiple key regions.

[0056] In one embodiment of the present invention, the feature fusion module is specifically used to: initialize multiple nodes of the graph neural network using the features of the multiple key regions, initialize the correlation matrix of the graph neural network using the correlation matrix, and fuse the features of the multiple key regions using the graph neural network to obtain the fused features of the image.

[0057] In one embodiment of the present invention, the privacy risk prediction module is specifically used to: stitch together the fusion features of the image with the overall features of the image to obtain the stitched features of the image; and predict the privacy risks of the image based on the stitched features.

[0058] Following on from the above, once the image privacy risk prediction model based on the key region dynamic analysis method described above is trained, it can be deployed as an API for image privacy risk prediction on various online social platforms. For the target image to be judged, after dynamic region analysis, a privacy risk value ranging from 0 to 1 can be provided, where a higher value indicates a higher privacy risk.

[0059] Based on the same inventive concept, embodiments of the present invention propose a method for predicting image privacy risks based on dynamic analysis of key regions, including:

[0060] The target image is input into the image privacy risk prediction model training method based on key region dynamic analysis in the above embodiment to train the image privacy risk prediction model and output the privacy risk value of the image; the privacy risk of the target image is judged based on the privacy risk value.

[0061] Based on the same inventive concept, embodiments of the present invention propose an image privacy risk prediction system based on dynamic analysis of key regions, comprising:

[0062] The privacy risk value prediction module is used to input the target image into the image privacy risk prediction model training method based on key region dynamic analysis mentioned above to train the image privacy risk prediction model and output the privacy risk value of the image.

[0063] The privacy risk assessment module is used to assess the privacy risk of the target image based on the privacy risk value.

[0064] Based on the same inventive concept, embodiments of the present invention propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described image privacy risk prediction model training method based on dynamic analysis of key regions and / or the above-described image privacy risk prediction method based on dynamic analysis of key regions.

[0065] Based on the same inventive concept, embodiments of the present invention propose an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, it implements the above-described image privacy risk prediction model training method based on key region dynamic analysis and / or the above-described image privacy risk prediction method based on key region dynamic analysis.

[0066] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A training method for an image privacy risk prediction model based on dynamic analysis of key regions, characterized in that, include: Step 1: Extract preliminary features from the images in the training dataset; Step 2: Cluster the preliminary features to obtain multiple key regions and their features of the image; Step 3: Based on the features of the multiple key regions, use a neural network to learn the correlation matrix of the multiple key regions of the image; Step 4: Based on the correlation matrix, fuse the features of the multiple key regions to obtain the fused features of the image; Step 5: Predict the privacy risks of the image based on the fused features; Step 2 includes: Step 21, pre-training a privacy prediction classification model using the training dataset; Step 22: Cluster the last feature map of the image before it is used for classification by the privacy prediction classification model in the channel dimension to obtain the multiple key regions; Step 23: Perform a weighted average of the preliminary features corresponding to the multiple key regions to obtain the features of the multiple key regions.

2. The image privacy risk prediction model training method based on key region dynamic analysis as described in claim 1, characterized in that, The images in the training dataset are classified into public images and private images, where public images are labeled as 0 and private images are labeled as 1.

3. The image privacy risk prediction model training method based on key region dynamic analysis as described in claim 1, characterized in that, Step 4 includes: Step 41: Initialize multiple nodes of the graph neural network using the features of the multiple key regions, and initialize the correlation matrix of the graph neural network using the correlation matrix; Step 42: Use the graph neural network to fuse the features of the multiple key regions to obtain the fused features of the image.

4. The image privacy risk prediction model training method based on key region dynamic analysis as described in claim 1, characterized in that, Step 5 includes: Step 51: Combine the fusion features of the image with the overall features of the image to obtain the combined features of the image; Step 52: Predict the privacy risks of the image based on the stitching features.

5. A training system for an image privacy risk prediction model based on dynamic analysis of key regions, characterized in that, include: The feature extraction module is used to extract preliminary features from images in the training dataset; A key region capture module is used to cluster the preliminary features to obtain multiple key regions and their features of the image; The correlation learning module is used to learn the correlation matrix of multiple key regions of the image based on the features of the multiple key regions using a neural network. The feature fusion module is used to fuse the features of the multiple key regions according to the correlation matrix to obtain the fused features of the image; A privacy risk prediction module is used to predict the privacy risks of the image based on the fused features; The key region capture module includes: a model pre-training module for pre-training a privacy prediction classification model using the training dataset; a key region acquisition module for clustering the last feature map of the image before classification by the privacy prediction classification model in the channel dimension to obtain the multiple key regions; and a key feature acquisition module for performing a weighted average of the preliminary features corresponding to the multiple key regions to obtain the features of the multiple key regions.

6. The image privacy risk prediction model training system based on dynamic analysis of key regions as described in claim 5, characterized in that, The images in the training dataset are classified into public images and private images, where public images are labeled as 0 and private images are labeled as 1.

7. The image privacy risk prediction model training system based on key region dynamic analysis as described in claim 5, characterized in that, The feature fusion module is specifically used for: Multiple nodes of the graph neural network are initialized using the features of the multiple key regions, and the correlation matrix of the graph neural network is initialized using the correlation matrix. The graph neural network is used to fuse features from multiple key regions to obtain the fused features of the image.

8. The image privacy risk prediction model training system based on key region dynamic analysis as described in claim 5, characterized in that, The privacy risk prediction module is specifically used for: The image's fusion features are combined with the image's overall features to obtain the image's stitched features; The privacy risks of the image are predicted based on the stitching features.

9. A method for predicting image privacy risks based on dynamic analysis of key regions, characterized in that, include: The target image is input into the method described in any one of claims 1-4 to train an image privacy risk prediction model to output the privacy risk value of the image; The privacy risk of the target image is determined based on the privacy risk value.

10. A system for predicting image privacy risks based on dynamic analysis of key regions, characterized in that, include: A privacy risk value prediction module is used to input the target image into the method described in any one of claims 1-4 to train an image privacy risk prediction model and output the privacy risk value of the image. The privacy risk assessment module is used to assess the privacy risk of the target image based on the privacy risk value.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the methods as described in claims 1-4 and / or the method as described in claim 9.

12. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, implements the methods as claimed in claims 1-4 and / or the method as claimed in claim 9.

Images