Privacy image detection method and model training method based on multi-modal feature fusion

Abstract

The invention provides a privacy image detection method based on multi-modal feature fusion and a model training method, which can be applied to the technical field of image processing and privacy protection. The privacy image detection method comprises the following steps: extracting label features and visual features in a scene graph of a privacy image; fusing the label features and the visual features in the scene graph to obtain fused features; updating the scene graphs based on the fusion features, and respectively calculating visual aggregation vectors in the updated scene graphs; fusing the updated visual aggregation vectors of the scene graph to obtain visual modal features; encoding the text description of the privacy image to obtain a text code, and updating the text code based on a cross-modal attention mechanism to obtain a text modal feature; and splicing and fusing the visual modal features and the text modal features to obtain multi-modal features, and processing the multi-modal features to obtain detection information of the privacy image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and privacy protection, and in particular to a privacy image detection method based on multimodal feature fusion and a training method for a privacy image detection model. Background Art

[0002] With the widespread use of the Internet, people often share images on social platforms or send them to online friends. However, image data contains a wealth of private information. When these images are widely disseminated on social networks, the risk of privacy leakage increases greatly. Due to the weak awareness of user privacy protection, sensitive information contained in images (such as face, intimate behavior, social relationships or other personal information) is often ignored. Therefore, it is crucial to automatically detect the privacy of images before users share them publicly.

[0003] Most existing image privacy detection methods use visual features to determine whether an image contains privacy information. Some works use deep learning methods to extract visual features of the entire image to determine whether the image belongs to the privacy category. However, these methods only rely on image-level visual features for privacy detection, ignoring the complex relationship between objects in the image, and the accuracy is often low. In order to extract fine-grained features in the image and consider the impact of co-occurrence relationships between objects on privacy detection, some methods model the image to be detected in the form of a graph. The nodes in the graph represent the objects in the original image, and the edges represent the associations between objects. However, these methods only consider the simple co-occurrence relationship of objects and ignore the explicit modeling of visual relationships that may lead to privacy leaks, such as hugs.

[0004] On the other hand, some research works use user-attached labels to predict privacy. These user labels are used to describe the image content and can also reflect the privacy category of the image from the side. However, users are not used to generating user labels for each image. Therefore, user labels in existing privacy image datasets are relatively scarce, which limits the improvement of privacy image detection or classification accuracy. Summary of the invention

[0005] In view of the above problems, the present invention provides a privacy image detection method based on multimodal feature fusion and a training method for a privacy image detection model to improve the detection precision and accuracy of privacy images.

[0006] According to a first aspect of the present invention, a privacy image detection method based on multimodal feature fusion is provided, comprising:

[0007] Extracting label features and visual features of nodes in the scene graph of the private image and label features and visual features of edges in the scene graph;

[0008] The label features and visual features of the nodes in the scene graph are fused to obtain the node fusion features, and the label features and visual features of the edges in the scene graph are fused to obtain the edge fusion features;

[0009] The scene graph is updated based on the node fusion features and the edge fusion features, and the visual aggregation vectors of the nodes and edges in the updated scene graph are calculated respectively;

[0010] The visual aggregation vectors of nodes and edges in the updated scene graph are fused to obtain visual modality features;

[0011] Encode the text description of the private image to obtain a text encoding, and update the text encoding based on a cross-modal attention mechanism to obtain a text modality feature;

[0012] The visual modal features and textual modal features are concatenated and fused to obtain multimodal features, and the multimodal features are processed to obtain detection information of private images.

[0013] According to an embodiment of the present invention, the above-mentioned extraction of label features and visual features of nodes in the scene graph of the privacy image and label features and visual features of edges in the scene graph includes:

[0014] Generate a scene graph of the private image, wherein the scene graph includes an object set, an object detection box set, and a predicate relationship set between objects;

[0015] The label features of the nodes are obtained by using the one-hot encoding of the set of objects in the scene graph, and the label features of the edges are obtained by using the one-hot encoding of the set of predicate relationships between objects in the scene graph;

[0016] The target image area corresponding to the object set in the scene graph is obtained by cropping the object detection frame set in the scene graph, and the feature of the target image area is extracted to obtain the visual feature of the node;

[0017] The visual features of the edges at the relation level are obtained by defining the intersection area between the object detection boxes in the scene graph.

[0018] According to an embodiment of the present invention, the above-mentioned fusing the label features and visual features of the nodes in the scene graph to obtain the node fusion features, and fusing the label features and visual features of the edges in the scene graph to obtain the edge fusion features includes:

[0019] Perform vector concatenation on the label features and visual features of the node, perform mapping processing on the vector concatenation results of the node, and activate the mapping processing results of the node to obtain the fusion features of the node;

[0020] The label features and visual features of the edge are vector-concatenated, the vector concatenation result of the edge is mapped, and the mapping result of the edge is activated to obtain the fusion feature of the edge.

[0021] According to an embodiment of the present invention, the updating of the scene graph based on the node fusion feature and the edge fusion feature includes:

[0022] The node fusion features are used as the initial node representation of the scene graph, and the edge fusion features are used as the initial edge representation of the scene graph;

[0023] Using the initial node representation to perform multiple rounds of updating of the node representation, the updated features of the nodes in the scene graph are obtained;

[0024] The initial edge representation is used to perform multiple rounds of updating of the edge representation to obtain the updated features of the edges in the scene graph.

[0025] According to an embodiment of the present invention, respectively calculating the visual aggregation vectors of the nodes and edges in the updated scene graph includes:

[0026] The updated features of the node are averaged to obtain the average features of the node, and the updated features and average features of the node are correlated, normalized, and weighted, as well as weighted aggregation based on the attention mechanism, to obtain the visual aggregation vector of the node;

[0027] The updated features of the edge are averaged to obtain the average features of the edge, and the updated features and the average features of the edge are subjected to correlation calculation, normalization, weight assignment, and weighted aggregation based on the attention mechanism to obtain the visual aggregation vector of the edge.

[0028] According to an embodiment of the present invention, the text description of the private image is encoded to obtain the text encoding, which includes:

[0029] Generate a text description of the private image and perform a mapping operation on the word vectors in the text description to obtain a word vector sequence;

[0030] Multi-directional hidden state calculation operations are performed on the word vector sequence, and the multiple calculation results are averaged to obtain a text encoding with word embedding representation.

[0031] According to an embodiment of the present invention, the text encoding is updated based on the cross-modal attention mechanism to obtain text modality features including:

[0032] Calculate the correlation score between the updated features of the nodes and edges and the word vectors in the text description;

[0033] Normalize the updated features of nodes and edges and use the relevance scores to get the attention distribution of word vectors in the text description;

[0034] Using the attention distribution and the normalized updated features of nodes and edges, we get the initial cross-modal representation of the text description.

[0035] The initial cross-modal representations are averaged and aggregated to obtain the textual modality features of the private image.

[0036] According to a second aspect of the present invention, a method for training a privacy image detection model is provided, comprising:

[0037] Generate scene graph samples of private image samples, and use the private image detection model to extract label feature samples and visual feature samples of the scene graph samples;

[0038] Fusing the label feature samples and the visual feature samples, and updating the scene graph samples based on the obtained fused feature samples to obtain updated scene graph samples;

[0039] Utilize the privacy image detection model to calculate the visual modality features of the updated scene graph samples;

[0040] Generate text description samples of private image samples, and use the private image detection model to encode the text description samples to obtain text feature samples;

[0041] The privacy image detection model is used to update the text feature samples based on the cross-modal attention mechanism to obtain text modality feature samples.

[0042] The visual modality feature samples and the text modality feature samples are concatenated and fused to obtain multimodal feature samples;

[0043] The multimodal feature samples are processed using the privacy image detection model to obtain classification information samples of the privacy image samples;

[0044] The cross entropy loss function is used to process the true value labels and classification information samples of the private image samples to obtain the loss value, and the parameters of the private image detection model are updated based on the loss value;

[0045] The operations of acquiring visual modal features and textual modal features of the privacy image samples, calculating the loss values, and updating the parameters of the privacy image detection model are iteratively performed until the preset training conditions are met, thereby obtaining a trained privacy image detection model, wherein the trained privacy image detection model is used to execute the above-mentioned privacy image detection method based on multimodal feature fusion.

[0046] According to an embodiment of the present invention, the above-mentioned privacy image detection model includes a pre-trained scene graph generation module, a graph convolutional neural network, a pre-trained convolutional neural network, a pre-trained image text description generation module and a cross-modal text encoding module.

[0047] According to an embodiment of the present invention, the pre-trained scene graph generation module includes a RelTR model;

[0048] Among them, the pre-trained image text description generation module includes an image encoder, a query transformer and a large language model unit.

[0049] The privacy image detection method based on multimodal feature fusion provided by the present invention generates a scene graph representation of the privacy image to be detected, explicitly models objects and the associations between objects, captures visual relationships that may leak privacy information, and solves the shortcomings of existing methods that only consider single objects or simple co-occurrence relationships between objects; at the same time, a trained privacy image detection model is used to generate a text description for the privacy image to be detected, that is, information such as objects, actions, scenes, etc. in the privacy image to be detected is explicitly expressed in the form of text, and text features are extracted and updated using cross-modal attention guided by image features, thereby solving the technical problem that user tags are relatively sparse and text features cannot be fully utilized; in addition, image features and text features are multimodally fused to fully utilize the complementary information between the two modalities, effectively improving the classification accuracy and detection accuracy of the privacy image to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0051] Figure 1 is an application scenario diagram of a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention;

[0052] Figure 2 is a flow chart of a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention;

[0053] Figure 3 is an architecture diagram and a data processing flow chart of a privacy image detection model according to an embodiment of the present invention;

[0054] Figure 4 is a structural block diagram of a privacy image detection device based on multimodal feature fusion according to an embodiment of the present invention;

[0055] Figure 5 is a block diagram of an electronic device suitable for implementing a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0057] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0058] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0059] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0060] Due to the rapid development of digital technology, user image data on the Internet faces the risk of privacy leakage or privacy tampering. How to identify or classify user image data on the Internet, especially image data containing user personal privacy information, and on the basis of identification or classification, effectively remind users and protect privacy has become a research hotspot for researchers in the field of image processing and privacy protection technology.

[0061] Since various privacy image data detection methods or classification methods in the prior art have low precision, poor accuracy and are heavily dependent on training data sets with true value labels, it is necessary to provide a privacy image detection method with high precision and high accuracy.

[0062] The embodiments of the present invention provide a privacy image detection method based on multimodal feature fusion, which is used to improve the classification precision and detection accuracy of privacy images.

[0063] Figure 13 is an application scenario diagram of a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention.

[0064] like Figure 1 As shown, the application scenario 100 according to this embodiment may include the field of image processing and privacy protection technology. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0065] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0066] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0067] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0068] It should be noted that the privacy image detection method based on multimodal feature fusion provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the privacy image detection device based on multimodal feature fusion provided in the embodiment of the present invention can generally be set in the server 105. The privacy image detection method based on multimodal feature fusion provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the privacy image detection device based on multimodal feature fusion provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0069] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0070] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The privacy image detection method based on multimodal feature fusion of the disclosed embodiment is described in detail.

[0071] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0072] In the technical solution of the present invention, the privacy images involved (including but not limited to user personal information, user speech and behavior information, user device information, user tag information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0073] Figure 2 4 is a flowchart of a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention.

[0074] like Figure 2 As shown, the privacy image detection method based on multimodal feature fusion of this embodiment includes operations S210 to S260.

[0075] In operation S210 , label features and visual features of nodes in a scene graph of a private image and label features and visual features of edges in the scene graph are extracted.

[0076] In an embodiment of the present invention, before obtaining the private image, the user's consent or authorization may be obtained. For example, before operation S210, a request for obtaining user information may be issued to the user. When the user agrees or authorizes that the user information may be obtained, operations S210 to S260 are performed.

[0077] Before performing multi-category feature extraction on the nodes and edges of a private image, it is necessary to use a relevant artificial intelligence model to generate a scene graph of the private image, and then perform feature extraction on the scene graph.

[0078] In operation S220, the label features and visual features of the nodes in the scene graph are fused to obtain node fusion features, and the label features and visual features of the edges in the scene graph are fused to obtain edge fusion features.

[0079] In operation S230 , the scene graph is updated based on the node fusion features and the edge fusion features, and visual aggregation vectors of nodes and edges in the updated scene graph are calculated respectively.

[0080] The above operation S230 is used to perform multi-round and multi-level updates on the fusion features of the nodes and edges in the scene graph, and calculate the visual aggregation features of the nodes and edges based on the fusion features of the nodes and edges after the multi-round and multi-level updates.

[0081] In operation S240 , the visual aggregation vectors of the nodes and edges in the updated scene graph are fused to obtain visual modality features.

[0082] The above operations S210 to S240 are visual processing of the private image, and are used to obtain features of the visual modality.

[0083] In operation S250, the text description of the private image is encoded to obtain a text encoding, and the text encoding is updated based on a cross-modal attention mechanism to obtain a text modality feature.

[0084] An artificial intelligence model is used to generate a text description of the private image, and the text description is encoded and updated to obtain text modal features.

[0085] The process of acquiring text modality features also utilizes the updated fusion features of nodes and edges of private images.

[0086] In operation S260, the visual modal features and the textual modal features are concatenated and fused to obtain multimodal features, and the multimodal features are processed to obtain detection information of the private image.

[0087] The above detection information includes classification results (such as whether it is a private image) and other information identification results involving user privacy.

[0088] The above operations S210 to S260 are performed using the trained privacy image detection model.

[0089] The privacy image detection method based on multimodal feature fusion provided by the present invention generates a scene graph representation of the privacy image to be detected, explicitly models objects and the associations between objects, captures visual relationships that may leak privacy information, and solves the shortcomings of existing methods that only consider single objects or simple co-occurrence relationships between objects; at the same time, a trained privacy image detection model is used to generate a text description for the privacy image to be detected, that is, information such as objects, actions, scenes, etc. in the privacy image to be detected is explicitly expressed in the form of text, and text features are extracted and updated using cross-modal attention guided by image features, thereby solving the technical problem that user tags are relatively sparse and text features cannot be fully utilized; in addition, image features and text features are multimodally fused to fully utilize the complementary information between the two modalities, effectively improving the classification accuracy and detection accuracy of the privacy image to be detected.

[0090] According to an embodiment of the present invention, the above-mentioned extraction of label features and visual features of nodes in a scene graph of a private image and label features and visual features of edges in the scene graph includes: generating a scene graph of a private image, wherein the scene graph includes an object set, an object detection frame set, and a predicate relationship set between objects. The label features of the nodes are obtained by using the one-hot encoding of the object set in the scene graph, and the label features of the edges are obtained by using the one-hot encoding of the predicate relationship set between objects in the scene graph; the target image area corresponding to the object set in the scene graph is obtained by cropping the object detection frame set in the scene graph, and the feature extraction of the target image area is performed to obtain the visual features of the nodes; the visual features of the edges at the relationship level are obtained by defining the intersection area between the object detection frames in the scene graph.

[0091] The following is a further detailed description of the generation process of the scene graph of the private image and the process of obtaining the labels of the nodes and edges and their visual features through specific embodiments.

[0092] Scene graph generation process: Use the scene graph generation model to generate a scene graph representation for each image in the dataset. Any existing scene graph generation model can be used to generate a scene graph for an image. The present invention uses the RelTR model to generate a visual scene graph, which can identify 50 categories of objects and 150 categories of visual relationships. In order to reduce the large number of redundant objects and relationships contained in the generated scene graph, which affects the subsequent image privacy classification, the model output results are filtered and screened, and only objects and relationships with confidence greater than the threshold are retained, and the Top-10 results are screened. The threshold is set to 0.3. The scene graph can be represented as ,in Represents the set of objects detected in a picture, Indicates The category labels of the objects. Represents the bounding box set of the object, Indicates the corresponding The detection box coordinates. Represents a collection of predicate relationships between objects, each of which Can be expressed as , that is, the object With Object Predicate Describes the visual relationship.

[0093] The process of obtaining label features of nodes and edges in the scene graph (i.e., the process of encoding label features): and Respectively represent objects and relational predicates The corresponding word vectors are shown in formulas (1) and (2):

[0094] (1),

[0095] (2),

[0096] in, and are the parameters that need to be updated during training, and Respectively represent the number of object categories and relationship categories supported by the scene graph generation algorithm, The dimension of the word embedding vector is . The present invention sets the number of object categories and the number of relationship categories to 50 and 150 respectively. The dimension is 300, and the external pre-trained word vector GloVe is used and Initialize.

[0097] The process of obtaining visual features of nodes and edges in the scene graph (i.e., the process of encoding visual features): by cropping the bounding box Get the object The corresponding image area is extracted using a pre-trained convolutional neural network (CNN) For the relationship , and its corresponding image area is defined as and The union box of these two detection boxes is used to obtain the visual features at the relationship level. The present invention uses the EfficientNet-b5 network pre-trained on ImageNet to extract visual features, and the dimension of the visual features is set to 2048 dimensions.

[0098] According to an embodiment of the present invention, the above-mentioned fusing the label features and visual features of the nodes in the scene graph to obtain the node fusion features, and fusing the label features and visual features of the edges in the scene graph to obtain the edge fusion features includes: vector splicing the label features and visual features of the nodes, mapping the vector splicing results of the nodes, and activating the mapping processing results of the nodes to obtain the node fusion features; vector splicing the label features and visual features of the edges, mapping the vector splicing results of the edges, and activating the mapping processing results of the edges to obtain the edge fusion features.

[0099] The following is a further detailed description of the process of acquiring the fusion features of nodes and edges in the above scene graph through a specific implementation method.

[0100] Based on the label features and visual features obtained above, the label features and visual features of the same object and relationship are first concatenated, and then the two are fused through a trainable neural network, as shown in formulas (3) and (4):

[0101] (3),

[0102] (4),

[0103] in is the trainable parameter matrix, represents the vector concatenation operation, Represents the activation function. After this step, each object node and relationship edge has a fused multimodal representation. In the experiment, the present invention uses ReLU as the activation function .

[0104] According to an embodiment of the present invention, the above-mentioned updating of the scene graph based on node fusion features and edge fusion features includes: using the node fusion features as the initial node representation of the scene graph, and using the edge fusion features as the initial edge representation of the scene graph; performing multiple rounds of updating the node representation using the initial node representation to obtain updated features of the nodes in the scene graph; performing multiple rounds of updating the edge representation using the initial edge representation to obtain updated features of the edges in the scene graph.

[0105] The following is a further detailed description of the updating process of the fusion features of nodes and edges in the above scene graph through a specific implementation method. The graph convolutional neural network (GCN) is used to update the features of nodes and edges. In the specific implementation, the present invention focuses on (1) node update: In order to reduce the cumulative impact of adjacent noise in certain scenarios, the node features are updated only based on the node's own information to avoid interference from irrelevant objects. (2) Edge update: Since an edge is used to connect two objects, its features should simultaneously fuse the edge's own information and the features of the two nodes connected by the edge. Assume that the depth of GCN is layer, and use and Respectively represent nodes Relationship In the Layer representation. Initially , as shown in formula (5):

[0106] (5),

[0107] In the In the layer, the update rules of nodes and edges can be defined as shown in formulas (6) and (7):

[0108] (6),

[0109] (7),

[0110] in and They are two multi-layer perceptron structures respectively. After layer iteration, the present invention will and is regarded as the final representation of nodes and edges, that is, the visual feature map of the entire image on the graph structure. In the specific implementation, the depth of GCN is set to 1 layer. and The dimensions are all 512.

[0111] According to an embodiment of the present invention, the above-mentioned calculation of the visual aggregation vectors of the nodes and edges in the updated scene graph includes: performing an averaging operation on the updated features of the nodes to obtain the average features of the nodes, and performing correlation calculation operations, normalization and weight assignment operations, and weighted aggregation operations based on the attention mechanism on the updated features and the average features of the nodes to obtain the visual aggregation vector of the nodes; performing an averaging operation on the updated features of the edges to obtain the average features of the edges, and performing correlation calculation operations, normalization and weight assignment operations, and weighted aggregation operations based on the attention mechanism on the updated features and the average features of the edges to obtain the visual aggregation vector of the edges.

[0112] The following is a further detailed description of the process of obtaining the visual aggregation vectors of the above nodes and edges through a specific implementation method.

[0113] For the previously acquired visual feature map , assuming it contains Nodes and The edge and node features are recorded as , the edge feature is denoted as , the node features and edge features are the feature representations after being updated by GCN.

[0114] In order to capture the global information of the node as a whole, the mean of the node features is first calculated, as shown in formula (8):

[0115] (8).

[0116] Then, for each node Calculate its The correlation and the trainable weight matrix And the attention mechanism is weighted. The node layer aggregation vector is shown in formula (9):

[0117] (9),

[0118] in, express The transpose of represents the training parameters, express function.

[0119] Similarly, the global mean is calculated for the edge features to obtain the center vector of the edge features in the graph, as shown in formula (10):

[0120] (10),

[0121] Then calculate each edge and The attention weights are calculated and weighted summed as shown in formula (11):

[0122] (11),

[0123] in, This operation makes the edge features occupy the corresponding positions in the graph embedding, thereby supplementing the interactive information that the node features cannot cover.

[0124] Finally, the node-level vector and the vectors at the edge level Concatenate to get the complete graph representation, as shown in formula (12):

[0125] (12),

[0126] in, Represents a vector-level concatenation operation. Through this representation, the node and edge features in the graph are fully fused and focused, thus forming a vector that has a high degree of generalization of the entire graph. The final complete graph representation feature is 1024 dimensions.

[0127] According to an embodiment of the present invention, the above-mentioned encoding of the text description of the private image to obtain the text encoding includes: generating a text description of the private image, and performing a mapping operation on the word vectors in the text description to obtain a word vector sequence; performing a multi-directional hidden state calculation operation on the word vector sequence, and averaging the obtained multiple calculation results to obtain a text encoding with a word embedding representation.

[0128] The text encoding process is further described in detail below through a specific implementation method.

[0129] Generate text descriptions for images using a pre-trained image description generation model (Image Caption) ,in Indicates that the text description contains words; in specific application scenarios, any pre-trained image description generation model can be selected to generate text descriptions. The present invention selects a pre-trained BLIP-2 model to automatically generate picture descriptions. The BLIP-2 framework consists of three core parts: an image encoder (Image Encoder), a query transformer (QueryTransformer, referred to as Q-Former), and a large language model (Large Language Model, LLM). Among them, Q-Former acts as a trainable bridge for feature docking between a fixed-weight image encoder and a large language model; in specific implementation, the image encoder uses ViT-G, and the LLM is based on OPT 2.7B. Through this pre-training framework, context-rich information such as objects, relationships between objects, scenes, etc. can be extracted from the image, and the corresponding initial text description can be output to provide basic semantic information on the text side for subsequent multimodal feature fusion.

[0130] Text feature encoding process: text description Each word in First, the word embedding layer is mapped to ,in is one-hot encoding, is the parameter matrix. The input is fed into the bidirectional GRU to calculate the hidden layer states from left to right and from right to left, as shown in formulas (13) and (14):

[0131] (13),

[0132] (14),

[0133] The final word embedding representation can be taken as the average of the hidden layer states in both directions, as shown in formula (15):

[0134] (15).

[0135] According to an embodiment of the present invention, the text encoding is updated based on the cross-modal attention mechanism to obtain the text modal features, including: calculating the correlation scores between the updated features of the nodes and the updated features of the edges and the word vectors in the text description; normalizing the updated features of the nodes and the updated features of the edges and using the correlation scores to obtain the attention distribution of the word vectors in the text description; using the attention distribution and the normalized updated features of the nodes and the normalized updated features of the edges to obtain an initial cross-modal representation of the text description; performing averaging and aggregation operations on the initial cross-modal representation to obtain the text modal features of the private image.

[0136] The following is a further detailed description of the process of acquiring the above-mentioned text modal features through a specific implementation method.

[0137] Cross-modal attention updates text features: Define node and edge features of GCN output in the image branch For node and edge features , calculate its correlation with the text word The score between is shown in formula (16):

[0138] (16),

[0139] in, and represents the learnable parameters, is the bias, is the row vector, is the activation function.

[0140] For each text word , for all graph nodes / edges Perform softmax normalization to obtain the attention distribution, as shown in formula (17):

[0141] (17).

[0142] For text words , that is, The image feature representation of text words is shown in formula (18):

[0143] (18).

[0144] Finally, The dimensions are averaged and aggregated to obtain the overall vector of the text, as shown in formulas (19) and (20):

[0145] (19),

[0146] (20),

[0147] in represents the MLP layer. The final text feature is 1024-dimensional.

[0148] After obtaining the final representation of the private image and text, the two are concatenated to obtain a fusion vector The concatenated vector is sent to the privacy image detection model for classification and detection to obtain the corresponding classification or detection information.

[0149] According to a second aspect of the present invention, a training method for a privacy image detection model is provided, comprising: generating a scene graph sample of a privacy image sample, and extracting a label feature sample and a visual feature sample of the scene graph sample by using the privacy image detection model; fusing the label feature sample and the visual feature sample, and updating the scene graph sample based on the obtained fused feature sample to obtain an updated scene graph sample; calculating the visual modal features of the updated scene graph sample by using the privacy image detection model; generating a text description sample of the privacy image sample, and encoding the text description sample by using the privacy image detection model to obtain a text feature sample; and performing an update operation on the text feature sample based on a cross-modal attention mechanism by using the privacy image detection model to obtain a text modal feature sample. This method comprises the following steps: concatenating and fusing visual modal feature samples and text modal feature samples to obtain multimodal feature samples; processing the multimodal feature samples using a privacy image detection model to obtain classification information samples of the privacy image samples; processing the true value labels and classification information samples of the privacy image samples using a cross entropy loss function to obtain loss values, and updating the parameters of the privacy image detection model based on the loss values; iteratively performing operations of acquiring visual modal features and text modal features of the privacy image samples, calculating loss values, and updating parameters of the privacy image detection model until preset training conditions are met, thereby obtaining a trained privacy image detection model, wherein the trained privacy image detection model is used to execute the above-mentioned privacy image detection method based on multimodal feature fusion.

[0150] According to an embodiment of the present invention, the above-mentioned privacy image detection model includes a pre-trained scene graph generation module, a graph convolutional neural network, a pre-trained convolutional neural network, a pre-trained image text description generation module and a cross-modal text encoding module.

[0151] According to an embodiment of the present invention, the above-mentioned pre-trained scene graph generation module includes a RelTR model; wherein the pre-trained image text description generation module includes an image encoder, a query transformer and a large language model unit.

[0152] The following is a specific implementation method and combined with the attached Figure 3 The training process of the privacy image detection model provided by the present invention and the application process of the trained privacy image detection model are further described in detail.

[0153] Figure 3 It is an architecture diagram and a data processing flow chart of a privacy image detection model according to an embodiment of the present invention.

[0154] First, we build a dataset for training the privacy image detection model. We use crawler technology to collect images shared by users from the Internet, including pictures shared by users on social networking sites and shared with online friends, and filter out low-resolution and damaged images. Multiple volunteers label the privacy category of each image. If the volunteers believe that the image contains private content and are unwilling to share it on the Internet, it will be labeled as "private", otherwise it will be labeled as "public".

[0155] like Figure 3 As shown, it mainly includes the training process of the model and the application process of the model, namely: by constructing a privacy image classification or detection dataset; constructing a scene graph representation of the image in the dataset; extracting the features of nodes and edges in the scene graph and using a graph convolutional neural network to update the features; using a graph embedding layer to extract the feature representation of the scene graph; generating a text description of the image and extracting text features; using a cross-modal attention layer to fine-grainedly align the text and scene graph; splicing image features and text features and sending them to the classification layer to obtain privacy detection results. The method provided by the present invention can fully explore the privacy content of the image and improve the accuracy of privacy image detection through the multimodal fusion of image and text features.

[0156] Based on the above-mentioned privacy image detection method based on multimodal feature fusion, the present invention also provides a privacy image detection device based on multimodal feature fusion. Figure 4 The device is described in detail.

[0157] Figure 4 4 is a structural block diagram of a privacy image detection device based on multimodal feature fusion according to an embodiment of the present invention.

[0158] like Figure 4 As shown, the privacy image detection device 400 based on multimodal feature fusion includes a label and visual feature extraction module 410, a label and visual feature fusion module 420, a fusion feature update module 430, a visual module acquisition module 440, a text modal feature acquisition module 450 and a multimodal fusion and detection module 460.

[0159] The label and visual feature extraction module 410 is used to extract label features and visual features of nodes in the scene graph of the private image and label features and visual features of edges in the scene graph. In one embodiment, the label and visual feature extraction module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0160] The label and visual feature fusion module 420 is used to fuse the label features and visual features of the nodes in the scene graph to obtain node fusion features, and to fuse the label features and visual features of the edges in the scene graph to obtain edge fusion features. In one embodiment, the label and visual feature fusion module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0161] The fusion feature update module 430 is used to update the scene graph based on the node fusion features and the edge fusion features, and calculate the visual aggregation vectors of the nodes and edges in the updated scene graph respectively; in one embodiment, the fusion feature update module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0162] The visual module acquisition module 440 is used to fuse the visual aggregation vectors of the nodes and edges in the updated scene graph to obtain visual modal features; in one embodiment, the visual module acquisition module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0163] The text modal feature acquisition module 450 is used to encode the text description of the private image to obtain a text encoding, and to update the text encoding based on a cross-modal attention mechanism to obtain a text modal feature. In one embodiment, the text modal feature acquisition module 450 can be used to perform the operation S250 described above, which will not be repeated here.

[0164] The multimodal fusion and detection module 460 is used to splice and fuse the visual modal features and the textual modal features to obtain multimodal features, and process the multimodal features to obtain detection information of the private image. In one embodiment, the multimodal fusion and detection module 460 can be used to perform the operation S260 described above, which will not be repeated here.

[0165] According to an embodiment of the present invention, any multiple modules among the label and visual feature extraction module 410, the label and visual feature fusion module 420, the fusion feature update module 430, the visual module acquisition module 440, the text modality feature acquisition module 450, and the multimodal fusion and detection module 460 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the label and visual feature extraction module 410, the label and visual feature fusion module 420, the fusion feature update module 430, the visual module acquisition module 440, the text modality feature acquisition module 450, and the multimodal fusion and detection module 460 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by an appropriate combination of any of them. Alternatively, at least one of the label and visual feature extraction module 410, the label and visual feature fusion module 420, the fusion feature update module 430, the visual module acquisition module 440, the text modality feature acquisition module 450, and the multimodal fusion and detection module 460 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0166] Figure 5 is a block diagram of an electronic device suitable for implementing a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention.

[0167] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0168] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0169] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage portion 508 as needed.

[0170] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0171] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0172] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the privacy image detection method based on multimodal feature fusion provided by the embodiment of the present invention.

[0173] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0174] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0175] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0176] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0177] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0179] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A privacy image detection method based on multimodal feature fusion, characterized in that: The method comprises: Extracting label features and visual features of nodes in a scene graph of a private image and label features and visual features of edges in the scene graph; The label features and visual features of the nodes in the scene graph are fused to obtain a node fusion feature, and the label features and visual features of the edges in the scene graph are fused to obtain an edge fusion feature; Updating the scene graph based on the node fusion feature and the edge fusion feature, and respectively calculating visual aggregation vectors of nodes and edges in the updated scene graph; Fusing the visual aggregation vectors of the nodes and edges in the updated scene graph to obtain visual modality features; Encoding the text description of the private image to obtain a text encoding, and updating the text encoding based on a cross-modal attention mechanism to obtain a text modality feature; The visual modal features and the textual modal features are concatenated and fused to obtain multimodal features, and the multimodal features are processed to obtain detection information of the private image.

2. The method according to claim 1, characterized in that Extracting label features and visual features of nodes in a scene graph of a private image and label features and visual features of edges in the scene graph includes: Generating a scene graph of the private image, wherein the scene graph includes a set of objects, a set of object detection frames, and a set of predicate relationships between objects; Obtaining label features of the nodes using one-hot encoding of a set of objects in the scene graph, and obtaining label features of the edges using one-hot encoding of a set of predicate relationships between objects in the scene graph; Obtaining a target image region corresponding to the object set in the scene graph by cropping the object detection frame set in the scene graph, and performing feature extraction on the target image region to obtain a visual feature of the node; The visual features of the edges at the relation level are obtained by defining the intersection area between the object detection boxes in the scene graph.

3. The method according to claim 1, characterized in that The label features and visual features of the nodes in the scene graph are fused to obtain the node fusion features, and the label features and visual features of the edges in the scene graph are fused to obtain the edge fusion features, including: Performing vector splicing on the label feature and the visual feature of the node, performing mapping processing on the vector splicing result of the node, and activating the mapping processing result of the node to obtain the fusion feature of the node; The label feature and the visual feature of the edge are vector-joined, the vector-joined result of the edge is mapped, and the mapping result of the edge is activated to obtain the fusion feature of the edge.

4. The method according to claim 1, characterized in that: Updating the scene graph based on the node fusion feature and the edge fusion feature includes: Using the node fusion feature as an initial node representation of the scene graph, and using the edge fusion feature as an initial edge representation of the scene graph; Perform multiple rounds of updating of node representations using the initial node representations to obtain updated features of nodes in the scene graph; The initial edge representation is used to perform multiple rounds of updating of edge representations to obtain updated features of the edges in the scene graph.

5. The method according to claim 4, characterized in that Calculating the visual convergence vectors of nodes and edges in the updated scene graph separately includes: Performing a mean operation on the updated features of the node to obtain an average feature of the node, and performing a correlation calculation operation, a normalization and a weight assignment operation, and a weighted aggregation operation based on an attention mechanism on the updated features and the average features of the node to obtain a visual aggregation vector of the node; A mean operation is performed on the updated features of the edge to obtain an average feature of the edge, and a correlation calculation operation, a normalization and weight assignment operation, and a weighted aggregation operation based on an attention mechanism are performed on the updated features and the average features of the edge to obtain a visual aggregation vector of the edge.

6. The method according to claim 4, characterized in that The text description of the private image is encoded, and the obtained text encoding includes: Generate a text description of the private image, and perform a mapping operation on word vectors in the text description to obtain a word vector sequence; A multi-directional hidden state calculation operation is performed on the word vector sequence, and a plurality of calculation results are averaged to obtain a text encoding with word embedding representation.

7. The method according to claim 6, characterized in that The text encoding is updated based on the cross-modal attention mechanism, and the obtained text modality features include: Calculating a correlation score between the updated feature of the node and the updated feature of the edge and the word vector in the text description; Normalizing the updated features of the node and the updated features of the edge and obtaining the attention distribution of the word vectors in the text description using the relevance score; Using the attention distribution and the normalized updated features of the nodes and the normalized updated features of the edges, an initial cross-modal representation of the text description is obtained; An averaging operation and a pooling operation are performed on the initial cross-modal representation to obtain a text modality feature of the private image.

8. A method for training a privacy image detection model, characterized in that: The method comprises: Generate a scene graph sample of the private image sample, and extract a label feature sample and a visual feature sample of the scene graph sample using a private image detection model; Fusing the label feature samples and the visual feature samples, and updating the scene graph samples based on the obtained fused feature samples to obtain updated scene graph samples; Calculating visual modality features of the updated scene graph samples using the privacy image detection model; Generate a text description sample of the private image sample, and encode the text description sample using the private image detection model to obtain a text feature sample; Using the privacy image detection model, an update operation based on a cross-modal attention mechanism is performed on the text feature sample to obtain a text modality feature sample; splicing and fusing the visual modality feature sample and the text modality feature sample to obtain a multimodal feature sample; Processing the multimodal feature sample using the privacy image detection model to obtain a classification information sample of the privacy image sample; Processing the true value label and classification information sample of the private image sample by using a cross entropy loss function to obtain a loss value, and updating the parameters of the private image detection model based on the loss value; Iteratively perform the operations of acquiring the visual modal features and the textual modal features of the privacy image sample, calculating the loss value, and updating the parameters of the privacy image detection model until a preset training condition is met, thereby obtaining a trained privacy image detection model, wherein the trained privacy image detection model is used to execute the method described in any one of claims 1 to 7.

9. The method according to claim 8, characterized in that The privacy image detection model includes a pre-trained scene graph generation module, a graph convolutional neural network, a pre-trained convolutional neural network, a pre-trained image text description generation module and a cross-modal text encoding module.

10. The method according to claim 9, characterized in that: The pre-trained scene graph generation module includes a RelTR model; The pre-trained image text description generation module includes an image encoder, a query transformer and a large language model unit.

Images