Privacy image detection method and model training method based on multimodal feature fusion

Abstract

The present invention provides a privacy image detection method and model training method based on multimodal feature fusion, which can be applied to the fields of image processing and privacy protection technology. The privacy image detection method includes: extracting label features and visual features from a scene graph of the privacy image; fusing the label features and visual features in the scene graph to obtain a fused feature; updating the scene graph based on the fused feature and respectively calculating the visual aggregation vectors in the updated scene graph; fusing the visual aggregation vectors in the updated scene graph to obtain a visual modal feature; encoding the text description of the privacy image to obtain a text encoding, and updating the text encoding based on a cross-modal attention mechanism to obtain a text modal feature; splicing and fusing the visual modal feature and the text modal feature to obtain a multimodal feature, and processing the multimodal feature to obtain detection information of the privacy image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and privacy protection, and in particular to a privacy image detection method based on multimodal feature fusion and a training method for a privacy image detection model. Background Art

[0002] With the widespread use of the internet, people often share images on social platforms or send them to online friends. However, image data contains a wealth of private information. When these images are widely disseminated on social networks, the risk of privacy leakage increases significantly. Due to users' weak awareness of privacy protection, they often overlook sensitive information contained in images (such as faces, intimate gestures, social relationships, or other personal information). Therefore, it is crucial to automatically detect privacy issues before users publicly share images.

[0003] Most existing image privacy detection methods use visual features to determine whether an image contains private information. Some work uses deep learning to extract visual features from the entire image to determine whether the image falls into a privacy category. However, these methods rely solely on image-level visual features for privacy detection, ignoring the complex relationships between objects in the image and often resulting in low accuracy. To extract fine-grained features from the image while considering the impact of co-occurrence relationships between objects on privacy detection, some methods model the image to be detected as a graph. Nodes in the graph represent objects in the original image, and edges represent connections between objects. However, these methods only consider simple co-occurrence relationships between objects and neglect explicit modeling of visual relationships that may lead to privacy leaks, such as hugs.

[0004] On the other hand, some research works have used user-added tags to predict privacy. These user tags are used to describe the image content and can also indirectly reflect the privacy category of the image. However, users are not accustomed to generating user tags for each image. Therefore, user tags in existing private image datasets are relatively scarce, which limits the improvement of private image detection or classification accuracy. Summary of the Invention

[0005] In view of the above problems, the present invention provides a private image detection method based on multimodal feature fusion and a training method for a private image detection model to improve the detection precision and accuracy of private images.

[0006] According to a first aspect of the present invention, a method for detecting private images based on multimodal feature fusion is provided, comprising:

[0007] Extracting label features and visual features of nodes in the scene graph of the private image and label features and visual features of edges in the scene graph;

[0008] The label features and visual features of the nodes in the scene graph are fused to obtain the node fusion features, and the label features and visual features of the edges in the scene graph are fused to obtain the edge fusion features;

[0009] Update the scene graph based on the node fusion features and edge fusion features, and calculate the visual aggregation vectors of the nodes and edges in the updated scene graph respectively;

[0010] The visual aggregation vectors of nodes and edges in the updated scene graph are fused to obtain visual modality features;

[0011] Encode the text description of the private image to obtain a text encoding, and then update the text encoding based on the cross-modal attention mechanism to obtain text modality features;

[0012] The visual modal features and text modal features are spliced and fused to obtain multimodal features, and the multimodal features are processed to obtain the detection information of the private image.

[0013] According to an embodiment of the present invention, the extraction of label features and visual features of nodes in a scene graph of a private image and label features and visual features of edges in the scene graph includes:

[0014] Generate a scene graph of the private image, where the scene graph includes a set of objects, a set of object detection boxes, and a set of predicate relationships between objects;

[0015] The label features of the nodes are obtained by using the one-hot encoding of the set of objects in the scene graph, and the label features of the edges are obtained by using the one-hot encoding of the set of predicate relationships between objects in the scene graph;

[0016] The target image area corresponding to the object set in the scene graph is obtained by cropping the object detection frame set in the scene graph, and the feature extraction of the target image area is performed to obtain the visual features of the node;

[0017] The visual features of the edge at the relation level are obtained by defining the intersection area between the object detection boxes in the scene graph.

[0018] According to an embodiment of the present invention, the above-mentioned fusion of the label features and visual features of the nodes in the scene graph to obtain the node fusion features, and the fusion of the label features and visual features of the edges in the scene graph to obtain the edge fusion features include:

[0019] Perform vector splicing on the label features and visual features of the node, perform mapping processing on the vector splicing results of the node, and activate the mapping processing results of the node to obtain the fusion features of the node;

[0020] The label features and visual features of the edge are vector-concatenated, the vector concatenation results of the edge are mapped, and the mapping results of the edge are activated to obtain the fusion features of the edge.

[0021] According to an embodiment of the present invention, updating the scene graph based on the node fusion feature and the edge fusion feature includes:

[0022] The node fusion features are used as the initial node representation of the scene graph, and the edge fusion features are used as the initial edge representation of the scene graph;

[0023] Using the initial node representation, multiple rounds of node representation updates are performed to obtain updated features of the nodes in the scene graph.

[0024] The initial edge representation is used to perform multiple rounds of edge representation updates to obtain updated features of the edges in the scene graph.

[0025] According to an embodiment of the present invention, respectively calculating the visual aggregation vectors of the nodes and edges in the updated scene graph includes:

[0026] The updated features of the node are averaged to obtain the average features of the node. The updated features and the average features of the node are then correlated, normalized, and weighted, as well as weightedly aggregated based on the attention mechanism to obtain the visual aggregation vector of the node.

[0027] The updated features of the edge are averaged to obtain the average features of the edge. The updated features and the average features of the edge are then correlated, normalized, weighted, and aggregated based on the attention mechanism to obtain the visual aggregation vector of the edge.

[0028] According to an embodiment of the present invention, encoding the text description of the private image to obtain the text encoding includes:

[0029] Generate a text description of the private image and perform a mapping operation on the word vectors in the text description to obtain a word vector sequence;

[0030] Multi-directional hidden state calculation operations are performed on the word vector sequence, and the multiple calculation results are averaged to obtain a text encoding with word embedding representation.

[0031] According to an embodiment of the present invention, the text encoding is updated based on the cross-modal attention mechanism to obtain text modality features including:

[0032] Calculate the correlation score between the updated features of the nodes and edges and the word vectors in the text description;

[0033] Normalize the updated features of nodes and edges and use the relevance scores to obtain the attention distribution of word vectors in the text description;

[0034] Using the attention distribution and the normalized updated features of nodes and edges, we obtain the initial cross-modal representation of the text description.

[0035] The initial cross-modal representations are averaged and aggregated to obtain the text modality features of the private image.

[0036] According to a second aspect of the present invention, a method for training a private image detection model is provided, comprising:

[0037] Generate scene graph samples of private image samples, and use the private image detection model to extract label feature samples and visual feature samples of the scene graph samples;

[0038] Fusing the label feature samples and the visual feature samples, and updating the scene graph samples based on the obtained fused feature samples to obtain updated scene graph samples;

[0039] Utilize the privacy image detection model to calculate the visual modality features of the updated scene graph samples;

[0040] Generate text description samples of private image samples and use the private image detection model to encode the text description samples to obtain text feature samples;

[0041] The privacy image detection model is used to update the text feature samples based on the cross-modal attention mechanism to obtain text modality feature samples.

[0042] The visual modality feature samples and the text modality feature samples are spliced and fused to obtain multimodal feature samples;

[0043] The multimodal feature samples are processed using the privacy image detection model to obtain classification information samples of the privacy image samples;

[0044] The cross-entropy loss function is used to process the true value labels and classification information samples of private image samples to obtain the loss value, and the parameters of the private image detection model are updated based on the loss value;

[0045] The operations of acquiring visual modal features and textual modal features of private image samples, calculating loss values, and updating parameters of the private image detection model are iteratively performed until preset training conditions are met, thereby obtaining a trained private image detection model. The trained private image detection model is used to execute the above-mentioned private image detection method based on multimodal feature fusion.

[0046] According to an embodiment of the present invention, the above-mentioned privacy image detection model includes a pre-trained scene graph generation module, a graph convolutional neural network, a pre-trained convolutional neural network, a pre-trained image text description generation module and a cross-modal text encoding module.

[0047] According to an embodiment of the present invention, the pre-trained scene graph generation module includes a RelTR model;

[0048] Among them, the pre-trained image text description generation module includes an image encoder, a query transformer and a large language model unit.

[0049] The privacy image detection method based on multimodal feature fusion provided by the present invention generates a scene graph representation of the privacy image to be detected, explicitly models objects and the associations between objects, and captures visual relationships that may leak private information, thereby addressing the shortcomings of existing methods that only consider single objects or simple co-occurrence relationships between objects. At the same time, a trained privacy image detection model is used to generate a text description for the privacy image to be detected, that is, information such as objects, actions, and scenes in the privacy image to be detected is explicitly expressed in the form of text. By extracting text features and updating the text features using cross-modal attention guided by image features, the technical problem of sparse user tags and the inability to fully utilize text features is addressed. In addition, the multimodal fusion of image features and text features fully utilizes the complementary information between the two modalities, effectively improving the classification accuracy and detection accuracy of the privacy image to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0051] Figure 1 is an application scenario diagram of a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention;

[0052] Figure 2 is a flowchart of a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention;

[0053] Figure 3 is an architecture diagram and data processing flow chart of a privacy image detection model according to an embodiment of the present invention;

[0054] Figure 4 is a structural block diagram of a privacy image detection device based on multimodal feature fusion according to an embodiment of the present invention;

[0055] Figure 5 4 is a block diagram of an electronic device suitable for implementing a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0057] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0058] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0059] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0060] Due to the rapid development of digital technology, user image data on the internet faces the risk of privacy leakage or tampering. How to identify or classify user image data on the internet, especially image data containing personal privacy information, and based on this identification or classification, provide effective user reminders and privacy protection has become a hot topic for researchers in the fields of image processing and privacy protection.

[0061] Since various privacy image data detection methods or classification methods in the existing technology have low precision, poor accuracy and are heavily dependent on training data sets with true value labels, it is necessary to provide a privacy image detection method with high precision and high accuracy.

[0062] Embodiments of the present invention provide a privacy image detection method based on multimodal feature fusion, which is used to improve the classification precision and detection accuracy of privacy images.

[0063] Figure 12 is an application scenario diagram of a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention.

[0064] like Figure 1 As shown, the application scenario 100 according to this embodiment may include the fields of image processing and privacy protection technology. A network 104 is used as a medium for providing a communication link between a first terminal device 101, a second terminal device 102, a third terminal device 103, and a server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0065] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0066] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0067] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0068] It should be noted that the privacy-preserving image detection method based on multimodal feature fusion provided in the embodiments of the present invention can generally be executed by the server 105. Accordingly, the privacy-preserving image detection apparatus based on multimodal feature fusion provided in the embodiments of the present invention can generally be disposed in the server 105. The privacy-preserving image detection method based on multimodal feature fusion provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from the server 105 and that is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Accordingly, the privacy-preserving image detection apparatus based on multimodal feature fusion provided in the embodiments of the present invention can also be disposed in a server or server cluster that is different from the server 105 and that is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0069] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0070] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The privacy image detection method based on multimodal feature fusion of the disclosed embodiment is described in detail.

[0071] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0072] In the technical solution of the present invention, the privacy images involved (including but not limited to user personal information, user speech and behavior information, user device information, user tag information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0073] Figure 2 4 is a flowchart of a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention.

[0074] like Figure 2 As shown, the privacy image detection method based on multimodal feature fusion of this embodiment includes operations S210 to S260.

[0075] In operation S210 , label features and visual features of nodes in a scene graph of a private image and label features and visual features of edges in the scene graph are extracted.

[0076] In an embodiment of the present invention, the user's consent or authorization may be obtained before obtaining the private image. For example, before operation S210, a request to obtain user information may be issued to the user. If the user agrees or authorizes the acquisition of user information, operations S210 to S260 are performed.

[0077] Before performing multi-category feature extraction on the nodes and edges of private images, it is necessary to use relevant artificial intelligence models to generate a scene graph of the private image, and then perform feature extraction on the scene graph.

[0078] In operation S220 , the label features and visual features of the nodes in the scene graph are fused to obtain node fusion features, and the label features and visual features of the edges in the scene graph are fused to obtain edge fusion features.

[0079] In operation S230 , the scene graph is updated based on the node fusion features and the edge fusion features, and visual aggregation vectors of the nodes and edges in the updated scene graph are calculated respectively.

[0080] The above operation S230 is used to perform multi-round and multi-level updates on the fusion features of the nodes and edges in the scene graph, and calculate the visual aggregation features of the nodes and edges based on the fusion features of the nodes and edges after the multi-round and multi-level updates.

[0081] In operation S240 , the visual aggregation vectors of the nodes and edges in the updated scene graph are fused to obtain visual modality features.

[0082] The above operations S210 to S240 are visual processing of the private image, and are used to obtain features of the visual modality.

[0083] In operation S250 , the text description of the private image is encoded to obtain a text encoding, and the text encoding is updated based on a cross-modal attention mechanism to obtain a text modality feature.

[0084] An artificial intelligence model is used to generate text descriptions of private images, and the text descriptions are encoded and updated to obtain text modal features.

[0085] The process of acquiring text modality features also utilizes the updated fusion features of nodes and edges of private images.

[0086] In operation S260 , the visual modal features and the text modal features are concatenated and fused to obtain multimodal features, and the multimodal features are processed to obtain detection information of the private image.

[0087] The above detection information includes classification results (such as whether it is a private image) and other information identification results involving user privacy.

[0088] The above operations S210 to S260 are performed using the trained privacy image detection model.

[0089] The privacy image detection method based on multimodal feature fusion provided by the present invention generates a scene graph representation of the privacy image to be detected, explicitly models objects and the associations between objects, and captures visual relationships that may leak private information, thereby addressing the shortcomings of existing methods that only consider single objects or simple co-occurrence relationships between objects. At the same time, a trained privacy image detection model is used to generate a text description for the privacy image to be detected, that is, information such as objects, actions, and scenes in the privacy image to be detected is explicitly expressed in the form of text. By extracting text features and updating the text features using cross-modal attention guided by image features, the technical problem of sparse user tags and the inability to fully utilize text features is addressed. In addition, the multimodal fusion of image features and text features fully utilizes the complementary information between the two modalities, effectively improving the classification accuracy and detection accuracy of the privacy image to be detected.

[0090] According to an embodiment of the present invention, the above-mentioned extraction of label features and visual features of nodes in a scene graph of a private image, as well as label features and visual features of edges in the scene graph, includes: generating a scene graph for the private image, wherein the scene graph includes a set of objects, a set of object detection boxes, and a set of predicate relationships between objects. Using one-hot encoding of the object set in the scene graph to obtain label features of nodes, and using one-hot encoding of the predicate relationships between objects in the scene graph to obtain label features of edges; cropping the set of object detection boxes in the scene graph to obtain a target image region corresponding to the object set in the scene graph, and performing feature extraction on the target image region to obtain visual features of nodes; and defining the intersection region between the object detection boxes in the scene graph to obtain visual features of edges at the relationship level.

[0091] The following describes in further detail the process of generating a scene graph of a private image and the process of obtaining node and edge labels and their visual features through specific embodiments.

[0092] Scene graph generation process: Use the scene graph generation model to generate a scene graph representation for each image in the dataset. Any existing scene graph generation model can be used to generate a scene graph for an image. The present invention uses the RelTR model to generate a visual scene graph, which can identify 50 categories of objects and 150 categories of visual relationships. In order to reduce the large number of redundant objects and relationships in the generated scene graph, which affects the subsequent image privacy classification, the model output results are filtered and screened, and only objects and relationships with confidence greater than the threshold are retained, and the Top-10 results are screened. The threshold is set to 0.3. The scene graph can be represented as ,in Represents the set of objects detected in a picture, Indicates the The category labels of the objects. Represents the bounding box set of the object, Indicates correspondence The detection box coordinates. Represents a set of predicate relationships between objects, each of which Can be expressed as , that is, the object with objects between predicates Describes the visual relationship.

[0093] The process of obtaining label features of nodes and edges in the scene graph (i.e., label feature encoding process): Let and Represent objects separately and relational predicates The corresponding word vectors are shown in formulas (1) and (2):

[0094] (1),

[0095] (2),

[0096] in, and are the parameters that need to be updated during training, and Respectively represent the number of object categories and relationship categories supported by the scene graph generation algorithm, The dimension of the word embedding vector is . The present invention sets the number of object categories and the number of relationship categories to 50 and 150 respectively. The dimension is 300, and the external pre-trained word vector GloVe is used and Initialize.

[0097] The process of obtaining visual features of nodes and edges in the scene graph (i.e., visual feature encoding process): by cropping the bounding box Get the object The corresponding image area is extracted using a pre-trained convolutional neural network (CNN) For the relationship , and its corresponding image area is defined as and The union box of these two detection boxes is used to obtain the relationship-level visual features. The present invention uses the EfficientNet-b5 network pre-trained on ImageNet to extract visual features, and the dimension of the visual features is set to 2048 dimensions.

[0098] According to an embodiment of the present invention, the above-mentioned fusing of the label features and visual features of the nodes in the scene graph to obtain the node fusion features, and fusing the label features and visual features of the edges in the scene graph to obtain the edge fusion features includes: vector splicing of the label features and visual features of the nodes, mapping processing of the vector splicing results of the nodes, and activating the mapping processing results of the nodes to obtain the node fusion features; vector splicing of the label features and visual features of the edges, mapping processing of the vector splicing results of the edges, and activating the mapping processing results of the edges to obtain the edge fusion features.

[0099] The following is a further detailed description of the process of obtaining the fusion features of nodes and edges in the above scene graph through a specific implementation method.

[0100] Based on the label features and visual features obtained above, the label features and visual features of the same object and relationship are first concatenated, and then the two are fused through a trainable neural network, as shown in formulas (3) and (4):

[0101] (3),

[0102] (4),

[0103] in is the trainable parameter matrix, Represents vector concatenation operation, Represents the activation function. After this step, each object node and relationship edge has a fused multimodal representation. In the experiment, the present invention uses ReLU as the activation function .

[0104] According to an embodiment of the present invention, the above-mentioned updating of the scene graph based on node fusion features and edge fusion features includes: using the node fusion features as the initial node representation of the scene graph, and using the edge fusion features as the initial edge representation of the scene graph; using the initial node representation to perform multiple rounds of updating the node representation to obtain updated features of the nodes in the scene graph; using the initial edge representation to perform multiple rounds of updating the edge representation to obtain updated features of the edges in the scene graph.

[0105] The following is a detailed description of the updating process of the fusion features of nodes and edges in the above scene graph through a specific implementation method. The graph convolutional neural network (GCN) is used to update the features of nodes and edges. In the specific implementation, the present invention focuses on (1) node update: In order to reduce the cumulative impact of adjacent noise in certain scenarios, the node features are updated only based on the node's own information to avoid interference from irrelevant objects. (2) Edge update: Since an edge is used to connect two objects, its features should simultaneously fuse the edge's own information and the features of the two nodes connected by the edge. Assuming the depth of GCN is layer, and use and Represents nodes respectively Relationship In the Layer representation. Initially , as shown in formula (5):

[0106] (5),

[0107] In the In the layer, the update rules of nodes and edges can be defined as shown in formulas (6) and (7):

[0108] (6),

[0109] (7),

[0110] in and They are two multi-layer perceptron structures. After the layer iteration, the present invention will and is regarded as the final representation of nodes and edges, that is, the visual feature map of the entire image on the graph structure. In the specific implementation, the depth of GCN is set to 1 layer. and The dimensions are all 512.

[0111] According to an embodiment of the present invention, the above-mentioned calculation of the visual aggregation vectors of the nodes and edges in the updated scene graph includes: performing an averaging operation on the updated features of the nodes to obtain the average features of the nodes, and performing correlation calculation operations, normalization and weight assignment operations, and weighted aggregation operations based on the attention mechanism on the updated features and the average features of the nodes to obtain the visual aggregation vector of the nodes; performing an averaging operation on the updated features of the edges to obtain the average features of the edges, and performing correlation calculation operations, normalization and weight assignment operations, and weighted aggregation operations based on the attention mechanism on the updated features and the average features of the edges to obtain the visual aggregation vector of the edges.

[0112] The following further describes in detail the process of obtaining the visual aggregation vectors of the above nodes and edges through specific implementation methods.

[0113] For the previously acquired visual feature map , assuming it contains Nodes and Edge, node features are recorded as , edge features are denoted as , node features and edge features are the feature representations after GCN update.

[0114] In order to capture the global information of the node as a whole, the mean of the node features is first calculated, as shown in formula (8):

[0115] (8).

[0116] Then, for each node Calculate its The correlation, and through the trainable weight matrix The node layer aggregation vector is shown in formula (9):

[0117] (9),

[0118] in, express The transpose of represents the training parameters, express function.

[0119] Similarly, the global mean is calculated for the edge features to obtain the center vector of the edge features in the graph, as shown in formula (10):

[0120] (10),

[0121] Then calculate each edge and The attention weights are weighted and summed as shown in formula (11):

[0122] (11),

[0123] in, This operation makes the edge features occupy the corresponding position in the graph embedding, thereby supplementing the interactive information that the node features cannot cover.

[0124] Finally, the node-level vector and the vectors at the edge level The complete graph representation is obtained by concatenation, as shown in formula (12):

[0125] (12),

[0126] in, Represents a vector-level concatenation operation. Through this representation, both node and edge features in the graph are fully fused and focused, forming a vector that highly summarizes the entire graph. The final complete graph representation has 1024 dimensions.

[0127] According to an embodiment of the present invention, encoding the text description of the private image to obtain the text encoding includes: generating a text description of the private image and performing a mapping operation on the word vectors in the text description to obtain a word vector sequence; performing a multi-directional hidden state calculation operation on the word vector sequence, and averaging the multiple calculation results to obtain a text encoding with a word embedding representation.

[0128] The text encoding process is further described in detail below through specific implementation methods.

[0129] Generate text descriptions for images using a pre-trained image description generation model (Image Caption) ,in Indicates that the text description contains words; in specific application scenarios, any pre-trained image description generation model can be selected to generate text descriptions. This invention selects the pre-trained BLIP-2 model to automatically generate image descriptions. The BLIP-2 framework consists of three core components: an image encoder (Image Encoder), a query transformer (Q-Former), and a large language model (LLM). The Q-Former acts as a trainable bridge for feature integration between the fixed-weight image encoder and the large language model; in specific implementation, the image encoder uses ViT-G, while the LLM is based on OPT 2.7B. Through this pre-trained framework, contextual information such as objects, relationships between objects, and scenes can be extracted from images, and the corresponding initial text description can be output, providing basic semantic information on the text side for subsequent multimodal feature fusion.

[0130] Text feature encoding process: text description Each word in First, the word embedding layer is mapped to ,in is one-hot encoding, is the parameter matrix. Input into the bidirectional GRU to calculate the hidden layer state from left to right and from right to left, as shown in formulas (13) and (14):

[0131] (13),

[0132] (14),

[0133] The final word embedding representation can be taken as the average of the hidden layer states in two directions, as shown in formula (15):

[0134] (15).

[0135] According to an embodiment of the present invention, the above-mentioned updating of text encoding based on the cross-modal attention mechanism to obtain text modal features includes: calculating the correlation score between the updated features of the nodes and the updated features of the edges and the word vectors in the text description; normalizing the updated features of the nodes and the updated features of the edges and using the correlation score to obtain the attention distribution of the word vectors in the text description; using the attention distribution and the normalized updated features of the nodes and the normalized updated features of the edges to obtain an initial cross-modal representation of the text description; and performing averaging and aggregation operations on the initial cross-modal representation to obtain the text modal features of the private image.

[0136] The following is a further detailed description of the process of acquiring the above-mentioned text modal features through a specific implementation method.

[0137] Cross-modal attention updates text features: defining node and edge features of GCN output in the image branch For node and edge features , calculate its difference with the text word The score between is shown in formula (16):

[0138] (16),

[0139] in, and represents the learnable parameters, is the bias, is the row vector, is the activation function.

[0140] For each text word , for all graph nodes / edges Perform softmax normalization to obtain the attention distribution, as shown in formula (17):

[0141] (17).

[0142] For text words , that is, with the The image feature representation of text words is shown in formula (18):

[0143] (18).

[0144] Finally, The dimensions are averaged and aggregated to obtain the overall vector of the text, as shown in formulas (19) and (20):

[0145] (19),

[0146] (20),

[0147] in Represents the MLP layer. The final text feature is 1024-dimensional.

[0148] After obtaining the final representation of the private image and text, the two are concatenated to obtain the fusion vector The concatenated vectors are fed into the privacy image detection model for classification and detection to obtain corresponding classification or detection information.

[0149] According to a second aspect of the present invention, a training method for a privacy-preserving image detection model is provided, comprising: generating scene graph samples of privacy-preserving image samples, and extracting label feature samples and visual feature samples of the scene graph samples using the privacy-preserving image detection model; fusing the label feature samples and the visual feature samples, and updating the scene graph samples based on the obtained fused feature samples to obtain updated scene graph samples; calculating visual modal features of the updated scene graph samples using the privacy-preserving image detection model; generating text description samples of the privacy-preserving image samples, and encoding the text description samples using the privacy-preserving image detection model to obtain text feature samples; and performing an update operation on the text feature samples based on a cross-modal attention mechanism using the privacy-preserving image detection model to obtain text modal feature samples. This method comprises the following steps: concatenating and fusing visual modal feature samples and text modal feature samples to obtain multimodal feature samples; processing the multimodal feature samples using a privacy image detection model to obtain classification information samples of the privacy image samples; processing the true value labels and classification information samples of the privacy image samples using a cross-entropy loss function to obtain a loss value, and updating the parameters of the privacy image detection model based on the loss value; iteratively performing operations of acquiring visual modal features and text modal features of the privacy image samples, calculating the loss value, and updating the parameters of the privacy image detection model until preset training conditions are met, thereby obtaining a trained privacy image detection model, wherein the trained privacy image detection model is used to execute the above-mentioned privacy image detection method based on multimodal feature fusion.

[0150] According to an embodiment of the present invention, the above-mentioned privacy image detection model includes a pre-trained scene graph generation module, a graph convolutional neural network, a pre-trained convolutional neural network, a pre-trained image text description generation module and a cross-modal text encoding module.

[0151] According to an embodiment of the present invention, the above-mentioned pre-trained scene graph generation module includes a RelTR model; wherein, the pre-trained image text description generation module includes an image encoder, a query transformer and a large language model unit.

[0152] The following is a specific implementation method and combined with the attached Figure 3 The training process of the privacy-preserving image detection model provided by the present invention and the application process of the trained privacy-preserving image detection model are further described in detail.

[0153] Figure 3 1 is an architecture diagram and data processing flow chart of a privacy image detection model according to an embodiment of the present invention.

[0154] First, we constructed a dataset for training the privacy image detection model. We used crawler technology to collect user-shared images from the internet, including pictures shared on social networking sites and shared with online friends, filtering out low-resolution and damaged images. We then had multiple volunteers label the privacy category of each image. If a volunteer believed that an image contained private content and was unwilling to share it on the internet, it would be labeled as "private," otherwise it would be labeled as "public."

[0155] like Figure 3 As shown, the method mainly includes the model training process and the model application process, namely: by constructing a privacy image classification or detection dataset; constructing a scene graph representation of the images in the dataset; extracting the features of nodes and edges in the scene graph and updating the features using a graph convolutional neural network; using a graph embedding layer to extract the feature representation of the scene graph; generating a text description of the image and extracting text features; using a cross-modal attention layer to perform fine-grained alignment of the text and scene graph; splicing image features and text features and feeding them into the classification layer to obtain privacy detection results. The method provided by the present invention can fully explore the privacy content of images and improve the accuracy of privacy image detection through the multimodal fusion of image and text features.

[0156] Based on the above-mentioned privacy image detection method based on multimodal feature fusion, the present invention also provides a privacy image detection device based on multimodal feature fusion. Figure 4 The device is described in detail.

[0157] Figure 4 4 is a structural block diagram of a privacy image detection device based on multimodal feature fusion according to an embodiment of the present invention.

[0158] like Figure 4 As shown, the privacy image detection device 400 based on multimodal feature fusion includes a label and visual feature extraction module 410, a label and visual feature fusion module 420, a fusion feature update module 430, a visual module acquisition module 440, a text modality feature acquisition module 450 and a multimodal fusion and detection module 460.

[0159] The label and visual feature extraction module 410 is used to extract label features and visual features of nodes in the scene graph of the private image, as well as label features and visual features of edges in the scene graph. In one embodiment, the label and visual feature extraction module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0160] The label and visual feature fusion module 420 is used to fuse the label features and visual features of the nodes in the scene graph to obtain node fusion features, and to fuse the label features and visual features of the edges in the scene graph to obtain edge fusion features; in one embodiment, the label and visual feature fusion module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0161] The fusion feature update module 430 is used to update the scene graph based on the node fusion features and the edge fusion features, and calculate the visual aggregation vectors of the nodes and edges in the updated scene graph respectively; in one embodiment, the fusion feature update module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0162] The visual module acquisition module 440 is used to fuse the visual aggregation vectors of the nodes and edges in the updated scene graph to obtain visual modal features. In one embodiment, the visual module acquisition module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0163] The text modal feature acquisition module 450 is used to encode the text description of the private image to obtain a text encoding, and to update the text encoding based on the cross-modal attention mechanism to obtain a text modal feature. In one embodiment, the text modal feature acquisition module 450 can be used to perform the operation S250 described above, which will not be repeated here.

[0164] Multimodal fusion and detection module 460 is configured to combine and fuse visual modal features with textual modal features to generate multimodal features, and then process the multimodal features to obtain detection information for the private image. In one embodiment, multimodal fusion and detection module 460 can be configured to perform operation S260 described above, and will not be further described here.

[0165] According to an embodiment of the present invention, any multiple modules among the label and visual feature extraction module 410, the label and visual feature fusion module 420, the fusion feature update module 430, the visual module acquisition module 440, the text modality feature acquisition module 450, and the multimodal fusion and detection module 460 can be combined into a single module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present invention, at least one of the label and visual feature extraction module 410, the label and visual feature fusion module 420, the fusion feature update module 430, the visual module acquisition module 440, the text modality feature acquisition module 450, and the multimodal fusion and detection module 460 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable manner of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the label and visual feature extraction module 410, the label and visual feature fusion module 420, the fusion feature update module 430, the visual module acquisition module 440, the text modality feature acquisition module 450, and the multimodal fusion and detection module 460 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0166] Figure 5 4 is a block diagram of an electronic device suitable for implementing a privacy image detection method based on multimodal feature fusion according to an embodiment of the present invention.

[0167] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 502 or programs loaded from a storage unit 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0168] Various programs and data required for the operation of the electronic device 500 are stored in the RAM 503. The processor 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The processor 501 executes the programs in the ROM 502 and / or RAM 503 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and RAM 503. The processor 501 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0169] According to an embodiment of the present invention, electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. Electronic device 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or modem. Communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 510 as needed, so that computer programs read from the removable media can be installed into storage section 508 as needed.

[0170] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0171] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above, and / or one or more memories other than ROM 502 and RAM 503.

[0172] Embodiments of the present invention also include a computer program product, comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the private image detection method based on multimodal feature fusion provided in an embodiment of the present invention.

[0173] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when the computer program is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0174] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 509, and / or installed from a removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0175] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509 and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0176] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0177] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0179] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A privacy image detection method based on multimodal feature fusion, characterized in that: The method comprises: Generating a scene graph of a private image, wherein the scene graph includes a set of objects, a set of object detection frames, and a set of predicate relationships between objects; Obtaining label features of nodes in the scene graph using one-hot encoding of a set of objects in the scene graph, and obtaining label features of edges in the scene graph using one-hot encoding of a set of predicate relationships between objects in the scene graph; obtaining a target image region corresponding to the object set in the scene graph by cropping the object detection frame set in the scene graph, and performing feature extraction on the target image region to obtain visual features of the nodes in the scene graph; Obtaining visual features of edges at a relational level in the scene graph by defining intersection regions between object detection boxes in the scene graph; fusing the label features and visual features of the nodes in the scene graph to obtain node fusion features, and fusing the label features and visual features of the edges in the scene graph to obtain edge fusion features; Using the node fusion feature as an initial node representation of the scene graph, and using the edge fusion feature as an initial edge representation of the scene graph; performing multiple rounds of updating node representations using the initial node representations to obtain updated features of the nodes in the scene graph; performing multiple rounds of updating edge representations using the initial edge representations to obtain updated features of edges in the scene graph; performing a mean operation on the updated features of the node to obtain an average feature of the node, and performing a correlation calculation operation, a normalization and a weight assignment operation, and a weighted aggregation operation based on an attention mechanism on the updated features and the average feature of the node to obtain a visual aggregation vector of the node; performing a mean operation on the updated features of the edge to obtain an average feature of the edge, and performing a correlation calculation operation, a normalization and a weight assignment operation, and a weighted aggregation operation based on an attention mechanism on the updated features and the average feature of the edge to obtain a visual aggregation vector of the edge; The visual aggregation vectors of nodes and edges in the updated scene graph are fused to obtain visual modality features; Encoding the text description of the private image to obtain a text encoding, and updating the text encoding based on a cross-modal attention mechanism to obtain a text modality feature; The visual modal features and the text modal features are spliced and fused to obtain multimodal features, and the multimodal features are processed to obtain detection information of the private image.

2. The method according to claim 1, characterized in that The label features and visual features of the nodes in the scene graph are fused to obtain node fusion features, and the label features and visual features of the edges in the scene graph are fused to obtain edge fusion features, including: Performing vector splicing on the label feature and the visual feature of the node, performing mapping processing on the vector splicing result of the node, and activating the mapping processing result of the node to obtain the fusion feature of the node; Performing vector splicing on the label feature and the visual feature of the edge, performing mapping processing on the vector splicing result of the edge, and activating the mapping processing result of the edge to obtain the fusion feature of the edge.

3. The method according to claim 1, characterized in that Encoding the text description of the private image to obtain the text encoding includes: Generating a text description of the private image, and performing a mapping operation on word vectors in the text description to obtain a word vector sequence; A multi-directional hidden state calculation operation is performed on the word vector sequence, and multiple calculation results obtained are averaged to obtain a text encoding with word embedding representation.

4. The method according to claim 3, characterized in that The text encoding is updated based on the cross-modal attention mechanism, and the obtained text modality features include: Calculating a correlation score between the updated features of the node and the updated features of the edge and the word vector in the text description; Normalizing the updated features of the nodes and the updated features of the edges and obtaining an attention distribution of word vectors in the text description using the relevance scores; Obtaining an initial cross-modal representation of the text description using the attention distribution and the normalized updated features of the nodes and the normalized updated features of the edges; An averaging operation and a pooling operation are performed on the initial cross-modal representation to obtain a text modality feature of the private image.

5. A method for training a private image detection model for executing the method according to any one of claims 1 to 4, characterized in that: The method comprises: Generate a scene graph sample of the private image sample, and extract a label feature sample and a visual feature sample of the scene graph sample using a private image detection model; Fusing the label feature samples and the visual feature samples, and updating the scene graph samples based on the obtained fused feature samples to obtain updated scene graph samples; Calculating visual modality features of the updated scene graph samples using the privacy image detection model; generating a text description sample of the private image sample, and encoding the text description sample using the private image detection model to obtain a text feature sample; Using the private image detection model, an update operation based on a cross-modal attention mechanism is performed on the text feature sample to obtain a text modality feature sample; splicing and fusing the visual modality feature sample and the text modality feature sample to obtain a multimodal feature sample; Processing the multimodal feature sample using the private image detection model to obtain a classification information sample of the private image sample; Processing the true value labels and classification information samples of the private image samples using a cross entropy loss function to obtain a loss value, and updating parameters of the private image detection model based on the loss value; Iteratively performing the operations of acquiring the visual modal features and the textual modal features of the private image sample, calculating the loss value, and updating the parameters of the private image detection model until preset training conditions are met, thereby obtaining a trained private image detection model, wherein the trained private image detection model is used to execute the method according to any one of claims 1 to 4.

6. The method according to claim 5, characterized in that The privacy image detection model includes a pre-trained scene graph generation module, a graph convolutional neural network, a pre-trained convolutional neural network, a pre-trained image text description generation module and a cross-modal text encoding module.

7. The method according to claim 6, wherein: The pre-trained scene graph generation module includes a RelTR model; The pre-trained image text description generation module includes an image encoder, a query transformer and a large language model unit.

Images