Fine-grained sensitive information identification method based on modal discriminator

Abstract

The invention discloses a fine-grained sensitive information identification method based on a modal discriminator, and relates to the technical field of information security, and the method comprises the following steps: making a data set, and carrying out the preprocessing; sending the preprocessed data set into a modal discriminator, and dividing the data set into a sensitive data set containing sensitive pictures and a general data set containing general pictures; single-mode sensitive information identification: text in the general data set is sent to a BERT model for sensitive information identification; multi-modal sensitive information identification: for a sensitive data set, text and pictures in the sensitive data set are sent to the corresponding model for multi-modal fusion, and sensitive information is identified; and outputting a sensitive information identification result. According to the method, the accuracy and efficiency of sensitive information identification are remarkably improved, powerful technical support is provided for data security and privacy protection, good adaptability and flexibility are achieved, and efficient identification performance can still be kept in various complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of information security, and in particular to a sensitive information identification method. Background Art

[0002] In the digital information age, the popularity of social media has brought people countless conveniences and global connections. Through social media, people can keep in touch with their friends and family, share their daily lives, and obtain various information and entertainment. Without effective privacy protection, users may face the risk of their personal information being accessed or abused by others.

[0003] The study of multimodal identification of sensitive information in online social networks is an important research area for the automatic identification of sensitive information in online social networks. This can be used for a variety of purposes, such as detecting and removing harmful content, protecting user privacy, and developing more effective moderation tools. By identifying sensitive information such as personal information, potentially offensive or harmful content, or illegal activities, this can help prevent unauthorized parties from obtaining users' personal data, such as financial information or login credentials, and using it for malicious purposes, such as identity theft or fraud. In addition, it can be used to monitor users' activities on the network and detect any suspicious behavior, such as attempts to access personal information without permission. By detecting and blocking unauthorized access to personal information, it can help protect users' privacy and protect their data from potential threats.

[0004] The widespread use of heterogeneous networks has spawned a large number of diverse and large-scale data types, which poses a severe challenge to traditional sensitive information identification technologies. Traditional methods are often unable to cope with multimodal data. For example, the text-based sensitive information identification method, device, electronic device and storage medium disclosed in the invention patent with application number 202211478845.2 is a processing technology that only targets specific modal data. In addition, although some traditional methods attempt to uniformly process data through multimodal fusion, the effect is still insufficient. Therefore, new methods need to be studied to more effectively tap the potential of data and enhance privacy protection capabilities in social networks. The deep connection between text and images is explored, and the mode of data processing is selected on this basis to quickly identify the user's sensitive information. Summary of the invention

[0005] In view of the technical problem that traditional sensitive information identification methods are difficult to process multimodal data, the present invention proposes a fine-grained sensitive information identification method based on modal discriminator.

[0006] In order to achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] A fine-grained sensitive information identification method based on modality discriminator comprises the following steps:

[0008] S1. Data collection and data preprocessing: Collect data from social networks including text and pictures corresponding to the text, generate a sensitive information list based on the collected data and user personalized needs, mark each piece of data according to the sensitive information list, organize the data into a one-to-one correspondence between text, picture and sensitive description, create a data set and perform preprocessing;

[0009] S2. Send the preprocessed data set to a modal discriminator, use the modal discriminator to identify the images in the preprocessed data set, and classify the images in the preprocessed data set into general images and sensitive images. According to the classification result, the data set is divided into a sensitive data set containing sensitive images and a general data set containing general images.

[0010] S3, single-modal sensitive information identification: the text in the general data set is fed into the BERT model for sensitive information identification;

[0011] S4. Multimodal sensitive information identification: For sensitive data sets, the text and images in the sensitive data sets are sent to the corresponding model for multimodal fusion and identification of sensitive information;

[0012] S5. Output the sensitive information identification results obtained through steps S3 and S4.

[0013] Preferably, the data set consists of a picture set and a sensitive description set, and the pictures and the texts of the sensitive descriptions correspond one to one; the preprocessing includes adjusting the pixels of the pictures and converting the texts of the sensitive descriptions into text strings.

[0014] Preferably, the modality identifier is composed of an input layer, a feature extraction layer, a contrastive learning layer, a decision layer and an output layer connected in sequence. The images and texts in the preprocessed data set enter the feature extraction layer through the input layer, the feature extraction layer outputs the embedding vectors of the images and texts, the similarity of the embedding vectors of the images and texts is calculated in the contrastive learning layer, the decision layer makes a modality identification decision based on the similarity and a set threshold, and the output layer outputs the identification result, wherein the feature extraction layer is a CLIP model.

[0015] Preferably, the method in which the feature extraction layer outputs the embedding vectors of pictures and texts is:

[0016] The preprocessed sensitive descriptions and pictures are input into the CLIP model through the input layer. The encoding layer of the CLIP model encodes the sensitive descriptions and pictures to obtain the corresponding semantic vector representations. The text encoder processes the text data, and the image encoder processes the picture data. The two encoders share parameters. The image encoder and the text encoder are mapped to the shared embedding space to obtain the embedding vectors V of the sensitive description text. text (T j) and the image embedding vector V image (I i ):

[0017] V text (T j ) = f text (T j )

[0018] V image (I i ) = f image (I i )

[0019] Among them, f text is the function for extracting text embedding vectors, f image A function for extracting image embedding vectors.

[0020] Preferably, the method for calculating the similarity of the embedding vectors of the image and the text in the contrastive learning layer is:

[0021]

[0022] similarity(V image (I i ),V text (T j )) is the similarity between the sensitive description embedding vector and the image embedding vector.

[0023] Preferably, the method by which the decision layer makes a modality identification decision based on the similarity and a set threshold is:

[0024] Given a similarity threshold θ, if the similarity between the sensitive description embedding vector and the image embedding vector is similarity(V image (I i ),V text (T j )) is greater than the threshold θ, then image I i Belongs to sensitive description text T j Category, otherwise picture I i It does not belong to sensitive description text T j Categories:

[0025]

[0026] Preferably, the method by which the output layer outputs the identification result is:

[0027] Define a Softmax image classifier with N categories. The Softmax image classifier converts similarity into category probability distribution P(y|I i ,T j ):

[0028]

[0029] in, is the embedding vector of the text description associated with category N.

[0030] After the above steps, the output is the multi-classification result of each image category. The binary classifier is used to convert the image category into a binary category: general image, sensitive image, and complete the multi-classification conversion into a binary category to obtain the identification result; the output of the binary classifier is the predicted probability of whether the image belongs to the sensitive category

[0031]

[0032] Among them, W is the weight matrix, b is the bias term, and Sigmoid is the Sigmoid function.

[0033] Preferably, the Softmax image classifier calculates the contrast loss through a cross entropy loss function:

[0034] For the positive sample pair The predicted similarity is

[0035]

[0036] For negative samples The predicted similarity is

[0037]

[0038] The sum of the cross entropy losses for the Softmax image classifier is:

[0039]

[0040] Preferably, the loss function of the binary classifier is:

[0041]

[0042] Among them, y sen =1 indicates positive class, y sen =0 indicates negative class;

[0043] The overall loss function of the Softmax image classifier and the binary classifier is: Loss total =Loss+λ×Loss classifier .

[0044] Preferably, the method for multimodal sensitive information identification is: preprocessing the text and images in the sensitive data set respectively and sending them to the BERT model and the VIT model for feature extraction, outputting vector representations of the text and the image, weighted averaging the outputs of the BERT model and the VIT model, fusing them through a fully connected layer to form fused features; sending the fused features to a fully connected layer classifier, and outputting the sensitive information category corresponding to each piece of data through the fully connected layer classifier.

[0045] Compared with the prior art, the present invention has the following beneficial effects: the present invention significantly improves the accuracy and efficiency of sensitive information identification, and provides strong technical support for data security and privacy protection. The modal discriminator can more finely distinguish the data features of different modalities, thereby achieving accurate capture of sensitive information. At the same time, the fine-grained processing method enables the system to pay attention to subtle differences in the data and avoid missing any potential sensitive information. In addition, the present invention has good adaptability and flexibility. With the continuous changes in the data environment and the diversification of sensitive information forms, the modal discriminator can be updated and optimized to adapt to new identification requirements. Therefore, the present invention can still maintain efficient recognition performance in various complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a flow chart of the present invention.

[0048] Figure 2 Schematic diagram of the structure of the modal discriminator of the present invention.

[0049] Figure 3 It is a structural diagram of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, a fine-grained sensitive information identification method based on modality discriminator includes the following steps:

[0052] S1. Data collection and data preprocessing: Collect data from social networks including text and pictures corresponding to the text, generate a sensitive information list based on the collected data and user personalized needs, and mark each piece of data according to the sensitive information list for subsequent experimental training and verification; organize the data into a one-to-one correspondence between text, picture, and sensitive description, create a data set, and perform preprocessing.

[0053] 1.1 Prepare the data set, legally collect dynamic data from social networks, and synthesize the data set. Each piece of data includes the user ID, the text and image of the social dynamics posted, and the sensitive information involved in each dynamic. The sensitive information is collected into a sensitive information list (location, sex, religion...).

[0054] 1.2 After the data set is prepared, check whether there is sensitive information in the picture of each data, and make a sensitive description description (location, sex, religion...) of the picture according to the user's sensitive information list. The sensitive description of the picture without sensitive information is (NULL) to prepare data for the subsequent modality identifier.

[0055] 1.3 The dataset consists of a picture set image{I1,I2,I3,....,I i} and sensitive description set description{T1,T2,T3,....,T j}, where the image and the sensitive description text are one-to-one corresponding, and the number of samples is N.

[0056] 1.4 Preprocess the image set and image sensitive description set in the dataset, adjust the image pixels to 224*224; convert the sensitive description text into a text string to meet the modality discriminator input requirements.

[0057] S2. Send the preprocessed data set to the modal discriminator, use the modal discriminator to identify the images in the preprocessed data set, and determine how the data is used to identify sensitive information. The modal discriminator compares and learns the images with sensitive descriptions to explore the deep connection between images and texts and identify whether there is sensitive information in the images, thereby dividing the images into general images and sensitive images. Based on the identification results, the data set is divided into a sensitive data set containing sensitive images and a general data set containing general images.

[0058] The modality discriminator is composed of an input layer, a feature extraction layer, a contrastive learning layer, a decision layer and an output layer connected in sequence. The images and texts in the preprocessed data set enter the feature extraction layer through the input layer. The feature extraction layer outputs the embedding vectors of the images and texts. The similarity of the embedding vectors of the images and texts is calculated in the contrastive learning layer. The decision layer makes a modality identification decision based on the similarity and the set threshold, and the output layer outputs the identification result. Among them, the feature extraction layer is a CLIP (Contrastive Language-Image Pre-Training) model.

[0059] like Figure 2 As shown in the figure, the preprocessed sensitive description description and image image are input into the CLIP model through the input layer, and the sensitive description and image are encoded by the CLIP model encoding layer to obtain the corresponding semantic vector representation. The CLIP model adopts a bidirectional encoder structure. One Transformer encoder, namely the text encoder, is responsible for processing text data, and the other Vision Transformer encoder, namely the image encoder, is responsible for image data. The two encoders share parameters. The CLIP model contains a shared embedding space for text and images. Both the image encoder and the text encoder are mapped to the shared embedding space to obtain the embedding vector V of the sensitive description text. text (T j ) and the embedding vector V of the image image image (I i ):

[0060] V text (T j ) = f text (T j )

[0061] V image (I i ) = f image (I i )

[0062] Among them, f text is the function for extracting text embedding vectors, f image A function for extracting image embedding vectors.

[0063] Embed the sensitive description text into a vector V text (T j ) and the embedding vector V of the image image image (I i ) is sent to the contrastive learning layer for contrastive learning to obtain the embedding vector V of the sensitive description text text (T j ) and the embedding vector V of the image image image (Ii ) similarity.

[0064] The similarity is calculated by cosine similarity:

[0065]

[0066] similarity(V image (I i ),V text (T j )) is the similarity between the sensitive description embedding vector and the image embedding vector.

[0067] The decision layer makes decisions based on the similarity between the sensitive description embedding vector and the image embedding vector:

[0068] Given a similarity threshold θ, determine the similarity between the sensitive description embedding vector and the image embedding vector similarity (V image (I i ),V text (T j )) and the threshold θ, if the similarity between the sensitive description embedding vector and the image embedding vector is similarity(V image (I i ),V text (T j )) is greater than the threshold θ, then image I i Belongs to sensitive description text T j Category, otherwise picture I i It does not belong to sensitive description text T j The mathematical representation of the category is as follows:

[0069]

[0070] In the output layer, a Softmax image classifier is defined. The number of categories is N (types of sensitive descriptions), which is the dimension of the output layer of the modal discriminator. The Softmax image classifier converts the similarity into a category probability distribution P(y|I i ,T j ), indicating that given image I i and sensitive description text T j The probability of belonging to category y is as follows:

[0071]

[0072] Among them, V image (I i ) is the embedding vector of image i, V text (T j) is the embedding vector of text j, is the embedding vector of the text description associated with category N.

[0073] In the Softmax image classifier, the contrast loss is calculated by the cross entropy loss function to measure the difference between the predicted similarity distribution and the similarity of the actual label. The formula is as follows:

[0074] For the positive sample pair The predicted similarity is

[0075]

[0076] For negative samples The predicted similarity is

[0077]

[0078] The sum of the cross entropy losses for the Softmax image classifier is:

[0079]

[0080] After the above steps, the output is a multi-classification result, which is the category of each picture. The picture category is converted into a binary classification (general picture, sensitive picture), that is, the picture with category NULL is mapped to a general picture, and the pictures of other categories are mapped to sensitive pictures. A fully connected layer is defined in the output layer of the CLIP model, and the number of categories is 2. The probability is calculated by the Sigmoid function and the output categories are 0 and 1. The output of the binary classifier is the probability of whether the predicted picture belongs to the sensitive class. Indicates that given a picture and text, it belongs to the sensitive category. The formula is as follows:

[0081]

[0082] Among them, W is the weight matrix, b is the bias term, and Sigmoid is the Sigmoid function.

[0083] The loss function of the binary classifier is expressed as follows:

[0084]

[0085] Among them, y sen =1 indicates positive class, y sen =0 indicates negative class, is the probability of whether the predicted image belongs to the sensitive class. The overall loss function of the Softmax image classifier and the binary classifier is:

[0086] Loss total=Loss+λ×Loss classifier

[0087] Through the above operations, the multi-classification is converted to binary classification, and the data set is divided into two parts: general data set and sensitive data set. The sensitive information of text in the general data set is processed by single mode, and the sensitive information of pictures and text in the sensitive data set is processed by multi-modal fusion.

[0088] S3, single-modal sensitive information identification: such as Figure 3 As shown in the figure, for general data sets, the text in the general data sets is directly processed in a single mode, and the text in the general data sets with general images is sent to the BERT (Bidirectional Encoder Representations from Transformers) model for sensitive information recognition;

[0089] 3.1 Use a single modality to identify sensitive information in text for general data sets, and annotate the text through a sensitive information list (the annotation content is the sensitive information contained in each piece of data). The general data set contains text text and the sensitive information label label of the text, which is used to process sensitive information identification.

[0090] 3.2 Integer encode the label label to convert it into a digital code, assign a unique integer code (0, 1, 2, 3...) to each category in the sensitive information list list, and the label label is a specific integer.

[0091] 3.3 Perform data preprocessing on the text, segment the input text sequence into word sequences, and convert the segmented text sequence into word embedding vectors through the word embedding model Word2Vec.

[0092] 3.4 Input the preprocessed text into the BERT model. BERT uses the Transformer structure, which calculates the self-attention weight for each position in the input sequence and outputs the hidden state of each position. By combining the two, we can finally obtain a 768-dimensional text representation and the sensitive information corresponding to each piece of data.

[0093] 3.5 Build an additional fully connected layer classifier based on the BERT model. The number of categories of the fully connected layer classifier is the number of labels. The text representation is mapped to the space of classification labels. The output dimension of the fully connected layer classifier is the number of categories of the classifier.

[0094] The above steps are used to complete the single-modal sensitive information identification.

[0095] S4. Multimodal sensitive information identification: For sensitive data sets, the text and images in the sensitive data sets are sent to the corresponding model for multimodal fusion and identification of sensitive information.

[0096] 4.1 Perform multimodal fusion sensitive information identification on the images and texts in the sensitive image dataset, and preprocess the text and images according to steps 3.3 and 1.4 respectively.

[0097] 4.2 After preprocessing, the text and images are sent to the BERT model and VIT (Vision Transformer) model for feature extraction respectively, and the vector representation of the text and image is output. The outputs of the BERT model and the VIT model are weighted averaged and fused through the fully connected layer to form the fused features.

[0098] 4.3 The fused features are sent to the fully connected layer classifier, and the sensitive information category corresponding to each piece of data is output through the fully connected layer classifier.

[0099] S5. Output the sensitive information identification results obtained through steps S3 and S4.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fine-grained sensitive information identification method based on modality discriminator, characterized in that: The following steps are involved: S1. Data collection and data preprocessing: Collect data from social networks including text and pictures corresponding to the text, generate a sensitive information list based on the collected data and user personalized needs, mark each piece of data according to the sensitive information list, organize the data into a one-to-one correspondence between text, picture and sensitive description, create a data set and perform preprocessing; S2. Send the preprocessed data set to a modal discriminator, use the modal discriminator to identify the images in the preprocessed data set, and classify the images in the preprocessed data set into general images and sensitive images. According to the classification result, the data set is divided into a sensitive data set containing sensitive images and a general data set containing general images. S3, single-modal sensitive information identification: the text in the general data set is fed into the BERT model for sensitive information identification; S4. Multimodal sensitive information identification: For sensitive data sets, the text and images in the sensitive data sets are sent to the corresponding model for multimodal fusion and identification of sensitive information; S5. Output the sensitive information identification results obtained through steps S3 and S4.

2. The fine-grained sensitive information identification method based on modality discriminator according to claim 1 is characterized in that: The data set consists of a picture set and a sensitive description set, and the pictures and the text of the sensitive descriptions correspond one to one; the preprocessing includes adjusting the pixels of the pictures and converting the text of the sensitive descriptions into text strings.

3. The fine-grained sensitive information identification method based on modality discriminator according to claim 2 is characterized in that: The modality discriminator is composed of an input layer, a feature extraction layer, a contrastive learning layer, a decision layer and an output layer connected in sequence. The images and texts in the preprocessed data set enter the feature extraction layer through the input layer. The feature extraction layer outputs the embedding vectors of the images and texts. The similarity of the embedding vectors of the images and texts is calculated in the contrastive learning layer. The decision layer makes a modality identification decision based on the similarity and a set threshold. The output layer outputs the identification result, wherein the feature extraction layer is a CLIP model.

4. The fine-grained sensitive information identification method based on modality discriminator according to claim 3 is characterized in that: The method for the feature extraction layer to output the embedding vectors of pictures and texts is: The preprocessed sensitive descriptions and pictures are input into the CLIP model through the input layer. The encoding layer of the CLIP model encodes the sensitive descriptions and pictures to obtain the corresponding semantic vector representations. The text encoder processes the text data, and the image encoder processes the picture data. The two encoders share parameters. The image encoder and the text encoder are mapped to the shared embedding space to obtain the embedding vectors V of the sensitive description text. text (T j ) and the image embedding vector V image (I i ): V text (T j )=f text (T j ) V image (I i )=f image (I i ) Among them, f text is the function for extracting text embedding vectors, f image A function for extracting image embedding vectors.

5. The fine-grained sensitive information identification method based on modal discriminator according to claim 3 or 4 is characterized in that: The method for calculating the similarity of the embedding vectors of the image and the text in the contrastive learning layer is: similarity(V image (I i ),V text (T j )) is the similarity between the sensitive description embedding vector and the image embedding vector.

6. The fine-grained sensitive information identification method based on modality discriminator according to claim 5 is characterized in that: The method by which the decision layer makes a modality identification decision based on the similarity and the set threshold is: Given a similarity threshold θ, if the similarity between the sensitive description embedding vector and the image embedding vector is similarity(V image (I i ),V text (T j )) is greater than the threshold θ, then image I i Belongs to sensitive description text T j Category, otherwise picture I i It does not belong to sensitive description text T j Categories:

7. The fine-grained sensitive information identification method based on modality discriminator according to claim 6 is characterized in that: The method for the output layer to output the identification result is: Define a Softmax image classifier with N categories. The Softmax image classifier converts similarity into category probability distribution P(y|I i ,T j ): Among them, V text (T jN ) is the embedding vector of the text description associated with category N. After the above steps, the output is the multi-classification result of each image category. The binary classifier is used to convert the image category into a binary category: general image, sensitive image, and complete the multi-classification conversion into a binary category to obtain the identification result; the output of the binary classifier is the predicted probability of whether the image belongs to the sensitive category Among them, W is the weight matrix, b is the bias term, and Sigmoid is the Sigmoid function.

8. The fine-grained sensitive information identification method based on modality discriminator according to claim 7 is characterized in that: The Softmax image classifier calculates the contrast loss through the cross entropy loss function: For the positive sample pair The predicted similarity is For negative samples The predicted similarity is The sum of the cross entropy losses for the Softmax image classifier is:

9. The fine-grained sensitive information identification method based on modality discriminator according to claim 8 is characterized in that: The loss function of the binary classifier is: Among them, y sen =1 indicates positive class, y sen =0 indicates negative class; The overall loss function of the Softmax image classifier and the binary classifier is: Loss total =Loss+λ×Loss classifier .

10. The fine-grained sensitive information identification method based on modality discriminator according to claim 1 is characterized in that: The method for multimodal sensitive information identification is as follows: preprocessing the text and images in the sensitive data set and sending them to the BERT model and the VIT model for feature extraction, outputting vector representations of the text and the image, weighted averaging the outputs of the BERT model and the VIT model, fusing them through a fully connected layer to form fused features; sending the fused features to a fully connected layer classifier, and outputting the sensitive information category corresponding to each piece of data through the fully connected layer classifier.

Images