Privacy leakage assessment method and system based on social privacy relationship mining
Through the combination of MTCNN, FaceNet, FP-Tree and Transformer, a social privacy relationship mining model was built, which solved the problem of risk assessment of social hidden relationship privacy leakage on social platforms, and achieved the integration of multimodal information and the accurate assessment of privacy risks.
Patent Information
- Application Number
- CN202411937030.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-04-29
AI Technical Summary
The prior art is difficult to quickly and accurately assess the privacy leakage risks of social hidden relationships on social platforms, and lacks effective integration and mining of multimodal data.
Face detection and recognition are performed through MTCNN and FaceNet, human pose estimation is performed by combining FP-Tree algorithm and YoloV8, intimate relationship classification is used by Transformer, social privacy relationship mining model is constructed, and privacy leakage risks are evaluated.
It has achieved the integration of multimodal information on social platforms, can dig out hidden attributes, accurately evaluate the risks of privacy leakage, and provides a new direction for privacy protection.
Smart Images
Figure CN119903480A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a privacy leakage assessment method and system, and in particular to a privacy leakage assessment method and system based on social privacy relationship mining, belonging to the technical field of data mining and privacy security. Background Art
[0002] Data security has always been a national concern, and the public's awareness of privacy protection has gradually increased. With the development of the Internet, the use of social software has become a daily routine, and the data on social platforms is growing exponentially every day. Platforms such as Facebook, Twitter, WeChat, and QO store videos, pictures, and texts submitted by users. These multimodal data contain a large amount of personal privacy information and pose a risk of privacy leakage. Social relationships are also a type of privacy attribute, which has not received enough attention in many cases. Exposing hidden social relationships may cause personal reputation damage and may also be exploited by attackers. Therefore, it is crucial to accurately and quickly assess the risk of privacy leakage.
[0003] Most of the existing technologies deal with and analyze multimodal explicit attributes, which are used in recommendation systems and retrieval systems. The research on hidden attributes is just starting. A small number of existing studies focus on using hidden attributes to achieve entity recognition or using explicit attributes to discover hidden relationships, while no one has yet touched upon the direction of mining social hidden relationships. The MTCNN algorithm is a face detection and face alignment method based on deep learning. It can simultaneously complete the tasks of face detection and face alignment. Compared with traditional algorithms, it has better performance and faster detection speed. FaceNet is a deep learning model for face recognition and verification, released by Google in 2015. Its main innovation is to embed face images into a high-dimensional vector space, so that similar faces are closer in this space, while different faces are farther away. This representation method allows face recognition tasks (such as face verification, face recognition, and clustering) to be implemented through simple distance metrics. The FP-tree mining algorithm is efficient, saves memory, and does not generate candidate sets, but it is complex to build and may lose information. YoloV8 has high detection accuracy, fast image processing speed, flexible deployment, and also improves small target detection. But it also has disadvantages, such as it may not be robust enough in complex scenes and the detection decisions are not easy to interpret. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to provide a privacy leakage assessment method and system based on social privacy relationship mining that can integrate multimodal information and mine hidden attributes.
[0005] Technical solution: The privacy leakage assessment method based on social privacy relationship mining described in the present invention includes:
[0006] Collect pictures containing faces on social platforms, and use MTCNN and FaceNet to collect face information data from the pictures;
[0007] Based on the face information data, subject alignment is performed in an open environment and a closed environment; the subject alignment means aligning face information data belonging to the same person in different pictures to the person; the closed environment means that there is face information data that can be referenced in the face database, and the open environment means that there is no face information data that can be referenced in the face database;
[0008] Based on the data after subject alignment, the face information appearing in all photos is combined into a list data structure through data mining algorithm, and the frequent items of face information are mined using FP-Tree algorithm;
[0009] Based on the human posture estimation key point detection and target detection algorithm provided by YoloV8, all photo data are processed to obtain the relationship data of each picture;
[0010] Based on the relationship data of each image, the Transformer is used to perform intimacy classification training to obtain a trained relationship classification Transformer model;
[0011] Based on the trained relation classification Transformer model, the original images containing the mined frequent items are classified into frequent item relation categories.
[0012] Use test data to test the model, mine privacy relationships and make coarse-grained category judgments to assess the risk of privacy leakage.
[0013] Furthermore, the subject alignment in a closed environment is specifically performed by comparing the detected and recognized face information data with the face information data in a database, calculating the face similarity, and when the similarity reaches a threshold, determining that they are the same person.
[0014] Furthermore, the subject alignment in an open environment is specifically to mark all faces in the image in sequence, calculate the face similarity in a sequential traversal manner, and complete the subject alignment.
[0015] Furthermore, all faces in the picture are marked in sequence, and face similarity is calculated by traversing in sequence to complete subject alignment, including: detecting all pictures in sequence, marking the faces appearing in the pictures in sequence, and storing the labels in a list structure; traversing the list structure in sequence from the first person backward, and calculating face similarity with the face information data of the next person in sequence. If the similarity reaches a preset threshold, they are identified as the same person, and the mark of the compared person is used to cover the mark of the compared person, so as to achieve alignment of face information data belonging to the same person in a group of data.
[0016] Furthermore, based on the data mining algorithm, the face information appearing in all photos is combined into a list data structure, and the frequent items of face information are mined using the FP-Tree algorithm, including:
[0017] (1) Scan the database and count the frequency of occurrence of each item as support; filter out frequent items according to the set minimum support threshold, and sort the frequent items in descending order of frequency;
[0018] (2) Construct FP-Tree;
[0019] (3) Mining conditional pattern bases: Starting from each frequent item, construct the corresponding conditional pattern base; the conditional pattern base is the set of all paths that contain the item in the FP-Tree; each path represents a pattern, and the items on the path are the items that co-occur with the corresponding frequent item; for each frequent item, after constructing its conditional pattern base, a conditional FP-Tree is constructed based on the pattern base;
[0020] (4) For each frequent item, repeat step (3), build a new FP-Tree from the conditional pattern base, and recursively mine frequent items until there are no more frequent items to mine;
[0021] (5) Return the result.
[0022] Furthermore, the constructing of FP-Tree includes:
[0023] Build an empty tree and create an empty root node;
[0024] Insert transactions. For each transaction, build a tree according to the sorted frequent item order. Specifically: for each item in the transaction, if the item already exists in the tree, add 1 to the node counter of the item. If the item does not exist in the tree, create a new node in the tree.
[0025] In the tree, the value of a node represents an item, and the counter of the node represents the frequency of the item under that path.
[0026] Furthermore, the human body posture estimation key point detection and target detection algorithm provided in YoloV8 processes all the photo data contained in the research object, including:
[0027] Calculate C by combining and arranging the x characters in the picture x 2 Group relational data; each group of relational data contains the Euclidean distance between the image item information and the key points of the human body, and the structure is [item type, Euclidean distance between the nose tip of person x and the nose tip of person x+1, Euclidean distance between the left shoulder and neck of person x and the right shoulder and neck of person x+1, Euclidean distance between the right shoulder and neck of person x and the left shoulder and neck of person x+1, Euclidean distance between the left hand of person x and the right hand of person x+1, Euclidean distance between the right hand of person x and the left hand of person x+1, Euclidean distance between the left hip of person x and the right hip of person x+1, Euclidean distance between the right hip of person x and the left hip of person x+1, picture label, upper left corner coordinate of the face detection frame of person x, upper left corner coordinate of the face detection frame of person x+1]; save the relational data in each picture into a CSV file.
[0028] Furthermore, based on the relationship data of each image, intimate relationship classification training is performed through Transformer to obtain a trained relationship classification Transformer model, including: using the Word2Vec model to extract the relationship data features of each image, using the ResNet101 model to extract the image features of the relationship subject, using the attention mechanism to fuse the features of two different modalities as training data, and using the Transformer containing an Encoder structure to perform intimate relationship classification training, and the intimate relationships include: lovers, besties, parents and children, ordinary friends, strangers, superiors and subordinates, colleagues and couples.
[0029] Based on the same inventive concept, the present invention also provides a privacy leakage assessment system based on social privacy relationship mining, comprising:
[0030] The preprocessing module is used to collect pictures containing faces on social platforms and collect facial information data for face detection and recognition in the pictures through MTCNN and FaceNet;
[0031] An alignment module, used to perform subject alignment in an open environment and a closed environment based on the face information data; the subject alignment means aligning face information data belonging to the same person in different pictures to the person; the closed environment means that there is face information data that can be referenced in the face database, and the open environment means that there is no face information data that can be referenced in the face database;
[0032] The analysis module is used to combine the face information appearing in all photos into a list data structure based on the subject alignment data through the data mining algorithm, and use the FP-Tree algorithm to mine the frequent items of face information;
[0033] The extraction module is used to process all photo data based on the key point detection and target detection algorithm for human posture estimation provided by YoloV8 to obtain the relationship data of each picture;
[0034] The training module is used to perform intimacy classification training through Transformer based on the relationship data of each image to obtain a trained relationship classification Transformer model;
[0035] The classification module is used to classify the relationships of the frequent items in the original images where the frequent items are mined based on the trained relationship classification Transformer model;
[0036] The evaluation module is used to test the model using test data, mine privacy relationships, make coarse-grained category judgments, and assess the risk of privacy leakage.
[0037] Based on the same inventive concept, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the privacy leakage assessment method based on social privacy relationship mining according to any of the above items.
[0038] Beneficial effects: Compared with existing research, the present invention mines social privacy relationships through data mining algorithms, uses Transformer to infer social privacy relationships, mines and infers social privacy relationships to experiment whether posting only images on social platforms can expose one's own social hidden relationships and whether there is a risk of privacy leakage. It can fully integrate multimodal information from various sources, mine hidden attributes, and understand the methods of such privacy attacks, laying the foundation for privacy protection to prevent associated privacy mining in the future, and providing a new direction for the field of privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a flow chart of a method according to an embodiment of the present invention;
[0040] Figure 2 Schematic diagram of FP-Tree in an embodiment of the present invention;
[0041] Figure 3 This is a first photo example diagram of an embodiment of the present invention;
[0042] Figure 4 This is a second photo example diagram of an embodiment of the present invention;
[0043] Figure 5 This is a third photo example of an embodiment of the present invention;
[0044] Figure 6 This is a fourth photo example diagram of an embodiment of the present invention;
[0045] Figure 7 This is a fifth photo example of an embodiment of the present invention;
[0046] Figure 8 This is a sixth photo example of an embodiment of the present invention;
[0047] Fig. 9 This is a seventh photo example diagram of an embodiment of the present invention;
[0048] Fig.10 This is the 8th photo example of the embodiment of the present invention;
[0049] Fig.11 This is a ninth photo example of an embodiment of the present invention;
[0050] Fig.12 This is the 10th photo example of the embodiment of the present invention;
[0051] Fig.13 This is the 11th photo example of an embodiment of the present invention. DETAILED DESCRIPTION
[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0053] As attached Figure 1 As shown, the privacy leakage assessment method based on social privacy relationship mining in this embodiment includes:
[0054] S1: Collect pictures containing faces on social platforms, and use MTCNN and FaceNet to collect face information data for face detection and recognition in the pictures;
[0055] S2: Based on the face information data, subject alignment is performed in an open environment and a closed environment; the subject alignment means aligning face information data belonging to the same person in different pictures to the person; the closed environment means that there is face information data that can be referenced in the face database, and the open environment means that there is no face information data that can be referenced in the face database;
[0056] S3: Based on the data after subject alignment, the face information appearing in all photos is combined into a list data structure through the data mining algorithm, and the frequent items of face information are mined using the FP-Tree algorithm;
[0057] S4: Based on the human posture estimation key point detection and target detection algorithm provided by YoloV8, all photo data are processed to obtain the relationship data of each picture;
[0058] S5: Based on the relationship data of each image, perform intimacy classification training through Transformer to obtain the trained relationship classification Transformer model;
[0059] S6: Based on the trained relation classification Transformer model, the original images containing the mined frequent items are classified into frequent item relations;
[0060] S7: Use test data to test the model, mine privacy relationships and make coarse-grained category judgments to assess the risk of privacy leakage.
[0061] Specifically: Step S1 includes:
[0062] Step S1.1: Use MTCNN to perform face detection on the image.
[0063] Step S1.2: Use FaceNet to perform face recognition.
[0064] Step S2 includes performing subject alignment in a closed environment, specifically, using the detected and recognized face information data to compare with the face information data in the database, calculating the face similarity, and when the similarity reaches a threshold, determining that they are the same person.
[0065] The subject alignment in an open environment is specifically to mark all faces in the picture in order, calculate the face similarity by traversing in sequence, and complete the subject alignment, including:
[0066] Detect all images in turn, mark the faces that appear in the images in order, and store the labels in a list structure; traverse the list structure from the first person backward in order, and calculate the facial similarity with the facial information data of the next person in turn. If the similarity reaches a preset threshold, they are identified as the same person, and the mark of the compared person is used to overwrite the mark of the compared person, so as to align the facial information data belonging to the same person in a group of data.
[0067] Assuming there are 11 photos, the results after alignment are shown in Table 1:
[0068] Table 1 Body alignment table
[0069]
[0070]
[0071] In step S3, the FP-Tree algorithm is used to mine frequent items of face information. The FP-Tree (Frequent Pattern Tree) algorithm is an efficient frequent pattern mining algorithm, mainly used to discover frequent items. It is a tree-based data structure that can effectively reduce the number of database scans and mine frequent items without generating candidates. The specific steps include:
[0072] (1) Scan the data and count item frequencies: First, scan the database and count the frequency of occurrence (i.e., support) of each item; filter out frequent items (i.e., items with support greater than or equal to the minimum support) according to the set minimum support (Min Support) threshold. These items are called frequent items; sort the frequent items in descending order of frequency to ensure that frequent items are processed first in the subsequent process.
[0073] (2) Construct FP-Tree, such as Figure 2 As shown:
[0074] ①Build an empty tree: create an empty root node;
[0075] ② Insert transaction: For each transaction, build a tree according to the order of the sorted frequent items. The transaction table is shown in Table 2:
[0076] Table 2 Transaction table
[0077] Transactions item T01 [1,2,3] T02 [1,5] T03 [1,7] T04 [1,9,10] T05 [11,1,13,14] T06 [10,1] T07 [9,1] T08 [10,1] T09 [9,1] T10 [11,10,9,1] T11 [1,9,10]
[0078] Specific steps: For each item in the transaction, if the item already exists in the tree, increase the node counter of the item by 1; if the item does not exist in the tree, create a new node in the tree.
[0079] In the tree, the value of the node represents the item, and the counter of the node represents the frequency of the item in the path. The frequency is calculated and the minimum support is set to 2, as shown in Table 3:
[0080] Table 3 Frequency table
[0081]
[0082]
[0083] Rearrange each item according to frequency, as shown in Table 4:
[0084] Table 4 Sorting table
[0085] Transactions item New Item T01 [1,2,3] [1] T02 [1,5] [1] T03 [1,7] [1] T04 [1,9,10] [1,9,10] T05 [11,1,13,14] [1,11] T06 [10,1] [1,10] T07 [9,1] [1,9] T08 [10,1] [1,10] T09 [9,1] [1,9] T10 [11,10,9,1] [1,9,10,11] T11 [1,9,10] [1,9,10]
[0086] (3) Mining conditional pattern base: After the FP-Tree is constructed, frequent pattern mining is required; starting from each frequent item, its conditional pattern base is constructed. As shown in Table 5:
[0087] Table 5 Conditional mode base correspondence table
[0088] item Conditional Pattern Base 11 {1,9,10：1}、{1：1} 10 {1,9：3}、{1：2} 9 {1：5} 1
[0089] The conditional pattern base refers to the set of all paths that contain the item in the FP-Tree. Each path represents a pattern, and the items on the path are the items that co-occur with the frequent item. For each frequent item, after constructing its conditional pattern base, a conditional FP-Tree is constructed based on the pattern base.
[0090] (4) Recursively mine frequent patterns: For each frequent item, repeat step 3, that is, build a new FP-Tree from the conditional pattern base and recursively mine frequent items; as shown in Table 6:
[0091] Table 6 Frequent item table
[0092]
[0093]
[0094] Each recursion will mine more frequent items through the conditional pattern base until there are no more frequent items to mine.
[0095] (5) Return results: Finally, all frequent items are mined and no candidate generation is required. These frequent items are extracted based on the transaction patterns in the original database.
[0096] Furthermore, the human body posture estimation key point detection and target detection algorithm provided in YoloV8 processes all the photo data contained in the research object, including:
[0097] Calculate C by combining and arranging the x characters in the picture x 2 Group relational data; each group of relational data contains the Euclidean distance between the image item information and the key points of the human body, and the structure is [item type, Euclidean distance between the nose tip of person x and the nose tip of person x+1, Euclidean distance between the left shoulder and neck of person x and the right shoulder and neck of person x+1, Euclidean distance between the right shoulder and neck of person x and the left shoulder and neck of person x+1, Euclidean distance between the left hand of person x and the right hand of person x+1, Euclidean distance between the right hand of person x and the left hand of person x+1, Euclidean distance between the left hip of person x and the right hip of person x+1, Euclidean distance between the right hip of person x and the left hip of person x+1, picture label, upper left corner coordinate of the face detection frame of person x, upper left corner coordinate of the face detection frame of person x+1]; save the relational data in each picture into a CSV file.
[0098] In step S4, the trained relationship classification Transformer model is obtained, including: using the Word2Vec model to extract the relationship data features of each image, using the ResNet101 model to extract the image features of the relationship subject, using the attention mechanism to fuse the features of two different modalities as training data, and performing intimate relationship classification training through a Transformer containing an Encoder structure. The intimate relationships include: lovers, besties, parents and children, ordinary friends, strangers, superiors and subordinates, colleagues and couples.
[0099] Based on the same inventive concept, this embodiment also provides a privacy leakage assessment system based on social privacy relationship mining, including:
[0100] The preprocessing module is used to collect pictures containing faces on social platforms and collect facial information data for face detection and recognition in the pictures through MTCNN and FaceNet;
[0101] An alignment module, used to perform subject alignment in an open environment and a closed environment based on the face information data; the subject alignment means aligning face information data belonging to the same person in different pictures to the person; the closed environment means that there is face information data that can be referenced in the face database, and the open environment means that there is no face information data that can be referenced in the face database;
[0102] The analysis module is used to combine the face information appearing in all photos into a list data structure based on the subject alignment data through the data mining algorithm, and use the FP-Tree algorithm to mine the frequent items of face information;
[0103] The extraction module is used to process all photo data based on the key point detection and target detection algorithm for human posture estimation provided by YoloV8 to obtain the relationship data of each picture;
[0104] The training module is used to perform intimacy classification training through Transformer based on the relationship data of each image to obtain a trained relationship classification Transformer model;
[0105] The classification module is used to classify the relationships of the frequent items in the original images where the frequent items are mined based on the trained relationship classification Transformer model;
[0106] The evaluation module is used to test the model using test data, mine privacy relationships, make coarse-grained category judgments, and assess the risk of privacy leakage.
[0107] Based on the same inventive concept, this embodiment also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the privacy leakage assessment method based on social privacy relationship mining according to any of the above items.
[0108] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A privacy leakage assessment method based on social privacy relationship mining, characterized in that: include: Collect pictures containing faces on social platforms, and use MTCNN and FaceNet to collect face information data from face detection and recognition in the pictures; Based on the face information data, subject alignment is performed in an open environment and a closed environment; the subject alignment means aligning face information data belonging to the same person in different pictures to the person; the closed environment means that there is face information data that can be referenced in the face database, and the open environment means that there is no face information data that can be referenced in the face database; Based on the data after subject alignment, the face information appearing in all photos is combined into a list data structure through data mining algorithm, and the frequent items of face information are mined using FP-Tree algorithm; Based on the human posture estimation key point detection and target detection algorithm provided by YoloV8, all photo data are processed to obtain the relationship data of each picture; Based on the relationship data of each image, the Transformer is used to perform intimacy classification training to obtain a trained relationship classification Transformer model; Based on the trained relation classification Transformer model, the original images containing the mined frequent items are classified into frequent item relation categories. Use test data to test the model, mine privacy relationships and make coarse-grained category judgments to assess the risk of privacy leakage.
2. The privacy leakage assessment method based on social privacy relationship mining according to claim 1 is characterized in that: The subject alignment in a closed environment is specifically performed by comparing the detected and recognized face information data with the face information data in a database, calculating the face similarity, and when the similarity reaches a threshold, determining that they are the same person.
3. The privacy leakage assessment method based on social privacy relationship mining according to claim 1 is characterized in that: The subject alignment in an open environment is specifically to mark all faces in the image in sequence, calculate the face similarity in a sequential traversal manner, and complete the subject alignment.
4. The privacy leakage assessment method based on social privacy relationship mining according to claim 3 is characterized in that: The method marks all faces in the picture in sequence, calculates face similarity by traversing the pictures in sequence, and completes subject alignment, including: detecting all pictures in sequence, marking faces appearing in the pictures in sequence, and storing the labels in a list structure; traversing the list structure in sequence from the first person backward, calculating face similarity with the face information data of the next person in sequence, and if the similarity reaches a preset threshold, it is determined that they are the same person, and the mark of the compared person is used to cover the mark of the compared person, so as to realize the alignment of face information data belonging to the same person in a group of data.
5. The privacy leakage assessment method based on social privacy relationship mining according to claim 4 is characterized in that: Based on the data mining algorithm, the face information appearing in all photos is combined into a list data structure, and the frequent items of face information are mined using the FP-Tree algorithm, including: (1) Scan the database and count the frequency of occurrence of each item as support; filter out frequent items according to the set minimum support threshold, and sort the frequent items in descending order of frequency; (2) Construct FP-Tree; (3) Mining conditional pattern bases: Starting from each frequent item, construct the corresponding conditional pattern base; the conditional pattern base is the set of all paths that contain the item in the FP-Tree; each path represents a pattern, and the items on the path are the items that co-occur with the corresponding frequent item; for each frequent item, after constructing its conditional pattern base, a conditional FP-Tree is constructed based on the pattern base; (4) For each frequent item, repeat step (3), build a new FP-Tree from the conditional pattern base, and recursively mine frequent items until there are no more frequent items to mine; (5) Return the result.
6. The privacy leakage assessment method based on social privacy relationship mining according to claim 1 is characterized in that: The construction of FP-Tree includes: Build an empty tree and create an empty root node; Insert transactions. For each transaction, build a tree according to the sorted frequent item order. Specifically: for each item in the transaction, if the item already exists in the tree, add 1 to the node counter of the item. If the item does not exist in the tree, create a new node in the tree. In the tree, the value of a node represents an item, and the counter of the node represents the frequency of the item under that path.
7. The privacy leakage assessment method based on social privacy relationship mining according to claim 1 is characterized in that: The human body posture estimation key point detection and target detection algorithm provided in YoloV8 processes all the photo data contained in the research object, including: Calculate C by combining and arranging the x characters that appear in the picture x 2 Group relational data; each group of relational data contains the Euclidean distance between the image item information and the key points of the human body, and the structure is [item type, Euclidean distance between the nose tip of person x and the nose tip of person x+1, Euclidean distance between the left shoulder and neck of person x and the right shoulder and neck of person x+1, Euclidean distance between the right shoulder and neck of person x and the left shoulder and neck of person x+1, Euclidean distance between the left hand of person x and the right hand of person x+1, Euclidean distance between the right hand of person x and the left hand of person x+1, Euclidean distance between the left hip of person x and the right hip of person x+1, Euclidean distance between the right hip of person x and the left hip of person x+1, picture label, upper left corner coordinate of the face detection frame of person x, upper left corner coordinate of the face detection frame of person x+1]; save the relational data in each picture into a CSV file.
8. The privacy leakage assessment method based on social privacy relationship mining according to claim 1 is characterized in that: The method is based on the relationship data of each picture, and performs intimate relationship classification training through Transformer to obtain a trained relationship classification Transformer model, including: using the Word2Vec model to extract the relationship data features of each picture, using the ResNet101 model to extract the image features of the relationship subject, using the attention mechanism to fuse the features of two different modalities as training data, and performing intimate relationship classification training through a Transformer containing an Encoder structure. The intimate relationships include: lovers, besties, parents and children, ordinary friends, strangers, superiors and subordinates, colleagues and couples.
9. A privacy leakage assessment system based on social privacy relationship mining, characterized in that: include: The preprocessing module is used to collect pictures containing faces on social platforms and collect facial information data for face detection and recognition in the pictures through MTCNN and FaceNet; An alignment module, used to perform subject alignment in an open environment and a closed environment based on the face information data; the subject alignment means aligning face information data belonging to the same person in different pictures to the person; the closed environment means that there is face information data that can be referenced in the face database, and the open environment means that there is no face information data that can be referenced in the face database; The analysis module is used to combine the face information appearing in all photos into a list data structure based on the subject alignment data through the data mining algorithm, and use the FP-Tree algorithm to mine the frequent items of face information; The extraction module is used to process all photo data based on the key point detection and target detection algorithm for human posture estimation provided by YoloV8 to obtain the relationship data of each picture; The training module is used to perform intimacy classification training through Transformer based on the relationship data of each image to obtain a trained relationship classification Transformer model; The classification module is used to classify the relationships of the frequent items in the original images where the frequent items are mined based on the trained relationship classification Transformer model; The evaluation module is used to test the model using test data, mine privacy relationships, make coarse-grained category judgments, and assess the risk of privacy leakage.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the privacy leakage assessment method based on social privacy relationship mining according to any one of claims 1 to 8 are implemented.