A privacy protection method for multimodal data

Abstract

The present invention relates to a privacy protection method for multimodal data applied in the field of data processing. Under the premise of not changing the original data, the multimodal data is transformed to generate transformed data containing key feature information, and the transformed data is stored in another form different from the original data. In the process of the user accessing the original data, the user is verified through the transformed data in two processes, namely, non-privacy and privacy. This not only prevents the user from directly contacting the original data, thus protecting the privacy security of the data, but also indirectly reflects the user's understanding of the original data through the verification result, effectively determines the relationship between the user and the original data, and confirms whether access is allowed. When the verification result is semi-correct, the confidentiality temporary storage and transfer process of the original data can be performed through a designated transferor, so that the available invisible use mode can be performed. On the basis of protecting the privacy security of the data, the user's urgent use of the data is alleviated.

Description

Technical Field

[0001] The present invention relates to a privacy protection method, and in particular to a privacy protection method for multimodal data applied in the field of data processing. Background Art

[0002] With the rapid development of information technology, multimodal data has become an important means to acquire, analyze and utilize information in various fields. However, the fusion processing of multimodal data involves multiple types of data, including text, images, audio, video, etc. These data often have the risk of privacy leakage during processing and transmission.

[0003] The specification of Chinese patent CN202010215476.2 discloses a privacy protection method and system for multimodal deep learning, and its main steps are as follows: obtain audio data to be processed, image data to be processed and text data to be processed; perform corresponding privacy protection processing on the audio data to be processed, image data to be processed and text data to be processed to obtain data to be identified; input the data to be identified into the emotion classification model to obtain classification results of each modality; use the weight self-selection algorithm of the dynamic routing protocol to automatically assign weights of different modal categories and different emotion category combinations to obtain weight coefficients; obtain emotion recognition results based on the weight coefficients and the classification results of each modality; the present invention can take corresponding privacy protection processing according to the data type, so as to prevent user information leakage and thus improve user experience.

[0004] In the prior art, when users access multimodal data, they generally use key authentication. However, with the development of computer technology, people are becoming more and more proficient in the use of computer technology. Therefore, the single key encryption method still poses a great risk to the protection of data. Summary of the invention

[0005] In view of the above-mentioned prior art, the technical problem to be solved by the present invention is that with the development of computer technology, the existing method of encrypting data through keys is relatively simple, and the verification method for users to access data is also relatively simple, which not only does not fully protect the privacy of data, but also makes it inconvenient for users to use the data.

[0006] To solve the above problems, the present invention provides a privacy protection method for multimodal data, comprising the following steps:

[0007] Step S1, storing the multimodal data (including image data, audio data, video data and text data) input into the server in a real database;

[0008] Step S2: Generate transformation data through a data transformation system:

[0009] Step S2-1, extracting key features from the image data, and dividing the extracted key features into privacy features and non-privacy features;

[0010] Generate a correct privacy verification text Aa and n wrong privacy verification texts (including text A1, text A2, ... and text An) according to the extracted privacy features; generate a correct non-privacy verification text Bb and n wrong non-privacy verification texts (including text B1, text B2, ... and text Bn) according to the non-privacy features, and the above multiple verification texts are all transformed data of the image data;

[0011] Step S2-2: Similarly, the same steps as step S2-1 are used to complete the generation process of the transformed data of both the audio data and the video data, and the transformed data of both are also in text form;

[0012] Step S2-3, extract key features from the text data, and also divide them into privacy features and non-privacy features. Generate a correct privacy verification image Jj and n wrong privacy verification images (including image J1, image J2, ... and image Jn) according to the extracted privacy features; generate a correct non-privacy verification image Gg and n wrong non-privacy verification images (including image G1, image G2, ... and image Gn) according to the non-privacy features. The above multiple verification images are all transformed data of the text data;

[0013] Step S3: Protect the privacy of multimodal data through the transformation verification system:

[0014] Step S3-1, when the user needs to access the image data, the first verification process is first performed: the non-privacy verification text Bb corresponding to the image data and n incorrect non-privacy verification texts (including text B1, text B2, ... and text Bn) are simultaneously displayed to the user, and the user is reminded to make a correct selection;

[0015] When the selection result of the first verification is wrong (i.e. Bb is not selected), the access result is directly determined to be failed. When the selection result is correct (i.e. Bb is selected), the second verification process is performed: the privacy verification text Aa corresponding to the image data and n wrong privacy verification texts (including text A1, text A2, ... and text An) are displayed to the user at the same time, and the user is reminded to make a correct selection. When the user makes a correct selection again (i.e. Aa is selected), the user is allowed to access the image data;

[0016] Step S3-2: Similarly, when the user needs to access one of the audio data, video data or text data, the transformed data corresponding to the data to be accessed is used to perform two verification processes on the user in the same manner as step S3-1.

[0017] As a further supplement to the present application, the data transformation system includes a data storage module, a feature extraction module, a privacy determination module, a text generation module and an image generation module, and the data storage module and the feature extraction module are both connected to a real database.

[0018] As a further supplement to the present application, the text generation module and the image generation module are both connected to a transformation database, and the transformation database is used to store the transformation data generated by both the text generation module and the image generation module.

[0019] As a further supplement to the present application, the transformation verification system includes a single-choice verification module, an access record module and an early warning module. The single-choice verification module is used to perform two verification processes on the user, and the access record module is used to record the number of customer access failures. When the number of failed accesses to the same data by the user within time H exceeds the allowed value, the early warning module issues an abnormal access alarm.

[0020] As another improvement of the present application, in step S3, when the selection result of the user's second verification is wrong, a designated transferor can be used to perform a confidential transfer operation on the accessed data.

[0021] As another improved supplement of the present application, the designated transferor is connected to the server via a USB interface and realizes data transmission with the server by means of key authentication.

[0022] As another improved supplement of the present application, the transformation verification system also includes a docking module and a transmission module, the docking module is used to verify the key, and the transmission module is used to transmit the data to be accessed to a designated transferor.

[0023] As another improved supplement of the present application, the designated transferor includes a storage unit, an internal confidentiality unit and a decryption output unit. The internal confidentiality unit is used to encrypt the original data transmitted from the server and save it in the storage unit in a ciphertext state. The decryption output unit is used to decrypt the ciphertext, restore it to the original plaintext, and transmit the data in the original plaintext state to another server.

[0024] To sum up, the present application transforms multimodal data without changing the original data, generates transformed data containing key feature information, and stores the transformed data in a form different from the original data, so that when a user needs to access a certain original data, the user is subjected to both non-privacy and privacy verification processes through the corresponding transformed data, which not only prevents the user from directly contacting the original data and protects the privacy and security of the data, but also indirectly reflects the user's understanding of the original data through the verification result, thereby effectively determining the relationship between the user and the original data and confirming whether the user is allowed to access and view the data; when the verification result is semi-correct, the confidentiality temporary storage and transfer process of the original data is performed by specifying a transferor, so as to realize the use of the accessed data in an available but invisible manner, which not only alleviates the user's urgent use of the data, but also prevents the user from directly accessing and viewing the data, effectively protecting the privacy and security of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a method flow chart of the first and second implementation modes of this application;

[0026] Figure 2 This is a system diagram of the first and second implementation modes of the present application;

[0027] Figure 3 The generation process of the transformation data in the first and second embodiments of this application Figure 1 ;

[0028] Figure 4 The generation process of the transformation data in the first and second embodiments of this application Figure 2 ;

[0029] Figure 5 The generation process of the transformation data in the first and second embodiments of this application Figure 3 ;

[0030] Figure 6 This is a system diagram for generating transformation data in the first and second implementation modes of the present application;

[0031] Figure 7 This is a flow chart of the verification process for privacy protection in the first and second implementation modes of this application;

[0032] Figure 8 This is a schematic diagram of the connection between the designated transferor and the server in the second implementation mode of the present application;

[0033] Fig. 9 This is a flow chart of data transmission between a designated transferor and a server in the second embodiment of the present application;

[0034] Fig.10This is a system diagram of a designated transfer device in the second embodiment of the present application. DETAILED DESCRIPTION

[0035] The following describes two implementation modes of the present application in detail with reference to the accompanying drawings.

[0036] The first implementation method:

[0037] The present invention provides a privacy protection method for multimodal data. Figure 1 , including the following steps:

[0038] Step S1, storing the multimodal data (including image data, audio data, video data and text data) input into the server in a real database;

[0039] Step S2: Generate transformation data through a data transformation system:

[0040] Step S2-1: Figure 3 As shown, key features in the image data are extracted, and the extracted key features are divided into privacy features and non-privacy features;

[0041] Generate a correct privacy verification text Aa and n incorrect privacy verification texts (including text A1, text A2, ... and text An) according to the extracted privacy features; generate a correct non-privacy verification text Bb and n incorrect non-privacy verification texts (including text B1, text B2, ... and text Bn, n is a positive integer greater than 1) according to the non-privacy features, and the above multiple verification texts are all transformed data of the image data;

[0042] Step S2-2: Similarly, the same steps as step S2-1 are used to complete the generation process of the transformed data of both the audio data and the video data, and the transformed data of both are also in text form;

[0043] That is: Figure 4 As shown, according to the key features in the audio data, a corresponding correct privacy verification text Cc, n incorrect privacy verification texts (including text C1, text C2... and text Cn), a correct non-privacy verification text Dd and n incorrect non-privacy verification texts (including text D1, text D2... and text Dn) are generated;

[0044] According to the key features in the video data, a corresponding correct privacy verification text Ee, n incorrect privacy verification texts (including text E1, text E2, ... and text En), a correct non-privacy verification text Ff and n incorrect non-privacy verification texts (including text F1, text F2, ... and text Fn) are also generated;

[0045] Step S2-3, as Figure 5 As shown, the key features in the text data are extracted and divided into privacy features and non-privacy features. A correct privacy verification image Jj and n wrong privacy verification images (including image J1, image J2, ... and image Jn) are generated according to the extracted privacy features; a correct non-privacy verification image Gg and n wrong non-privacy verification images (including image G1, image G2, ... and image Gn) are generated according to the non-privacy features. The above multiple verification images are all transformed data of the text data;

[0046] Supplementary explanation: In the process of transforming multimodal data, the present application preferentially adopts text form as the form of existence of the transformed data of image data, audio data and video data, and adopts image form as the form of existence of the transformed data of text data, but is not limited to this. In the specific implementation process, other forms can also be used as the form of existence of the transformed data of the above data, as long as the transformed data form is different from the original data form, such as: image form can also be used as the form of existence of the transformed data of video data, and audio form can be used as the form of existence of the transformed data of text data;

[0047] Step S3: Protect the privacy of multimodal data through the transformation verification system:

[0048] Step S3-1, taking image data as an example: combining Figure 7 As shown, when the user needs to access the image data, the first verification process is performed: the non-privacy verification text Bb corresponding to the image data and n incorrect non-privacy verification texts (including text B1, text B2, ... and text Bn) are simultaneously displayed to the user, and the user is reminded to make a correct selection;

[0049] When the selection result of the first verification is wrong (i.e. Bb is not selected), the access result is directly determined to be failed. When the selection result is correct (i.e. Bb is selected), the second verification process is performed: the privacy verification text Aa corresponding to the image data and n wrong privacy verification texts (including text A1, text A2, ... and text An) are displayed to the user at the same time, and the user is reminded to make a correct selection. When the user makes a correct selection again (i.e. Aa is selected), the user is allowed to access the image data;

[0050] The above judgment principle is as follows: 1. For the first verification about non-privacy features, when the user makes an incorrect choice, it indicates that the user has insufficient understanding of the data to be accessed, and the data may not belong to the user himself or to a person related to him. Therefore, in order to improve the protection of data and the protection of privacy information in the data, there is no need to display the correct privacy verification text Aa and the incorrect privacy verification text to the user. The access result can be directly judged as failed, and the user can choose to give up the access or re-access; when the user makes the correct choice for both verifications, it can be indicated to a certain extent that the user has a full understanding of the data to be accessed, and the data belongs to the user himself or to a person related to him, so the user is allowed to access the data;

[0051] Step S3-2: Similarly, when the user needs to access one of the audio data, video data or text data, the transformed data corresponding to the data to be accessed is used to perform two verification processes on the user in the same manner as step S3-1.

[0052] From the above process, it can be seen that after saving the multimodal original data, the present application transforms the multimodal data to generate transformed data containing key feature information, and stores the transformed data in a form different from the original data. This enables the user to undergo both non-privacy and privacy verification processes through the corresponding transformed data when the user needs to access certain original data. This not only prevents the user from directly contacting the original data and protects data privacy, but also indirectly reflects the user's understanding of the original data through the verification results, thereby effectively determining the relationship between the user and the original data and confirming whether the user is allowed to access and view the data.

[0053] See also Figure 2 The data transformation system includes a data storage module, a feature extraction module, a privacy determination module, a text generation module and an image generation module. The data storage module and the feature extraction module are both connected to a real database. Figure 6 As shown, the text generation module and the image generation module are both connected to a transformation database, so that after the text generation module and the image generation module generate transformation data, they input the transformed data into the transformation database for storage, which is convenient for use when verifying the user. The transformation database is used to store the transformation data generated by the text generation module and the image generation module.

[0054] See also Figure 2The transformation verification system includes a single-choice verification module, an access record module and an early warning module. The single-choice verification module is used to perform two verification processes on the user. The access record module is used to record the number of customer access failures. When the number of failed accesses to the same data by the user within a time H exceeds the allowed value, the early warning module issues an abnormal access alarm, making it difficult for the user to access the same data again after multiple failures, further improving data privacy security.

[0055] Second implementation method:

[0056] This embodiment adds a designated transfer device and its use process on the basis of embodiment 1, which is as follows: In step S3, combined with Figure 7 As shown, when the user's selection result of the second verification is wrong, the designated transferor can be used to perform a confidential transfer operation on the accessed data.

[0057] See also Figure 8 The designated transferor is connected to the server through a USB interface and realizes data transmission with the server through key verification. The transformation verification system also includes a docking module and a transmission module. The docking module is used to verify the key, and the transmission module is used to transmit the data to be accessed to the designated transferor. The uniqueness of the designated transferor is effectively guaranteed by key verification, so that the data transfer process cannot be easily replaced by other media devices.

[0058] See also Fig. 9 and Fig.10 The designated transfer device includes a storage unit, an internal confidentiality unit and a decryption output unit. The internal confidentiality unit is used to encrypt the original data transmitted from the server and store it in the storage unit in a ciphertext state, so that it is difficult for users to directly view the data in the designated transfer device. It only performs data storage and transfer. The decryption output unit is used to decrypt the ciphertext, restore it to the original plaintext, and transmit the data in the original plaintext state to another server. After the data is transferred to another server in its original state, the data privacy protection steps (steps S1 and S2) in implementation mode 1 will also be performed, so that when other users need to access the data on the server, they also need to perform two verification processes (step S3).

[0059] In this implementation, when the user verification result is semi-correct (the first verification result is correct, and the second verification result is incorrect), the data to be accessed can be used in an invisible manner by specifying a transferor to perform confidential temporary storage and transfer of the original data. This not only alleviates the user's urgent use of the data, but also prevents the user from directly accessing and viewing the data, thereby effectively protecting the privacy and security of the data.

[0060] In view of current practical needs, the above-mentioned implementation mode adopted in this application is not limited to the scope of protection. Various changes made within the knowledge scope of technical personnel in this field without departing from the concept of this application still fall within the scope of protection of the present invention.

Claims

1. A privacy protection method for multimodal data, characterized by: The following steps are involved: Step S1, storing multimodal data input into a server in a real database, the multimodal data including image data, audio data, video data and text data; Step S2: Generate transformation data through a data transformation system: Step S2-1, extracting key features from the image data, and dividing the extracted key features into privacy features and non-privacy features; A correct privacy verification text Aa and n incorrect privacy verification texts are generated according to the extracted privacy features, including text A1, text A2.... and text An; a correct non-privacy verification text Bb and n incorrect non-privacy verification texts are generated according to the non-privacy features, including text B1, text B2.... and text Bn, and the above correct privacy verification text, incorrect privacy verification text, correct non-privacy verification text and incorrect non-privacy verification text are all transformed data of the image data; Step S2-2: Similarly, the same steps as step S2-1 are used to complete the generation process of the transformed data of both the audio data and the video data, and the transformed data of both are also in text form; Step S2-3, extract key features from the text data, and also divide them into privacy features and non-privacy features. According to the extracted privacy features, a correct privacy verification image Jj and n incorrect privacy verification images are generated, including image J1, image J2.... and image Jn; according to the non-privacy features, a correct non-privacy verification image Gg and n incorrect non-privacy verification images are generated, including image G1, image G2.... and image Gn. The above correct privacy verification image, incorrect privacy verification image, correct non-privacy verification image and incorrect non-privacy verification image are all transformed data of the text data; Step S3: Protect the privacy of multimodal data through the transformation verification system: Step S3-1, when the user needs to access the image data, the first verification process is first performed: the non-privacy verification text Bb corresponding to the image data and n incorrect non-privacy verification texts, including text B1, text B2, ... and text Bn, are simultaneously displayed to the user, and the user is reminded to make the correct selection; When the selection result of the first verification is wrong, that is, Bb is not selected, it is directly determined that the access result has failed. When the above selection result is correct, that is, Bb is selected, the second verification process is performed: the privacy verification text Aa corresponding to the image data and n wrong privacy verification texts, including text A1, text A2... and text An, are simultaneously displayed to the user, and the user is reminded to make a correct selection. When the user selects the correct one again, that is, Aa, the user is allowed to access the image data; Step S3-2: Similarly, when the user needs to access one of the audio data, video data or text data, the transformed data corresponding to the data to be accessed is used to perform two verification processes on the user in the same manner as step S3-1.

2. The method for protecting privacy of multimodal data according to claim 1, characterized in that: The data transformation system includes a data storage module, a feature extraction module, a privacy determination module, a text generation module and an image generation module. The data storage module and the feature extraction module are both connected to a real database.

3. The method for protecting privacy of multimodal data according to claim 2, characterized in that: The text generation module and the image generation module are both connected to a transformation database, and the transformation database is used to store the transformation data generated by both the text generation module and the image generation module.

4. The method for protecting privacy of multimodal data according to claim 1, characterized in that: The transformation verification system includes a single-choice verification module, an access record module and an early warning module. The single-choice verification module is used to perform two verification processes on the user. The access record module is used to record the number of customer access failures. When the number of failed accesses to the same data by the user within time H exceeds the allowed value, the early warning module issues an abnormal access alarm.

5. The method for protecting privacy of multimodal data according to claim 1, characterized in that: In step S3, when the selection result of the second verification by the user is wrong, a designated transferor is used to perform a confidential transfer operation on the accessed data.

6. A method for protecting privacy of multimodal data according to claim 5, characterized in that: The designated transferor is connected to the server via a USB interface and realizes data transmission with the server by means of key authentication.

7. A method for protecting privacy of multimodal data according to claim 6, characterized in that: The transformation verification system further includes a docking module and a transmission module, wherein the docking module is used to verify the key, and the transmission module is used to transmit the data to be accessed to a designated transferor.

8. A method for protecting privacy of multimodal data according to claim 7, characterized in that: The designated transferor includes a storage unit, an internal confidentiality unit and a decryption output unit. The internal confidentiality unit is used to encrypt the original data transmitted from the server and save it in the storage unit in a ciphertext state. The decryption output unit is used to decrypt the ciphertext, restore it to the original plaintext, and transmit the data in the original plaintext state to another server.

Images