A method for deduplication management of similar ciphertext data in the Internet of Things scenario

Abstract

The present invention discloses a method for deduplication management of ciphertext similar data in an Internet of Things scenario, which can realize deduplication of similar data ciphertext; according to data management requirements, data similarity retention label generation and similarity data ciphertext deduplication are designed; data similarity is compared by similarity retention labels to improve data comparison speed; similar data comparison and similar data ciphertext deduplication are performed with the help of edge nodes to improve data processing response timeliness and save data storage space; the present invention also comprehensively considers data storage security requirements and Internet of Things data availability requirements, and performs secure data permission expansion on stored data, allowing users to query their own data and similar data ciphertexts, so as to facilitate users to perform data analysis. This method can save storage space and transmission bandwidth to a great extent, and improve Internet of Things data management efficiency and expand data availability.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, discloses the application of data deduplication in a new field, and specifically relates to a deduplication management method for ciphertext similar data in an Internet of Things scenario. Background Art

[0002] With the development of the Internet of Things and the intelligent society, the data of the Internet of Things has exploded, and it is difficult for local storage space to store and manage it effectively. Some Internet of Things devices use technologies such as iterative overwriting to reduce storage pressure. For example, the storage period of surveillance video is only one month, which greatly reduces the availability and traceability of Internet of Things data, making it difficult to meet the requirements of some enterprises or users for data analysis or anomaly viewing.

[0003] The emergence of cloud storage provides a practical remedy for the storage management of massive data. However, due to the one-to-many service architecture of cloud storage and the information barriers between different users, a large amount of redundant data occupies cloud storage space. Data deduplication technology can keep only one piece of data and delete other duplicate copies, effectively solving the data redundancy problem. However, due to the special characteristics of IoT data, existing data deduplication solutions are not applicable to IoT data.

[0004] Unlike ordinary Internet data, IoT data is standardized, with the same data format and size. Moreover, data from the same device in the same region is mostly similar and carries the same information. Due to slight differences in the specific data and timestamps recorded, a large amount of similar data cannot be classified as the same data, making the current data deduplication method unable to identify similar data and unsuitable for the current IoT system.

[0005] In addition, the real-time characteristics of IoT data also have high requirements for the response speed of data processing. Edge computing can use edge nodes to pre-process data and pass the processed valuable data to CSP, thereby reducing network transmission pressure and saving storage costs. Compared with cloud computing, edge computing can provide lower latency and is more adaptable to real-time data processing. Since users lose physical control of their data during edge computing and cloud storage, data security is the basic demand of users. However, the goal of encryption technology is to provide semantic security of plaintext data so that ciphertext data cannot be distinguished, which increases the difficulty of data deduplication. Although the same data can be encrypted into the same ciphertext through key transfer and other methods, it still cannot meet the needs of ciphertext deduplication of similar data. Therefore, how to break the information barriers between users without transferring keys and achieve ciphertext deduplication of similar data is a problem worth exploring.

[0006] Moreover, most current data management systems only consider data security and storage costs, ignoring data accessibility and scalability. In IoT systems, data storage is mainly used for data analysis; if analysis cannot be performed, the value of data storage will be reduced. Therefore, we also need to consider how to improve the availability and scalability of IoT data. Summary of the invention

[0007] To solve the above problems, the present invention discloses a method for deduplication management of ciphertext similar data in an IoT scenario, which effectively solves the contradiction between CSP data management and IoT data storage requirements. The present invention constructs a parallel edge-based similar data deduplication system framework, uses edge nodes to perform deduplication operations on data, improves the efficiency of data upload, and reduces transmission pressure. In addition, the present invention can directly compare the similarity of ciphertexts and perform similarity deduplication operations, which greatly reduces storage costs while ensuring data security. Our system also takes into account the availability of data and improves data availability through data permission expansion.

[0008] The technical solution adopted by the present invention is: a method for deduplication management of ciphertext similar data in an Internet of Things scenario, comprising the following steps:

[0009] S1, the IoT data owner calculates the data similarity retention tag and initiates a data upload request to the nearest edge node; the edge node performs data similarity comparison on the data based on the received upload request to determine whether similar data already exists. The data owner performs similarity encryption on the data based on the similarity comparison results and the data security requirements;

[0010] S2, the data owner uploads the encrypted data and its auxiliary parameters. Based on the data similarity comparison results, the edge node performs similarity ciphertext deduplication operations on the regular data and performs temporary storage management. The edge node regularly sends the deduplicated valid data to the CSP, and the CSP performs similarity association storage management on the data.

[0011] S3, when the data owner initiates a data query request, CSP verifies the user ownership. If the verification is successful, the user rights are expanded to allow the data owner to access his or her own data and similarity-related data.

[0012] As a further improvement of the present invention, the step S1 further comprises:

[0013] S11, a data upload request initiated by the data owner. The data owner uses a similarity-preserving label generation method to calculate the similarity-preserving label of the data and initiates a data upload request to the nearest edge node.

[0014] S12, based on the received upload request, the edge node determines whether similar data already exists; the data owner performs similarity encryption processing on the data based on the similarity comparison result and the data security requirements;

[0015] Step S12 includes the following sub-steps:

[0016] S121, after the IoT data upload request arrives, the edge node retains the received similarity label t w' , and the existing data label t w ∈{T} to calculate the Hamming distance. When the Hamming distance between two labels is less than the given threshold, the two data are considered to be similar.

[0017] S122, if similar data exists, the CSP returns a key auxiliary parameter P to the data owner. w , the user will be a random number r 2 ∈{0,1} λ , auxiliary parameter P w and the fuzzy summary of the file w' As input to the key regeneration method, the encryption key k is regenerated w , denoted as FE-REP(r 2 ,t w' ,P w )→k w ;

[0018] S123, if there is no similar data, the data owner directly uses the fuzzy extraction-based key generation method FE-KG (1 λ ,r 2 ,t w )→k w , generate the encryption key of the data; this method uses security parameter λ, random number r 2 ∈{0,1} λ and data label t w As input, output a random string encryption key R w and auxiliary information P w , where k w =R w ;

[0019] S124, the data owner uses the key to calculate the encrypted data and obtain the corresponding ciphertext. The specific encryption process can be expressed as: Where G(k w )→{0,1} |w| is a pseudo-random generator with an output length of |w| The pseudo-random encryption key G().

[0020] As a further improvement of the present invention, the step S2 further comprises:

[0021] S21, the data owner uploads the encrypted data and its auxiliary parameters. Based on the data similarity comparison results, the edge node performs similarity ciphertext deduplication operations on the regular data and performs temporary storage management;

[0022] Step S21 includes the following sub-steps:

[0023] S211, the data owner only needs to store the file tag t w As a credential for user ownership verification, and upload the data ciphertext c w Its auxiliary parameter P w To the nearest edge node;

[0024] S212: If there is no similar data on the edge node, the edge node stores the similarity retention label t w , key auxiliary parameter P w And, the ciphertext c w , and press label t w Size determines the storage order;

[0025] S213, if similar data already exists on the edge node, the edge node will no longer store it repeatedly, and directly perform a deduplication operation on the data;

[0026] S22, the edge node regularly sends the deduplicated valid data to the CSP, and the CSP performs similarity association storage management on the data;

[0027] Specifically, step S22 includes the following sub-steps:

[0028] S221, the edge node regularly sends the deduplicated valid data to the CSP, and the CSP compares the newly uploaded data with the existing data for data similarity;

[0029] S221, since the edge nodes and CSP are stored in the order of label size, when the CSP searches for similar data, the pointer only needs to move in the order of label size without repetition;

[0030] S221, if there is similar data, similarity association storage is performed, that is, firstly stored in the order of tags, and a similar data link is added at the end of each data;

[0031] S221, if there is no similar data, insert and store it directly in the order of tags.

[0032] As a further improvement of the present invention, the step S3 further comprises:

[0033] S31, when the data analyst initiates a data access request, the CSP first proves the ownership of the data analyst and feeds back the result to the data analyst;

[0034] S32, if the verification is successful, according to the CSP access control policy, the data analyst can not only obtain the ciphertext of his own data w Its auxiliary parameter P w , you can also access similar data ciphertext c w' , the user decrypts the ciphertext to obtain the plaintext data, so as to facilitate the user to perform data analysis;

[0035] Specifically, step S32 includes the following sub-steps:

[0036] S321, the user first sets t w and P w As input, the key regeneration function FE-REP(r 2 ,t w' ,P w )→k w Perform key recovery to obtain the decryption key k w ;

[0037] S322, the user uses the key k w For the ciphertext data c w and c w' Decrypt and obtain plaintext data;

[0038] S33, if the verification fails, the user cannot obtain the ciphertext data.

[0039] Beneficial effects of the present invention:

[0040] 1. When the data owner initiates a data upload request and uploads a similarity-preserving label, the data similarity can be directly compared through the label, and similarity ciphertext deduplication operations can be performed, which greatly reduces storage costs while ensuring data security.

[0041] 2. The present invention also constructs a parallel edge-based similar data deduplication system framework. In the present invention, data similarity comparison and deduplication operations are directly performed by edge nodes, and edge nodes only upload deduplicated valid data to CSP for storage management, which improves data transmission efficiency and reduces transmission pressure.

[0042] 3. This system also allows CSP to extend data permissions for authorized data, allowing data owners to access their own data and similar data, thus improving the availability of IoT data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1is a flow chart of the method steps of the present invention;

[0044] Figure 2 It is a system framework diagram of the present invention;

[0045] Figure 3 It is a response interaction diagram of the present invention;

[0046] Figure 4 It is the overall system flow chart of the present invention. DETAILED DESCRIPTION

[0047] symbol definition w,w' IoT Data <![CDATA[t w ,t w' ]]> Similarity Preserving Labels <![CDATA[k w ,k w' ]]> Encryption Key <![CDATA[P w ]]> Secondary key parameters <![CDATA[c w ,c w' ]]> Data ciphertext d Preset data similarity threshold <![CDATA[r 1,2 ]]> Secure Random Numbers <![CDATA[PoW w ]]> Data ownership verification <![CDATA[link w ]]> Data Links This paper proposes a method for deduplication management of ciphertext similar data in the IoT scenario. This method relies on the similarity encryption method to manage IoT data securely, so that similar data encryption is still similar, and similar data ciphertext deduplication can be achieved. The edge node is introduced to process IoT data, and the parallel operation meets the real-time requirements of IoT data. By generating a label generation method that maintains similarity, the edge node can compare the similarity of data ciphertexts, and combined with the characteristics of IoT data, only similar data is deduplicated for regular data, further ensuring the security of abnormal data. In addition, the edge node only uploads the deduplicated valid data to the CSP, reducing communication overhead and saving storage space; CSP stores data by similarity association and allows data owners to view their own data and similar data, improving the availability and scalability of data.

[0048] The technical solution disclosed in the present invention is further described in detail below through specific steps.

[0049] like Figure 1 , 2 As shown in , this paper provides a method for deduplication management of ciphertext similar data in the IoT scenario, including the following steps:

[0050] (1) Data upload request initiated by the data owner. The data owner uses the following method to calculate the similarity retention label of the data and initiates a data upload request to the nearest edge node.

[0051] Specifically, the label generation method designed by the present invention mainly has six processes: word segmentation, hashing, binary conversion, weighting, merging, and dimensionality reduction:

[0052] Word segmentation: Segment a given piece of text to generate n feature words and assign a weight to each word.

[0053] Hash: Determine whether the word segment is numeric data. If not, map each word vector through the hash function to generate an n-bit binary string.

[0054] Binarization conversion: Determine whether the segmented word is numerical data. If so, perform binarization conversion on the data as the hash of the segmented word.

[0055] Weighting: Calculate the weight vector W = hash × weight, that is, multiply the positions with 1 in the hash binary string by the segmentation weight of the feature word, and multiply the positions with 0 in the binary string by the segmentation weight of the feature word and then take the negative, and then obtain the weight vector.

[0056] Merging: For a text, after calculating the weight vectors of all feature words in the text, accumulate these weight vectors to obtain a new weight vector.

[0057] Dimensionality reduction: Process the weight vector obtained after merging. Set the positions greater than 0 to 1 and the positions less than 0 to 0, and the similarity retention label of the text can be obtained.

[0058] (2) According to the received upload request, the edge node determines whether there is already similarity data. The data owner performs similarity encryption processing on the data according to the similarity comparison result and in combination with the security requirements of the data.

[0059] Step (2) includes the following sub-steps:

[0060] (2.1) After the IoT data upload request arrives, the edge node calculates the Hamming distance between the received similarity retention label t w' , and the existing data label t w ∈ {T}. When dis Ham (t w' , t w ) < t, then these two data are considered similar data.

[0061] (2.2) If there is similar data, the CSP returns a key auxiliary parameter P w to the data owner. The user takes P w and the fuzzy digest t w' of the file as the input of the key regeneration method, and outputs the private key k w , denoted as FE-REP(r 2 , t w' , P w ) → k w .

[0062] (2.3) If there is no similar data, the data owner directly runs a key generation method FE-KG(1 λ , r 2 , t w ) → k w , to generate the encryption key of the data; with the security parameter λ, random number r 2∈{0,1} λ and data label t w As input, output a random string encryption key R w and auxiliary information P w , where k w =R w .

[0063] (2.4) The data owner encrypts the data by calculation and obtains the corresponding ciphertext. The similarity encryption method uses file w and a private key k w As input, returns the ciphertext Where G(k w )→{0,1} |w| is a pseudo-random generator with an output length of |w| The pseudo-random encryption key G().

[0064] (3) The data owner uploads the encrypted data and its auxiliary parameters. Based on the data similarity comparison results, the edge node performs similarity ciphertext deduplication operations on the regular data and performs temporary storage management.

[0065] Step (3) includes the following sub-steps:

[0066] (3.1) The data owner only needs to store the file label t w As a credential for user ownership verification, and upload the data ciphertext c w Its auxiliary parameter P w To the nearest edge node.

[0067] (3.2) If there is no similar data on the edge node, the edge node stores t w , P w and c w , and press t w The size determines the storage order.

[0068] (3.3) If similar data already exists on the edge node, the edge node will no longer store it repeatedly and will directly deduplicate the data.

[0069] (4) The edge node regularly sends the deduplicated valid data to the CSP, and the CSP performs similarity association storage management on the data.

[0070] Step (4) includes the following sub-steps:

[0071] (4.1) When CSP receives the valid data after deduplication uploaded by the edge node regularly, CSP compares the newly uploaded data with the existing data for data similarity.

[0072] (4.2) Since edge nodes and CSP are stored in the order of label size, when CSP searches for similar data, the pointer only needs to move in the order of label size without repetition.

[0073] (4.3) If similar data exists, similarity association storage is performed, that is, storage is performed in label order first, and a similar data link is added at the end of each data, such as Figure 3 shown.

[0074] (4.4) If there is no similar data, store them directly in label order.

[0075] (5) When the data owner initiates a data query request, CSP performs PoW w User ownership verification: if the verification is passed, the data owner is allowed to access his or her own data and its similarity-related data.

[0076] Step (5) includes the following sub-steps:

[0077] (5.1) While CSP stores and manages data, it also expands the user's access rights. When a data analyst initiates a data access request, CSP first proves ownership of the data analyst and feeds back the result to the data analyst.

[0078] (5.2) If the verification is successful, according to the CSP's access control policy, the user's own data ciphertext c is returned w Its auxiliary parameter P w , and its similar data ciphertext c w' , to facilitate users to perform data analysis.

[0079] (5.3) The user first sets t w and P w As input, the key is recovered through the key regeneration function, denoted as FE-REP(r 2 ,t w' ,P w )→k w .

[0080] (5.4) The user uses the key k w For the ciphertext data c w and c w' Decrypt and obtain the plaintext data.

[0081] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.

Claims

1. A method for deduplication management of ciphertext similar data in an Internet of Things scenario, characterized by: The steps include: S1, the IoT data owner calculates the data similarity retention label and initiates a data upload request to the nearest edge node; the edge node compares the data similarity based on the received upload request to determine whether similar data already exists; The data owner performs similarity encryption processing on the data based on the similarity comparison results and the data security requirements; The step S1 further comprises: S11, a data upload request initiated by the data owner. The data owner uses a similarity-preserving label generation method to calculate the similarity-preserving label of the data and initiates a data upload request to the nearest edge node; S12, based on the received upload request, the edge node determines whether similar data already exists; the data owner performs similarity encryption processing on the data based on the similarity comparison result and the data security requirements; Step S12 includes the following sub-steps: S121, after the IoT data upload request arrives, the edge node retains the received similarity label t w' , and the existing data label t w ∈{T} to calculate the Hamming distance. When the Hamming distance between two labels is less than the given threshold, the two data are considered to be similar. S122, if similar data exists, the CSP returns a key auxiliary parameter P to the data owner. w , the user sets the random number r2∈{0,1} λ , auxiliary parameter P w and the fuzzy summary of the file w' As input to the key regeneration method, the encryption key k is regenerated w , denoted as FE-REP(r2,t w' ,P w )→k w ; S123, if there is no similar data, the data owner directly uses the fuzzy extraction-based key generation method FE-KG (1 λ ,r2,t w )→k w , generate the encryption key of the data; the key generation method uses the security parameter λ, the random number r2∈{0,1} λ and data label t w As input, output a random string encryption key R w and auxiliary information P w , where k w =R w ; S124, the data owner uses the key to calculate the encrypted data and obtain the corresponding ciphertext. The specific encryption process can be expressed as c w =(w⊕G(k w ,r1))⊕G(t w ,r2); where G(k w )→{0,1} |w| is a pseudo-random generator that outputs a pseudo-random encryption key G() of length |w|; S2, the data owner uploads the encrypted data and its auxiliary parameters. According to the data similarity comparison result, the edge node performs similarity ciphertext deduplication operation on the similar data and performs temporary storage management; the edge node regularly sends the deduplicated valid data to the CSP, and the CSP performs similarity association storage management on the data; the step S2 further includes: S21, the data owner uploads the encrypted data and its auxiliary parameters. Based on the data similarity comparison results, the edge node performs similarity ciphertext deduplication operations on the regular data and performs temporary storage management; Step S21 includes the following sub-steps: S211, the data owner only needs to store the file tag t w As a credential for user ownership verification, and upload the data ciphertext c w Its auxiliary parameter P w To the nearest edge node; S212: If there is no similar data on the edge node, the edge node stores the similarity retention label t w , key auxiliary parameter P w And, the ciphertext c w , and press label t w Size determines the storage order; S213, if similar data already exists on the edge node, the edge node will no longer store it repeatedly, and directly deduplicate the data; S22, the edge node regularly sends the deduplicated valid data to the CSP, and the CSP performs similarity association storage management on the data; Specifically, step S22 includes the following sub-steps: S221, the edge node regularly sends the deduplicated valid data to the CSP, and the CSP compares the newly uploaded data with the existing data for data similarity; S221, since the edge nodes and CSP are stored in the order of label size, when the CSP searches for similar data, the pointer only needs to move in the order of label size without repetition; S221, if there is similar data, similarity association storage is performed, that is, firstly stored in the order of tags, and a similar data link is added at the end of each data; S221, if there is no similar data, directly insert and store it in the order of tags; S3, when the data owner initiates a data query request, the CSP performs user ownership verification. If the verification is successful, the user rights are extended to allow the data owner to access his own data and its similarity-related data. The step S3 further includes: S31, when the data analyst initiates a data access request, the CSP first proves the ownership of the data analyst and feeds back the result to the data analyst; S32, if the verification is successful, according to the CSP's access control policy, the data analyst can not only obtain the ciphertext of his own data w Its auxiliary parameter P w , you can also access similar data ciphertext c w' , the user decrypts the ciphertext to obtain the plaintext data, so as to facilitate the user to perform data analysis; Specifically, step S32 includes the following sub-steps: S321, the user first sets t w and P w As input, the key regeneration function FE-REP(r2,t w' ,P w )→k w Perform key recovery to obtain the decryption key k w ; S322, the user uses the key k w For the ciphertext data c w and c w' Decrypt and obtain plaintext data; S33, if the verification fails, the user cannot obtain the ciphertext data.

Images