A method for deduplication of medical big data

Abstract

This invention discloses a method for deduplication of medical big data, comprising: acquiring medical data to be processed; calculating the similarity of the medical data to obtain data files sorted by maximum similarity; performing deduplication operations on the data files; and then rewriting the data based on maximum similarity to achieve the deletion of duplicate medical data. This method for deduplication of medical big data provides by using similarity calculation to determine whether medical data is redundant and simultaneously removing noisy data, and by rewriting data blocks based on maximum similarity to improve recovery and deletion performance. Furthermore, by designing a blockchain strategy and a deduplication recovery algorithm, the method efficiently ensures the security and integrity of medical data. Therefore, this method is applicable to the medical industry and is highly efficient and secure.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a method for deleting duplicate data for medical big data. Background Art

[0002] With the development of economy and technology, digitization and informatization have been widely integrated into all walks of life. The development of digitization and informatization in the medical industry has led to the generation of medical big data.

[0003] In addition to the characteristics of large-scale, high-speed, diverse and high-value that general big data has, medical big data also has the characteristics of multimodality, redundancy, security, incompleteness and timeliness. Among them, redundancy and security have brought great challenges to the application of medical big data. Since the growth of medical data is much faster than the growth of storage devices, and data storage, sufficient equipment and energy consumption are expensive, data redundancy has brought great pressure to the medical system. Therefore, many studies are devoted to eliminating redundant data, improving resource utilization and saving storage space.

[0004] Existing research on redundant data deletion mainly includes file-granularity-based deduplication methods and block-granularity-based deduplication methods. File-granularity-based deduplication methods use files as granularity, use secure hash functions (such as SHA-1, SHA-256) to perform hash calculations on each file to obtain a unique hash value, and then determine whether the file is redundant by comparing whether the hash values ​​of the files are consistent; block-granularity-based deduplication methods use different partitioning methods to divide the file into blocks, then calculate hash values ​​for the blocks, and determine whether the blocks are redundant by comparing whether the hash values ​​of the blocks are equal, such as fixed size partitioning (FSP) and variable size partitioning (VSP). File-based deduplication methods achieve a lower deduplication detection rate (DER) due to their coarse granularity, while block-based deduplication methods achieve a higher DER due to their fine-grained block division. Therefore, they are widely used in various backup systems and primary storage systems. DER is a key indicator for measuring deduplication algorithms. The higher the DER, the lower the storage overhead (DER = (total data - deduplicated data) / total data).

[0005] However, deduplication technology disperses data blocks in various containers, destroying the locality of data and generating a lot of fragments, which affects the system recovery and deletion performance. Existing deduplication methods improve recovery performance at the expense of DER, and do not deeply explore the characteristics of medical data to simultaneously improve the system's DER, recovery and deletion performance, and cannot determine which medical data has redundancy; in addition, there are methods that rely on the time attributes of the backup version when improving recovery performance, but medical data has uncertain time attributes. More importantly, medical data has higher security and compliance requirements for personal information protection than ordinary data, and existing encrypted deduplication technology requires verification by a third-party organization and cannot effectively guarantee the security of medical data. Summary of the invention

[0006] The purpose of the present invention is to provide a method for deduplicating medical big data that is efficient, safe and applicable to the medical industry.

[0007] The method for deleting duplicate data for medical big data provided by the present invention comprises the following steps:

[0008] S1. Obtain medical data to be processed;

[0009] S2. Calculate the similarity of the medical data obtained in step S1, thereby obtaining a data file sorted by the maximum similarity;

[0010] S3. Perform a deduplication operation on the data file obtained in step S2, and then rewrite it based on the maximum similarity, thereby completing the deletion of duplicate medical data.

[0011] The method for deleting duplicate data for medical big data also includes the following steps:

[0012] S4. Based on blockchain technology, medical data after deduplication is securely protected.

[0013] The blockchain technology is used to securely protect the medical data after deduplication, and specifically includes the following steps:

[0014] A. Design the following metadata structure:

[0015] {<Fingerprint 1, Block 1 size>,...,<Fingerprint i, Block i size>,...,<Fingerprint n, Block n size>}

[0016] B. During data recovery, the block size in the metadata is used to find the block location in the container, and the block content is found based on the block fingerprint in the metadata;

[0017] C. When restoring a file, after locating the block position, calculate the block fingerprint for each block and compare the metadata fingerprint with the calculated block fingerprint to see if they are the same:

[0018] If they are the same, the block is correct, and the correct block is used to recover the file;

[0019] If they are different, it means that the current block has been tampered with;

[0020] D. When storing evidence data, store lightweight metadata files on the blockchain, and use the stored lightweight metadata files instead of the original files to calculate evidence, thereby improving audit efficiency;

[0021] E. Build smart contracts based on the blockchain platform and establish a group isolation mechanism to achieve data sharing and data isolation among several medical institutions.

[0022] The calculation of the block fingerprint described in step C specifically uses the SHA-1 function to calculate the block fingerprint.

[0023] Step S2 performs similarity calculation on the medical data obtained in step S1, thereby obtaining data files sorted according to the maximum similarity. Specifically, based on the similarity between files, each data stream is divided into segments of a fixed size; each segment is then divided into blocks within the segment, and the hash value of each block is calculated and stored; next, the similarity of the files is calculated based on the number of shared blocks between the files; finally, the maximum similarity of each file is calculated, thereby obtaining data files sorted according to the maximum similarity.

[0024] The similarity calculation of the medical data obtained in step S1, so as to obtain the data files sorted according to the maximum similarity, specifically includes the following steps:

[0025] a. Divide each data stream into fixed-size segments based on the similarity between files;

[0026] b. Divide each segment obtained in step a into several blocks;

[0027] c. Calculate the hash value of each block obtained in step b and store it;

[0028] d. Based on the number of shared blocks between files, the similarity of each file is calculated to obtain data files sorted by the maximum similarity.

[0029] The step a of dividing each data stream into segments of a fixed size specifically divides each data stream into segments of a fixed size in a parallel manner.

[0030] Step b divides each segment obtained in step a into several blocks, specifically by using the following steps to divide the segments and adjust the boundary positions of each block:

[0031] The data in the segment is divided into several blocks of different sizes using the content-based blocking method. Then, for the last block of each segment and the first block of the next segment, the positions of the two blocks are re-determined using the following steps: If the sliding window size is m, the last m-1 bytes of the last block of the current segment and the first m-1 bytes of the first block of the next segment are re-divided into blocks based on the content-based blocking method.

[0032] The step c of calculating the hash value of each block obtained in the step b specifically uses the SHA-1 function to calculate the hash value of each block obtained in the step b.

[0033] The step d described in step 1 is to calculate the similarity of each file based on the number of shared blocks between the files, thereby obtaining data files sorted according to the maximum similarity, which specifically includes the following steps:

[0034] d1. Input file set F{F1,F2,...,F n} and the total number of blocks corresponding to each file is a set B{B1,B2,...,B n};

[0035] d2. For the current input file F i , both will be the current input file F i The fingerprint of the first i-1 files F1~F i-1 The fingerprints of the ith file are compared in turn, and then the similarity S between the ith file and the jth file is calculated using the following formula ij and the similarity S between the jth file and the ith file ji :

[0036]

[0037]

[0038] Where C ij For file F i With file F j The number of shared blocks between B i is the total number of blocks corresponding to the i-th file; B j is the total number of blocks corresponding to the jth file; the value of i is i∈(2,n), and the value of j is j∈(1,i-1);

[0039] d3. Based on the similarity obtained in step d2, the maximum similarity of each file is calculated using the following steps:

[0040] d3-1. Define a strongly connected graph G to store the similarities between files; the vertices in the graph G are the file set F, and the edges are the similarities between files;

[0041] d3-2. By comparing the weights of the edges, the file with the greatest similarity to each file is obtained, thereby obtaining data files sorted by the maximum similarity.

[0042] The deduplication operation described in step S3 specifically includes the following steps:

[0043] Using the data files sorted by maximum similarity, the maximum similarity max(S ij ) defines the sum of weighted similarities Sum to determine whether the file is redundant: where,

[0044]

[0045] Where D i Indicates file F i The size of; j∈(1,n) and i≠j;

[0046] Finally, a threshold h is set to remove noise data in the deduplication process: among the output files sorted according to the maximum similarity, files with similarity lower than the threshold h will not be deduplicated.

[0047] The rewriting based on the maximum similarity described in step S3 is performed to complete the deletion of duplicate medical data, which specifically includes the following steps:

[0048] (1) Obtain the data file of step S2, which is specifically the maximum similarity sorting file {F1, F2, ..., F n}, where the same as file F i-1 The most similar file is F i , i≤n;

[0049] (2) Define the intermediate file Class(i,i) to temporarily save the file F i The only block of

[0050] (3) When the file F i When deduplication is performed, compare the file F i and file F i-1 All blocks in:

[0051] If the current block is file F i-1 and F i If there is a shared block, the current block is written to the file Class(i,i-1);

[0052] If the current block is not file F i-1 and F iIf the current block is a shared block of file F i The only block and write the current block to the file Class(i,i);

[0053] (4) When the file F i+1 When deduplication is performed, compare the file F i+1 Compare the file F with the block of file Class(i,i-1) i+1 With the block of file Class(i,i):

[0054] File F i+1 and file F i The shared block is written to the file Class(i+1,i), and the file F i+1 and file F i-1 Continuous shared blocks are written to Class(i+1,i-1), and file F i+1 The unique block is written to the file Class(i+1,i+1); a continuous shared block is defined as a block shared by three consecutive blocks;

[0055] (5) Store the unique blocks in the files Class(i-1,i-1), Class(i,i-1), and Class(i,i), and delete the files Class(i-1,i-1), Class(i,i-1), and Class(i,i).

[0056] The deduplication method for medical big data provided by the present invention determines whether the medical data is redundant based on similarity calculation and removes noise data at the same time, rewrites the data based on the maximum similarity to improve the recovery performance and deletion performance, and effectively ensures the security and integrity of the medical data by designing a blockchain audit strategy and a deduplication recovery algorithm; therefore, the method of the present invention is suitable for the medical industry, and has high efficiency, good security, and can significantly reduce storage overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 The figure is a schematic diagram of the method flow of the present invention.

[0058] Figure 2 Schematic diagram of the overall framework of the method of the present invention.

[0059] Figure 3 The figure is a schematic diagram of the process of similarity calculation in the method of the present invention.

[0060] Figure 4 The figure is a schematic diagram of the process of parallel pipeline deduplication of data in the method of the present invention.

[0061] Figure 5 The figure is a schematic diagram of the maximum similarity rewriting process in the method of the present invention.

[0062] Figure 6 Schematic diagram of the blockchain structure in the method of the present invention.

[0063] Figure 7 Schematic diagram of performance comparison between the method of the present invention and the prior art.

[0064] Figure 8 Schematic diagram for comparing the audit efficiency of the method of the present invention on medical data of different sizes. DETAILED DESCRIPTION

[0065] like Figure 1 Shown is a schematic diagram of the method flow of the method of the present invention, Figure 2 The corresponding overall framework diagram is as follows: The method for deleting duplicate data for medical big data provided by the present invention comprises the following steps:

[0066] S1. Obtain medical data to be analyzed;

[0067] S2. Calculate the similarity of the medical data obtained in step S1, so as to obtain data files sorted by the maximum similarity; specifically, divide each data stream into segments of fixed size based on the similarity between files; then divide each segment into blocks within the segment, calculate and store the hash value of each block; then calculate the similarity of the files based on the number of shared blocks between the files; finally calculate the maximum similarity of each file, so as to obtain data files sorted by the maximum similarity;

[0068] The specific implementation includes the following steps:

[0069] a. Based on the similarity between files, each data stream is divided into segments of fixed size; specifically, each data stream is divided into segments of fixed size by parallel partitioning, thereby making full use of the CPU resources of multi-core and multi-processor;

[0070] b. Divide each segment obtained in step a into several blocks; specifically, divide the segments using the following steps and adjust the boundary positions of each block:

[0071] The data in the segment is divided into several blocks of different sizes using the content-based block method. Then, for the last block of each segment and the first block of the next segment, the positions of the two blocks are re-determined using the following steps: If the sliding window size is m, the last m-1 bytes of the last block of the current segment and the first m-1 bytes of the first block of the next segment are re-divided into blocks using the content-based block method.

[0072] like Figure 3As shown, the data stream is divided into multiple segments: {segment A1, segment A2, …, segment A n}, the block sets obtained by segment A1 and segment A2 are {C1,C2,…,C k} and {D1,D2,…,D m}, then re-adjust the last block position of segment A1 and the first block position of segment A2 to obtain a higher DER;

[0073] c. Calculate the hash value of each block obtained in step b (for example, use the SHA-1 function to calculate the hash value of each block obtained in step b) and store it;

[0074] d. Based on the number of shared blocks between files, the similarity of each file is calculated, thereby obtaining data files sorted according to the maximum similarity; specifically comprising the following steps:

[0075] d1. Input file set F{F1,F2,...,F n} and the total number of blocks corresponding to each file is a set B{B1,B2,...,B n};

[0076] d2. For the current input file F i , both will be the current input file F i The fingerprint of the first i-1 files F1~F i-1 The fingerprints of the ith file are compared in turn, and then the similarity S between the ith file and the jth file is calculated using the following formula ij and the similarity S between the jth file and the ith file ji :

[0077]

[0078]

[0079] Where C ij For file F i With file F j The number of shared blocks between B i is the total number of blocks corresponding to the i-th file; B j is the total number of blocks corresponding to the jth file; the value of i is i∈(2,n), and the value of j is j∈(1,i-1);

[0080] d3. Based on the similarity obtained in step d2, the maximum similarity of each file is calculated using the following steps:

[0081] d3-1. Define a strongly connected graph G to store the similarities between files; the vertices in the graph G are the file set F, and the edges are the similarities between files;

[0082] d3-2. By comparing the weights of the edges, the file with the greatest similarity to each file is obtained, thereby obtaining data files sorted by the greatest similarity;

[0083] In addition, using the maximum similarity max(S ij ) Define the sum of weighted similarities Sum to determine whether the file is redundant: The calculation of Sum is shown in the following formula:

[0084]

[0085] Where D i Indicates file F i The size of; j∈(1,n) and i≠j;

[0086] Finally, a threshold h is set to remove noise data in the deduplication process: among the output files sorted according to the maximum similarity, files with similarity lower than the threshold h will not be deduplicated;

[0087] In the specific implementation, a fingerprint index structure is defined to compare the number of shared blocks between files. The fingerprint index structure mainly includes three parts: SHA1-FP structure, EqualFP1() function and EqualAllFP() function. The SHA1-FP structure stores four fingerprints of SHA1: FP1, FP2, FP3 and FP4. EqualFP1() and EqualAllFP() are used to determine whether the fingerprints of two blocks are equal. EqualFP1() determines whether FP1 is equal. When FP1 is equal, EqualAllFP() is used to determine whether FP2, FP3 and FP4 are equal to determine whether the two fingerprints are consistent.

[0088] The pseudo code of this method is as follows:

[0089] Input: graph G, source vertex 0, total number of files n, file path set P, threshold h, all file size set D;

[0090] Output: the sum of weighted similarities Sum, the output path set outPath after removing noise based on the maximum similarity sorting;

[0091] 1: Define vector Vis to save the accessed vertices, define vector Du to save the file similarity, and variables tempVertex and tempDu to save the current output vertex and the maximum similarity of the current output vertex respectively.

[0092] 2:Du←-1,Du0←0,tempDuand tempVertex←0

[0093] 3:fori←0to n-1do

[0094] 4:Set lastVertexandlastDuto save last output vertex and the maximumsimilarity of last output vertex

[0095] 5:Define visitedVertex←-1,max←-1

[0096] 6:forj←0to n-1do

[0097] 7:ifvertex j is traversed and Du j ≥Maxthen

[0098] 8:max←Du j ,visitedVertex←j

[0099] 9:end if

[0100] 10:end for

[0101] 11:tempDu←Du j ,tempVertex←visitedVertex

[0102] 12:ifvisitedVertex=-1then

[0103] 13:return-1

[0104] 14:Vis[visitedVertex]←true

[0105] 15:D v ,D l ←Get the filesize of visitedVertex,lastVertex

[0106] 16:iflastDu>h&&tempDu>h then

[0107] 17:fileSum+=D v ,similaritySum+=tempDu×D v

[0108] 18:Writepath of visitedVertexto outPath

[0109] 19:elseiflastDu<h&&tempDu> h then

[0110] 20:fileSum+=D l ,similaritySum+=lastDu×D l

[0111] 21:fileSum+=D v ,similaritySum+=tempDu×D v

[0112] 22:Writepath oflastVertexandvisitedVertexto outPath

[0113] 23:end if

[0114] 24:Update the similarityDu between the output vertex and remaining unvisited vertexes

[0115] 25: end for

[0116] 26:return Sum←similaritySum / fileSum

[0117] The experimental results show that Sum and DER are almost the same, so the redundancy of data can be determined by the value of Sum. Hospital information systems or medical research institutions can easily determine whether medical data is redundant by this value. At the same time, the Simiprim algorithm can remove noise data by setting a threshold h, that is, in the output files sorted according to the maximum similarity, files with similarity lower than the threshold h are discarded. Because noise data rarely share blocks with other data, the noise data is directly stored without the need to perform deduplication.

[0118] S3. De-duplication of the data file obtained in step S2 is performed, and then rewritten based on the maximum similarity, thereby deleting duplicate medical data;

[0119] Traditional container-based deduplication methods generate a large amount of fragments during storage, destroying the locality of data; fragments are defined as unnecessary additional blocks loaded when reading a block;

[0120] by Figure 5 As shown: File1 has 6 blocks {Chunk1, Chunk2, Chunk3, Chunk4, Chunk5, Chunk6}, File2 has 5 blocks {Chunk1', Chunk2', Chunk3, Chunk4', Chunk5}; Assuming the size of the container is 3, the blocks of File1 are stored in the first two containers in sequence, among which Chunk3 and Chunk5 are shared blocks between File2 and File1; Therefore, the only block of File2 {Chunk1', Chunk2', Chunk4'} is stored in the third container; When looking for Chunk3 and Chunk5 of File2, the extra Chunk1 and Chunk2 of the first container and the extra Chunk4 of the second container are loaded respectively, among which Chunk3 and Chunk5 are defined as fragments of File2 because unnecessary blocks are loaded when reading blocks;

[0121] To address this problem, fragmentation can be minimized by rewriting blocks; for example, Chunk3 and Chunk5 of File1 are blocks shared by File1 and File2. If the only block of File1 is stored in Container1', the shared blocks of File1 and File2 are stored in Container2', and the only block of File2 is stored in Container3'. No fragmentation will be generated when File1 and File2 read the blocks; in addition, the present invention finds that the reference relationship of the blocks is highly correlated with the similarity of the files, that is, the more similar the two files are, the more shared blocks there are; therefore, the present invention proposes the following steps to further improve the recovery performance and deletion performance of the system:

[0122] (1) Obtain the data file of step S2, which is specifically the maximum similarity sorting file {F1, F2, ..., F n}, where the same as file F i-1 The most similar file is F i , i≤n;

[0123] (2) Define the intermediate file Class(i,i) to temporarily save the file F i The only block of

[0124] (3) When the file F i When deduplication is performed, compare the file F i and file F i-1 All blocks in:

[0125] If the current block is file F i-1 and F i If there is a shared block, the current block is written to the file Class(i,i-1);

[0126] If the current block is not file F i-1 and F i If the current block is a shared block of file F i The only block and write the current block to the file Class(i,i);

[0127] (4) When the file F i+1 When deduplication is performed, compare the file F i+1 Compare the file F with the block of file Class(i,i-1) i+1 With the block of file Class(i,i):

[0128] File F i+1 and file F i The shared block is written to the file Class(i+1,i), and the file F i+1 and file F i-1 Continuous shared blocks are written to Class(i+1,i-1), and file F i+1 The unique block is written to the file Class(i+1,i+1); a continuous shared block is defined as a block shared by three consecutive blocks;

[0129] (5) Store the only blocks in the files Class(i-1,i-1), Class(i,i-1), and Class(i,i), and delete the files Class(i-1,i-1), Class(i,i-1), and Class(i,i);

[0130] The above steps are performed in a parallel pipeline manner; since block division and fingerprint calculation are intensive computing tasks, their computing speed can be improved through multi-threading. Therefore, after the data is segmented, all subtasks of the block (block division, fingerprint calculation, indexing and storage) are processed by pipeline, and multi-threading is used for parallel block division and parallel hash calculation; pipeline means that when the current block is performing index search for fingerprint, the previous block can perform fingerprint calculation task, and the next block can perform data storage task;

[0131] like Figure 5As shown, file A, file B and file C are divided into 5 blocks respectively; the shared blocks between file B and file A are {Chunk1, Chunk3, Chunk4}, the shared blocks between file C and file A are {Chunk1, Chunk4}, and the shared blocks between file C and file B are {Chunk1, Chunk4, Chunk5'}; therefore, when deduplication is performed on file B, {Chunk1, Chunk3, Chunk4} and {Chunk2', Chunk5'} are written to file Class (A, B) and file Class (B, B) respectively; since {Chunk2, Chunk5} is the only block of file A, {Chunk2, Chunk5} is written to archive file Archive1;

[0132] When deleting files, since the archive file stores the unique blocks of each file, you only need to delete each archive file, and the time complexity of deletion is O(1).

[0133] In general, through the above method, the method of the present invention generates almost no fragments when restoring and deleting files; at the same time, a fast response time is obtained when performing the restoration and deletion operations;

[0134] S4. The traditional method of verifying data integrity is to calculate the hash of the original file after data recovery, and determine whether the data is correct by comparing whether the hash value is consistent with the hash of the metadata; however, for large-scale medical data, in addition to integrity, security is also an indispensable aspect, where integrity in the present invention refers to the correctness of the data, and security refers to the ability of the data to have correct authorized access and resist attacks;

[0135] Blockchain technology ensures the security and integrity of data through chain structure, encryption algorithm, timestamp and other technologies. Figure 6 As shown; however, there are few studies on medical data deduplication solutions based on blockchain, and large-scale medical data poses a greater challenge to the audit efficiency of blockchain; therefore, based on the previous module and the permission chain platform FISCO BCOS (FISCO BCOS is an alliance chain that supports data storage, multi-group transactions, group consensus, and group isolation), the present invention proposes the following method for securely protecting medical data after deduplication based on regional blockchain technology; specifically, it includes the following steps:

[0136] A. In order to ensure the integrity of medical data, the present invention designs a metadata structure for deduplication and a recovery algorithm for deduplication. Since deduplication relies on fingerprints and block sizes to locate the position of blocks in the container during recovery, the present invention designs the following metadata structure:

[0137] {<Fingerprint 1, Block 1 size>,...,<Fingerprint i, Block i size>,...,<Fingerprint n, Block n size>}

[0138] B. During data recovery, the block size in the metadata is used to find the block location in the container, and the block content is found based on the block fingerprint in the metadata;

[0139] C. When restoring a file, after locating the block position, calculate the block fingerprint for each block (for example, use the SHA-1 function to calculate the block fingerprint), and compare the metadata fingerprint with the calculated block fingerprint to see if they are the same:

[0140] If they are the same, the block is correct, and the correct block is used to recover the file;

[0141] If they are different, it means that the block has been tampered with; however, since the tampered block needs to locate the block based on the metadata, and the metadata is stored on the highly secure blockchain, the probability of a block being tampered with is extremely small;

[0142] D. When storing evidence data, store lightweight metadata files on the blockchain, and use the stored lightweight metadata files instead of the original files to calculate evidence, thereby improving audit efficiency;

[0143] In order to ensure the correctness of metadata and improve the audit efficiency of blockchain, the present invention designs a blockchain audit strategy; when storing evidence data, if the original medical data is directly stored on the blockchain, a large amount of storage space, computing resources and network bandwidth will be occupied when broadcasting messages between nodes, causing network congestion and poor availability, so the present invention stores lightweight data on the blockchain; since the deduplication algorithm recovery process relies on metadata, each metadata represents a unique file, so the present invention stores lightweight metadata files instead of original files to calculate evidence, and the metadata size is much smaller than the original file, so the audit efficiency will be greatly improved; when the average block size of deduplication is 4k=4096 bytes, the improvement is multiples of At the same time, the summary of metadata is stored on the blockchain, and users need to verify their permissions through private key signatures when accessing data, so the security mechanism of the blockchain ensures the security of medical data.

[0144] In addition, according to the obfuscation and diffusion principles, the audit efficiency proposed by the present invention improves the security of data. The present invention proves the security of the method through statistical analysis, as described below:

[0145] According to the obfuscation and diffusion principle, any small change in a binary file has a 50% probability of affecting each bit. In order to measure the obfuscation and diffusion performance of the method of the present invention, the present invention conducted K experiments on the same file on SHA-384 and the method of the present invention. Specifically, the SHA-384 of the original file was first calculated, and then 1 bit of the original file was arbitrarily modified to calculate the SHA-384 of the modified file. The two results were compared to obtain the number of changed bits, and a statistical analysis experiment was performed using specific parameters, which are defined as follows:

[0146] The average number of bits changed is

[0147] The percentage of the average number of bits changed is Y = (Z / L) × 100%;

[0148] The standard deviation of the bit number change is

[0149] The standard deviation of ΔZ is

[0150] The experimental results are shown in Table 1:

[0151] Table 1 Schematic diagram of security analysis of the method of the present invention and SHA-384

[0152]

[0153] The experimental analysis results are shown in Table 1. When the values ​​of Z and Y are 192 and 0.5 respectively, the obfuscation and diffusion effects of the algorithm are better, and the smaller the standard deviation, the better. From Table 1, it can be seen that the Z and Y of the method of the present invention are close to the theoretical values, and the standard deviation is smaller than SHA-384, so the method of the present invention has better anti-collision performance.

[0154] E. Build smart contracts based on the blockchain platform and establish a group isolation mechanism to achieve data sharing and data isolation among several medical institutions.

[0155] The effect of the method of the present invention is described below with reference to examples:

[0156] The present invention has been experimented on multiple medical data sets, and the experimental results are as follows: Figure 7 and Figure 8 shown. Figure 7 (a) shows that the method of the present invention achieves higher DER than the traditional methods based on fixed-size blocking, Rabin blocking and Fastcdc blocking, that is, it saves more storage space, and noise reduction has a significant effect on improving DER. Figure 7(b) shows that the recovery throughput of the method of the present invention is improved by 12.90 times, 9.16 times and 7.78 times respectively compared with the traditional methods based on fixed-size blocking, Rabin blocking and Fastcdc, which means that the method of the present invention significantly improves the recovery throughput of the system and eliminates fragmentation. Figure 7 (c) tested the deletion performance of the method of the present invention on different medical data sets. The results showed that the deletion throughput of the method of the present invention on each data set was improved by 10.68 times, 19.71 times and 4.25 times respectively compared with the traditional method based on Fastcdc. Figure 8 The storage performance of the method of the present invention for medical data of different sizes was tested. When the file size was 63.7M and 105.8M, the storage speed of the method of the present invention was 146 times and 148 times that of the traditional method, respectively, which is basically consistent with the above theoretical analysis. In general, compared with the existing work, the present invention, under the premise of being able to judge the redundancy of medical data, for the first time proposed to reduce storage overhead and improve overall performance by removing noise data; then, based on the maximum similarity rewriting algorithm, the similarity of data is used instead of the time attribute to eliminate the fragmentation generated by reading data blocks during recovery and deletion, thereby improving the recovery performance and deletion performance; finally, the audit strategy based on blockchain and the recovery algorithm for deduplication are designed to efficiently ensure the security and integrity of medical data.

[0157] The similarity calculation proposed in this invention can determine whether medical data is redundant and remove noise data at the same time. The rewriting algorithm based on maximum similarity improves the recovery performance and deletion performance. The security and integrity of medical data are efficiently guaranteed by designing blockchain audit strategies and duplicate data deletion recovery algorithms. Specifically, the main innovations are as follows:

[0158] (1) Based on the similarity calculation method, the redundancy of data can be determined by summing the weighted similarities, and the noise data that affects the system performance can be eliminated by setting a threshold, which significantly reduces the storage overhead and improves the backup throughput and recovery throughput. The method of the present invention applies denoising to deduplication technology for the first time to improve the overall system performance and reduce overhead.

[0159] (2) Based on the maximum similarity rewriting algorithm, the locality of data is utilized to classify and organize the data layout, which further improves the recovery performance and deletion performance of duplicate data deletion while maintaining a high DER.

[0160] (3) Based on blockchain, the present invention designs and implements a secure data deduplication solution for medical systems. The designed audit strategy improves the audit efficiency by 146 times, and statistical data analysis proves that the method has high security.

[0161] (4) The present invention has conducted a large number of experiments on various medical data, proving that the method of the present invention can not only ensure the security and integrity of medical data, but also greatly reduce storage overhead and improve overall performance, and can be easily deployed in hospital information systems, medical research institutions, etc.

Claims

1. A method for deduplicating medical big data, comprising the following steps: S1. Obtain medical data to be processed; S2. Calculate the similarity of the medical data obtained in step S1, so as to obtain a data file sorted by the maximum similarity; specifically comprising the following steps: a. Divide each data stream into fixed-size segments based on the similarity between files; b. Divide each segment obtained in step a into several blocks; c. Calculate the hash value of each block obtained in step b and store it; d. Based on the number of shared blocks between files, the similarity of each file is calculated, thereby obtaining data files sorted according to the maximum similarity; specifically comprising the following steps: d1. Input file set F{F1,F2,...,F n } and the total number of blocks corresponding to each file is a set B{B1,B2,...,B n }; d2. For the current input file F i , both will be the current input file F i The fingerprint of the first i-1 files F1~F i-1 The fingerprints of the ith file are compared in turn, and then the similarity S between the ith file and the jth file is calculated using the following formula ij and the similarity S between the jth file and the ith file ji : Where C ij For file F i With file F j The number of shared blocks between B i is the total number of blocks corresponding to the i-th file; B j is the total number of blocks corresponding to the jth file; the value of i is i∈(2,n), and the value of j is j∈(1,i-1); d3. Based on the similarity obtained in step d2, the maximum similarity of each file is calculated using the following steps: d3-1. Define a strongly connected graph G to store the similarities between files; the vertices in the graph G are the file set F, and the edges are the similarities between files; d3-2. By comparing the weights of the edges, the file with the greatest similarity to each file is obtained, thereby obtaining data files sorted by the greatest similarity; S3. Perform a deduplication operation on the data file obtained in step S2, and then rewrite it based on the maximum similarity, thereby completing the deletion of duplicate medical data; The deduplication operation specifically includes the following steps: Using the data files sorted by maximum similarity, the maximum similarity max(S ij ) defines the sum of weighted similarities Sum to determine whether the file is redundant: where, Where D i Indicates file F i The size of; j∈(1,n) and i≠j; Finally, a threshold h is set to remove noise data in the deduplication process: among the output files sorted according to the maximum similarity, files with similarity lower than the threshold h will not be deduplicated; The rewriting based on maximum similarity is performed to delete duplicate medical data, and specifically includes the following steps: (1) Obtain the data file of step S2, which is specifically the maximum similarity sorting file {F1, F2, ..., F n }, where the same as file F i-1 The most similar file is F i , i≤n; (2) Define the intermediate file Class(i,i) to temporarily save the file F i The only block of (3) When the file F i When deduplication is performed, compare the file F i and file F i-1 All blocks in: If the current block is file F i-1 and F i If there is a shared block, the current block is written to the file Class(i,i-1); If the current block is not file F i-1 and F i If the current block is a shared block of file F i The only block and write the current block to the file Class(i,i); (4) When the file F i+1 When deduplication is performed, compare the file F i+1 Compare the file F with the block of file Class(i,i-1) i+1 With the block of file Class(i,i): File F i+1 and file F i The shared block is written to the file Class(i+1,i), and the file F i+1 and file F i-1 Continuous shared blocks are written to Class(i+1,i-1), and file F i+1 The unique block is written to the file Class(i+1,i+1); a continuous shared block is defined as a block shared by three consecutive blocks; (5) Store the unique blocks in the files Class(i-1,i-1), Class(i,i-1), and Class(i,i), and delete the files Class(i-1,i-1), Class(i,i-1), and Class(i,i).

2. A method for deduplicating medical big data according to claim 1, characterized in that The following steps are also included: S4. Based on blockchain technology, medical data after deduplication is securely protected.

3. A method for deduplicating medical big data according to claim 2, characterized in that The blockchain technology is used to securely protect the medical data after deduplication, and specifically includes the following steps: A. Design the following metadata structure: {<Fingerprint 1, Block 1 size>,...,<Fingerprint i, Block i size>,...,<Fingerprint n, Block n size>} B. During data recovery, the block size in the metadata is used to find the block location in the container, and the block content is found based on the block fingerprint in the metadata; C. When restoring a file, after locating the block position, calculate the block fingerprint for each block and compare the metadata fingerprint with the calculated block fingerprint to see if they are the same: If they are the same, the block is correct, and the correct block is used to recover the file; If they are different, it means that the current block has been tampered with; D. When storing evidence data, store lightweight metadata files on the blockchain and use the stored lightweight metadata files instead of the original files to calculate evidence, thereby improving audit efficiency; E. Build smart contracts based on the blockchain platform and establish a group isolation mechanism to achieve data sharing and data isolation among several medical institutions.

4. A method for deduplicating medical big data according to claim 3, characterized in that The step a of dividing each data stream into segments of a fixed size specifically divides each data stream into segments of a fixed size in a parallel manner.

5. A method for deduplicating medical big data according to claim 4, characterized in that Step b divides each segment obtained in step a into several blocks, specifically by using the following steps to divide the segments and adjust the boundary positions of each block: The data in the segment is divided into several blocks of different sizes using the content-based blocking method. Then, for the last block of each segment and the first block of the next segment, the positions of the two blocks are re-determined using the following steps: If the sliding window size is m, the last m-1 bytes of the last block of the current segment and the first m-1 bytes of the first block of the next segment are re-divided into blocks based on the content-based blocking method.

Images