Data storage method based on on-chip compression and efficient data reconstruction

Abstract

The invention provides a data storage method based on on-chip compression and efficient data reconstruction, and relates to the field of data security, and the method comprises the following steps: 1, transmitting original data to an on-chip controller, and carrying out data segmentation; 2, generating an N-item polynomial containing original data; 3, obtaining k shares through a pseudo-random number generator; 4, performing dynamic compression operation on the addresses of the k shares; 5, when the data needs to be read or updated, t shares in the k shares are taken, and irrelevant d shares are read; step 6, solving through a formula to obtain complete first w coefficients; and 7, verifying the integrity of the data after polynomial reconstruction. According to the method, the original data is divided into a plurality of mutually incoherent shares, so that the confidentiality of the data content is greatly enhanced, the security of the data access mode is improved, and a series of overhead in the access mode protection process is reduced.

Description

Technical Field

[0001] The present application relates to the field of data security technology, and in particular to a data storage method based on on-chip compression and efficient data reconstruction. Background Art

[0002] Due to the interconnection of a large number of mobile smart devices on the Internet, a huge amount of data is generated. When a large amount of personal data needs to be outsourced to cloud computing, protecting sensitive and private data is critical for many applications. In today's intelligent society, people use machine learning to remotely collect, store and process a large amount of personal data for model training in order to achieve the ultimate goal, but an important problem is that this data may be leaked or abused by malicious servers. Therefore, in order to protect the security of stored data, it is necessary to pay attention to the protection of data content confidentiality.

[0003] Leading companies such as Intel and AMD have adopted Advanced Encryption Standard Counter Mode (AES-CTR) as their preferred memory encryption scheme. AES-CTR can pre-compute the encryption key stream to facilitate parallel processing, reduce effective encryption and decryption latency, and prevent sensitive information in storage from being exposed, violated, and compromised. However, a key challenge comes from the necessity to store the version number (VN) in memory and maintain its integrity. Without complete VNs, data cannot be accurately decrypted. Therefore, in order to ensure integrity, AES-CTR requires additional mechanisms such as message authentication codes and hash functions to maintain, but this not only increases storage requirements, but also causes considerable latency due to the complex verification process involved in each memory access. Although AES-CTR solves the problem of encrypting data content, encryption alone is not enough. Data access patterns may still be leaked, and attackers can infer some sensitive information from the number of queries and query paths of the client to the data.

[0004] In the prior art, an oblivious random access machine (ORAM) is usually used to protect the data in storage and the access pattern to alleviate this security issue. ORAM is one of the most effective cryptographic primitives for protecting the access pattern of stored data. It works by obfuscating the access pattern to memory so that each access looks random and is computationally indistinguishable from other accesses. However, the practicality of ORAM is hindered by its significant performance overhead. Even Path ORAM, one of the most popular tree structures, only requires half the virtual block storage capacity, but its performance is several orders of magnitude lower than non-ORAM memory access. Summary of the invention

[0005] The present application is made in view of the above-mentioned problems, and its purpose is to provide a data storage method based on on-chip compression and efficient data reconstruction, which divides the original data into multiple unrelated shares. If an external attacker obtains some of the shares, the original data will not be known, which greatly enhances the confidentiality of the data content; during the access process, only t random shares out of k shares need to be obtained from the memory to reconstruct the polynomial, and the confusion of d shares brings uncertainty and randomness to the access process, which improves the security of the data access mode and reduces a series of overheads of the access mode protection process; by dynamically compressing the addresses of the k shares corresponding to each data in the mapping table, the storage overhead of the mapping table is reduced, the expansion of the on-chip space is increased, and the on-chip storage overhead is reduced.

[0006] Specifically, a first aspect of the present application provides a data storage method based on on-chip compression and efficient data reconstruction, comprising the following steps: Step 1: When external data needs to be written to the storage, the original data is first transferred to the on-chip controller, and then the data is segmented to divide the original data into ~ part; Step 2: Generate an N-term polynomial containing the original data using the segmented data segments as partial coefficients, and save the coefficients of the polynomial. The polynomial is expressed as follows: ; in: ~ is the original data after segmentation; ~ The coefficient is a combination of the coefficient seed in the pseudo-random number generator and the controller; The purpose of saving the coefficients of the polynomial is to use them for subsequent data reconstruction and data integrity verification.

[0007] Step 3: After generating the polynomial, generate ~ Substitute a total of k random values ​​into the polynomial described in step 2 to obtain ~ There are k polynomial values ​​in total. Combining the two gives ~ There are k shares in total; Step 4: Store the obtained k shares into the memory. The mapping table needs to record the k shares and the corresponding original data, and dynamically compress the addresses of the k shares; The mapping table needs to record k shares and the corresponding original data, that is, each data corresponds to at least k shares. Compared with the traditional one-to-one mapping relationship, the storage overhead is more expensive and limits the on-chip space expansion. For this reason, the mapping table dynamically compresses the recorded share position to reduce the storage space it occupies.

[0008] Step 5: When the data needs to be read or updated, first check the location of the share corresponding to the data in the mapping table, take t of the k shares, and read the irrelevant d shares. The taken shares are { ~ } in the form of storage in Stash; Step 6: Save the coefficients ~ , and obtain w shares from Stash and substitute them into the polynomial described in step 2, and solve the complete first w coefficients through the formula of Barycentric interpolation theory. The formula is as follows: Solution formula: ; Reconstruct the polynomial after solving for the coefficients: ; in: ~ are the coefficients of the first w terms obtained by solving the formula; ~ are the polynomial coefficients saved in step 2; Step 7: After the polynomial is reconstructed, the integrity of the data is verified. If the verification passes, the first w coefficients are obtained. ~ Reorganize into original data and perform read, write or update operations normally; otherwise, return data error.

[0009] Furthermore, the on-chip controller is mainly an area for processing data, and this area is often protected in a computer. Under normal circumstances, any operation will not leak private information to the outside, and data can be processed safely.

[0010] Furthermore, the dynamic compression operation specifically includes: Step 1: The mapping table records the different offsets of the k shares corresponding to the original data in different data blocks in storage. The k shares from the same data must be stored in k different data blocks. In order to prevent the leakage of important information such as access mode, data blocks are set in the memory to store shares. A share usually occupies 16B, so a data block is set to store k shares. And the shares corresponding to one original data cannot be stored in the same data block, that is, k shares from the same data must be stored in k different data blocks.

[0011] Step 2: Encode the data block according to the offset, and the offset of the share position in the data block is also encoded and recorded in binary, and the final recorded value is represented by a binary number pair; Step 3: Dynamically compress the pair based on the continuity of the binary '0 / 1' in the pair, and use a counter to record the number of consecutive repetitions of '0' in the pair. and the number of consecutive occurrences of '1' ; Step 4: Set the compression threshold As a criterion for whether compressibility is possible, if the counter records repeated values or If the value is larger than the compression threshold, compression can be performed. or The data is converted into the corresponding binary as its compressed value, otherwise it cannot be compressed and the original value is retained.

[0012] Furthermore, after the dynamic compression operation is completed, it is necessary to determine whether the compressed value contains the initial value. If the compressed value contains the initial value, there is no need to add an additional identifier in front. If the compressed value does not contain the initial value, it is necessary to invert the first bit of the original initial value and then add it to the first bit of the compressed value to identify this compression and form the final value.

[0013] Furthermore, the read-independent d shares function to confuse access modes.

[0014] Furthermore, the relationship among k, t and w is k>t>w.

[0015] Furthermore, the data integrity verification specifically includes: since the t shares taken out from the storage are all decomposed by the same polynomial, the remaining (tw) shares are substituted into the reconstructed polynomial for verification. If each of the (tw) shares is The new calculated All of them are the same as the original share If the two data are the same, it means that the data has not been leaked or tampered with; otherwise, it means that the data has been leaked or tampered with.

[0016] In the second aspect, the present application also provides a computing device, which has the function of implementing the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect, which will not be repeated here. The function can be implemented by hardware, or by executing the corresponding software implementation through hardware. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the device includes an acquisition module, a training module, and optionally, a construction module. These modules can implement the function of the training node in the method example of the first aspect above. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0017] In the third aspect, the present application also provides a computing device, which is used to implement the functions of the method described in the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The structure of the computing device includes a processor and a memory, and the memory is used to store instructions and / or data. The memory is coupled to the processor, and when the processor executes the program instructions stored in the memory, the function of the training node in the example of the first aspect above can be implemented. The structure of the computing device also includes a communication interface for communicating with other devices.

[0018] In a fourth aspect, the present application also provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0019] In a fifth aspect, the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect and various possible designs of the first aspect.

[0020] In a sixth aspect, the present application also provides a computing chip, which is connected to a memory, and the chip is used to read and execute a software program stored in the memory, and to execute the methods in the above-mentioned first aspect and various possible implementation methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present drawings or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present drawings. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0022] Figure 1 A flowchart of the steps of a data storage method based on on-chip compression and data efficient reconstruction; Figure 2is a flow chart of a process of processing data by an on-chip controller; Figure 3 Flowchart of the on-chip binary data compression process.

[0023] The purpose, features and advantages of this figure will be further described in conjunction with the embodiments with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0025] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0026] Unless otherwise specified, all embodiments and optional embodiments of the present application can be combined with each other to form a new technical solution.

[0027] Unless otherwise specified, all technical features and optional technical features of this application can be combined with each other to form a new technical solution.

[0028] If there is no special explanation, all steps of the present application can be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), which means that the method may include steps (a) and (b) performed sequentially, or may include steps (b) and (a) performed sequentially. For example, the method may further include step (c), which means that step (c) may be added to the method in any order. For example, the method may include steps (a), (b) and (c), or may include steps (a), (c) and (b), or may include steps (c), (a) and (b), etc.

[0029] If there is no special explanation, the "include" and "comprising" mentioned in this application are open-ended or closed-ended. For example, the "include" and "comprising" may mean that other components not listed may also be included or only the listed components may be included or only the listed components may be included.

[0030] If not specifically stated, in this application, the term "or" is inclusive. For example, the phrase "A or B" means "A, B, or both A and B". More specifically, any of the following conditions satisfies the condition "A or B": A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); or both A and B are true (or exist).

[0031] In order to better understand the solutions of the embodiments of the present application, some relevant terms and concepts that may be involved in the embodiments of the present application are first introduced below.

[0032] (1) On-Chip Controller refers to a controller integrated on a single chip, usually used to manage specific functions or peripherals. These controllers can be an important part of embedded systems, used to achieve efficient data transmission, power management, clock control, etc. In computers, this area is often protected, and any operation under normal circumstances will not leak private information to the outside, so data can be processed securely.

[0033] (2) Map is a data structure used to store key-value pairs. Each key is unique, while the value can be repeated. The main feature of Map is the efficient search and insertion of key-value pairs.

[0034] (3) Stash buffer, usually used to temporarily store data so that it can be restored when needed. In Git, the gitstash command is used to save changes to the current working directory, which are stored in a stack-like buffer. Each stash record has a unique identifier that can be used to reference the stash record in subsequent operations.

[0035] (4) Pseudo-random number generator (PRNG) is an algorithm used to generate seemingly random number sequences. Although these number sequences have a certain degree of randomness in statistics, they are actually determined by the initial seed value and the algorithm. PRNG is usually based on an initial seed value and a series of complex mathematical operations (such as linear congruential generator, Mersenne twiddle and other algorithms) to generate seemingly random number sequences.

[0036] like Figure 1As shown, in this embodiment, a data storage method based on on-chip compression and data efficient reconstruction includes the following steps: Step 1: When external data needs to be written to the storage, the original data is first transferred to the on-chip controller, and then the data is segmented to divide the original data into ~ part; Step 2: Generate an N-term polynomial containing the original data using the segmented data segments as partial coefficients, and save the coefficients of the polynomial. The polynomial is expressed as follows: ; in: ~ is the original data after segmentation; ~ The coefficient is a combination of the coefficient seed in the pseudo-random number generator and the controller; The purpose of saving the coefficients of the polynomial is to use them for subsequent data reconstruction and data integrity verification.

[0037] Step 3: After generating the polynomial, generate ~ Substitute a total of k random values ​​into the polynomial described in step 2 to obtain ~ There are k polynomial values ​​in total. Combining the two gives ~ There are k shares in total; Polynomial Values To randomly generate Substitute x into the polynomial in step 2 to get the result.

[0038] Step 4: Store the obtained k shares into the memory. The mapping table needs to record the k shares and the corresponding original data, and dynamically compress the addresses of the k shares; The mapping table needs to record k shares and the corresponding original data, that is, each data corresponds to at least k shares. Compared with the traditional one-to-one mapping relationship, the storage overhead is more expensive and limits the on-chip space expansion. For this reason, the mapping table dynamically compresses the recorded share position to reduce the storage space it occupies.

[0039] Step 5: When the data needs to be read or updated, first check the location of the share corresponding to the data in the mapping table, take t of the k shares, and read the irrelevant d shares. The taken shares are { ~ } in Stash; in order to confuse external attackers and prevent data leakage, the system often only needs to use t shares out of k shares (k>t) to restore the polynomial.

[0040] At the same time, in order to obfuscate the data access pattern, the system can read additional insignificant d shares through the mapping table to confuse the access pattern. Then all shares taken from the memory will be { ~ } in the form of temporary storage in Stash for data reconstruction.

[0041] Step 6: Save the coefficients ~ , and obtain w shares from Stash and substitute them into the polynomial described in step 2, and solve the complete first w coefficients through the formula of Barycentric interpolation theory. The formula is as follows: Solution formula: ; Reconstruct the polynomial after solving for the coefficients: ; in: ~ are the coefficients of the first w terms obtained by solving the formula; ~ are the polynomial coefficients saved in step 2; Step 7: After the polynomial is reconstructed, the integrity of the data is verified. If the verification passes, the first w coefficients are obtained. ~ Reorganize into original data and perform read, write or update operations normally; otherwise, return data error.

[0042] Furthermore, the on-chip controller is mainly an area for processing data, which is often protected in a computer. Under normal circumstances, any operation will not leak private information to the outside, and data can be processed securely.

[0043] like Figure 2 As shown, the data segmentation, mapping and reconstruction are completed in the on-chip controller, and the segmented shares are stored in the storage area. When certain data is needed, it will be obtained from the storage area and then transmitted back to the on-chip controller.

[0044] Furthermore, the dynamic compression operation specifically includes: Step 1: The mapping table records the different offsets of the k shares corresponding to the original data in different data blocks in storage. The k shares from the same data must be stored in k different data blocks. In order to prevent the leakage of important information such as access mode, data blocks are set in the memory to store shares. A share usually occupies 16B, so a data block is set to store k shares. And the shares corresponding to one original data cannot be stored in the same data block, that is, k shares from the same data must be stored in k different data blocks.

[0045] Step 2: Encode the data block according to the offset, and the offset of the share position in the data block is also encoded and recorded in binary, and the final recorded value is represented by a binary number pair; In this embodiment, assuming that k is 256, there are at least 256 data blocks in the storage area and 256 shares can be stored in the data blocks. Therefore, if a share is located at the 128th position of the 128th data block, the final recorded value is represented by the number pair ('10000000', '10000000').

[0046] Step 3: Dynamically compress the pair based on the continuity of the binary '0 / 1' in the pair, and use a counter to record the number of consecutive repetitions of '0' in the pair. and the number of consecutive occurrences of '1' ; Step 4: Set the compression threshold As a criterion for whether compressibility is possible, if the counter records repeated values or If it is larger than the compression threshold, compression can be performed. or The data is converted into the corresponding binary as its compressed value, otherwise it cannot be compressed and the original value is retained.

[0047] In this embodiment, Figure 3 As shown in the figure, the compression process of the binary number '10000000' is as follows: first, the counter records the number of consecutive repetitions of '1' in the first segment of the number pair as R1=1, and then obtains the number of consecutive repetitions of '0' as R0=7, where the compression threshold is R1<3, and R0>3, so '10000000' is finally compressed to '1110', that is, the last three digits are compressed.

[0048] Furthermore, after the dynamic compression operation is completed, it is necessary to determine whether the compressed value contains the initial value. If the compressed value contains the initial value, there is no need to add an additional identifier in front. If the compressed value does not contain the initial value, it is necessary to invert the first bit of the original initial value and then add it to the first bit of the compressed value to identify this compression and form the final value.

[0049] The first 1 in the compressed value '1110' is the compressed value in the initial value that can mark the following three bits, so there is no need to add an additional mark in front. This reduces 4 bits compared to the previous 8-bit binary value. If the compressed value does not include the initial value, the first bit of the original initial value needs to be inverted and added to the first bit of the compressed value to mark this compression and form the final value. Although different binary compression situations will be different, due to the existence of the compression threshold, the overall value after compression will be significantly lower than the original size, reducing the storage space occupied.

[0050] Furthermore, the role of reading the irrelevant d shares is to confuse the access pattern.

[0051] Furthermore, the relationship among k, t, and w is k>t>w.

[0052] Furthermore, the integrity of the data is verified, specifically including: since the t shares taken out from the storage are all decomposed by the same polynomial, the remaining (tw) shares are substituted into the reconstructed polynomial for verification. If each of the (tw) shares is The new calculated All of them are the same as the original share If the two data are the same, it means that the data has not been leaked or tampered with; otherwise, it means that the data has been leaked or tampered with.

[0053] It should be noted that the present application is not limited to the above-mentioned embodiments. The above-mentioned embodiments are only examples, and the embodiments having the same structure as the technical idea and exerting the same effect within the scope of the technical solution of the present application are all included in the technical scope of the present application. In addition, without departing from the scope of the main purpose of the present application, various modifications that can be thought of by those skilled in the art to the embodiments and other methods of combining some of the constituent elements in the embodiments are also included in the scope of the present application.

Claims

1. A data storage method based on on-chip compression and efficient data reconstruction, characterized in that: The following steps are involved: Step 1: When external data needs to be written to the storage, the original data is first transferred to the on-chip controller, and then the data is segmented to divide the original data into ~ part; Step 2: Generate an N-term polynomial containing the original data using the segmented data segments as partial coefficients, and save the coefficients of the polynomial. The polynomial is expressed as follows: ; in: ~ is the original data after segmentation; ~ The coefficient is a combination of the coefficient seed in the pseudo-random number generator and the controller; Step 3: After generating the polynomial, generate ~ Substitute a total of k random values ​​into the polynomial described in step 2 to obtain ~ There are k polynomial values ​​in total. Combining the two gives ~ There are k shares in total; Step 4: Store the obtained k shares into the memory. The mapping table needs to record the k shares and the corresponding original data, and dynamically compress the addresses of the k shares; Step 5: When the data needs to be read or updated, first check the location of the share corresponding to the data in the mapping table, take t of the k shares, and read the irrelevant d shares. The taken shares are { ~ } in the form of storage in Stash; Step 6: Save the coefficients ~ , and obtain w shares from Stash and substitute them into the polynomial described in step 2, and solve the complete first w coefficients through the formula of Barycentric interpolation theory. The formula is as follows: Solution formula: ; Reconstruct the polynomial after solving for the coefficients: ； in: ~ are the coefficients of the first w terms obtained by solving the formula; ~ are the polynomial coefficients saved in step 2; Step 7: After the polynomial is reconstructed, the integrity of the data is verified. If the verification passes, the first w coefficients are obtained. ~ Reorganize into original data and perform read, write or update operations normally; otherwise, return data error.

2. A data storage method based on on-chip compression and data efficient reconstruction according to claim 1, characterized in that: The on-chip controller is mainly an area for processing data. This area is often protected in a computer. Under normal circumstances, any operation will not leak private information to the outside, and data can be processed safely.

3. A data storage method based on on-chip compression and data efficient reconstruction according to claim 1, characterized in that: The dynamic compression operation specifically includes: Step 1: The mapping table records the different offsets of the k shares corresponding to the original data in different data blocks in storage. The k shares from the same data must be stored in k different data blocks. Step 2: Encode the data block according to the offset, and the offset of the share position in the data block is also encoded and recorded in binary, and the final recorded value is represented by a binary number pair; Step 3: Dynamically compress the pair based on the continuity of the binary '0 / 1' in the pair, and use a counter to record the number of consecutive repetitions of '0' in the pair. and the number of consecutive occurrences of '1' ; Step 4: Set the compression threshold As a criterion for whether compressibility is possible, if the counter records repeated values or If it is larger than the compression threshold, compression can be performed. or The data is converted into the corresponding binary as its compressed value, otherwise it cannot be compressed and the original value is retained.

4. A data storage method based on on-chip compression and data efficient reconstruction according to claim 3, characterized in that: After the dynamic compression operation is completed, it is necessary to determine whether the compressed value contains the initial value. If the compressed value contains the initial value, there is no need to add an additional identifier in front. If the compressed value does not contain the initial value, it is necessary to invert the first bit of the original initial value and then add it to the first bit of the compressed value to identify this compression and form the final value.

5. The data storage method based on on-chip compression and data efficient reconstruction according to claim 1, characterized in that: The role of the read-independent d shares is to confuse the access mode.

6. A data storage method based on on-chip compression and data efficient reconstruction according to claim 1, characterized in that: The relationship among k, t and w is k>t>w.

7. A data storage method based on on-chip compression and data efficient reconstruction according to claim 1, characterized in that: The data integrity verification specifically includes: since the t shares taken out from the storage are all decomposed by the same polynomial, the remaining (tw) shares are substituted into the reconstructed polynomial for verification. If each of the (tw) shares is The new calculated All of them are in the original share If the data is the same as the original data, it means that the data has not been leaked or tampered with; otherwise, it means that the data has been leaked or tampered with.

Images