Real-time classification storage method of unstructured data based on FPGA hardware
By using a multi-core RISC-V processor based on FPGA hardware to classify and store unstructured data, the high risk of data leakage caused by the unified encryption method in the existing technology is solved, and data storage with higher security is achieved.
Patent Information
- Application Number
- CN202410648894.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-23
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-05-23
AI Technical Summary
Existing technologies use a uniform encryption method for storing unstructured data, which cannot meet the needs of all data and leads to a high risk of data leakage.
A multi-core RISC-V processor based on FPGA hardware is used to perform security verification on unstructured data. The data is divided into different security storage levels through a data classification strategy. The DMA module is used to store the data of different levels in DDR devices, and different encryption levels and access permissions are used for management.
By classifying and storing data, the risk of data leakage is reduced, the storage needs of different types of data are met, and security is improved.
Smart Images

Figure CN118550916B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unstructured data classification storage, and in particular to a real-time classification storage method for unstructured data based on FPGA hardware. Background Art
[0002] More than 80% of the world's total data is unstructured data. A large amount of unstructured data such as documents and images is stored and circulated in various terminals within the enterprise.
[0003] In the related art, when storing unstructured data, a unified encryption method is usually used to encrypt the stored data, which cannot meet the needs of all data and has a high risk of leakage. Summary of the invention
[0004] In order to solve the above technical problems, the present invention provides a real-time classification method for unstructured data based on FPGA hardware. The data to be stored is first divided into different security storage levels, and then different methods are used to store the data according to the security storage levels, thereby meeting the storage requirements of all data and greatly reducing the risk of data leakage.
[0005] The technical solution adopted by the present invention is as follows:
[0006] A real-time classification and storage method for unstructured data based on FPGA hardware, wherein a multi-core RISC-V processor is arranged on the FPGA, and the real-time classification and storage method for unstructured data comprises: upon receiving a data storage request instruction sent by an enterprise user, using the multi-core RISC-V processor to perform security verification on a data source corresponding to corresponding data to be stored, and after successful verification, communicating with a PCIE port of the FPGA through a DMA module to receive the data to be stored, wherein the PCIE port receives the data to be stored from an external device; using a data classification strategy to classify the data to be stored through the multi-core RISC-V processor, so as to divide the data to be stored into different security storage levels, and using the DMA module to store the data to be stored of different security storage levels into corresponding address spaces in a DDR device, wherein the encryption levels, access rights and management methods of the data in different address spaces are all different.
[0007] In one embodiment of the present invention, the data classification strategy is used to classify the data to be stored so as to divide the data to be stored into different security storage levels, including the following steps: obtaining the collection time of the data to be stored in the external device, the collector information corresponding to the data to be stored, and the historical browsing records; obtaining the first coefficient of the data to be stored according to the collection time, and obtaining the second coefficient of the data to be stored according to the collector information, and obtaining the third coefficient of the data to be stored according to the historical browsing records; obtaining the criticality coefficient of the data to be stored according to the first coefficient, the second coefficient and the third coefficient; and dividing the data to be stored into different security storage levels according to the criticality coefficient.
[0008] In one embodiment of the present invention, the first coefficient is calculated according to the acquisition time using the following formula:
[0009]
[0010] Among them, m1 is the first coefficient, T1 is the current time, T2 is the acquisition time, c1 and d1 are set values.
[0011] In one embodiment of the present invention, the second coefficient is calculated according to the collector information using the following formula:
[0012]
[0013] Among them, m2 is the second coefficient, L1 is the collector level, K is a set value greater than 1, and c2 is a set value.
[0014] In one embodiment of the present invention, obtaining the third coefficient of the data to be stored according to the historical browsing record includes: dividing the time from the acquisition time to the current moment into multiple time ranges, and setting a corresponding first proportional factor for each time range; obtaining the historical browsing frequency in each time range; and calculating the third coefficient according to the first proportional factor and the historical browsing frequency in each time range by the following formula:
[0015]
[0016] Wherein, m3 is the third coefficient, α is the adjustment coefficient, and χ i is the first proportional factor corresponding to the i-th time range, f i is the historical browsing frequency corresponding to the i-th time range, and n is the number of divided time ranges.
[0017] In one embodiment of the present invention, the historical browsing record includes the historical browsing frequency of the data to be stored, and the third coefficient of the data to be stored is obtained according to the historical browsing record by the following formula:
[0018] m3=c3*f,
[0019] Wherein, m3 is the third coefficient, c3 is a set value, and f is the historical browsing frequency of the data to be stored.
[0020] In one embodiment of the present invention, the criticality coefficient is calculated according to the first coefficient, the second coefficient and the third coefficient by the following formula:
[0021] m=β1*m1+β2*m2+β3*m3,
[0022] Among them, β1 is the proportional factor corresponding to the first coefficient, β2 is the proportional factor corresponding to the second coefficient, β3 is the proportional factor corresponding to the third coefficient, and m is the critical coefficient.
[0023] Beneficial effects of the present invention:
[0024] The present invention first divides the data to be stored into different security storage levels, and then adopts different methods to store the data according to the security storage levels, thereby being able to meet the storage requirements of all data and greatly reducing the risk of data leakage. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a flow chart of a method for real-time classification and storage of unstructured data based on FPGA hardware according to an embodiment of the present invention;
[0026] Figure 2 The present invention is a flowchart of a method for dividing data to be stored into different security storage levels according to an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] Figure 1 The present invention is a flowchart of a method for real-time classification and storage of unstructured data based on FPGA hardware according to an embodiment of the present invention.
[0029] It should be noted that a multi-core RISC-V (Reduced Instruction Set Computing) processor is provided on the FPGA (Field Programmable Gate Array).
[0030] like Figure 1 As shown, the real-time classification storage method for unstructured data based on FPGA hardware in an embodiment of the present invention may include the following steps:
[0031] S1, when receiving a data storage request instruction sent by an enterprise user, a multi-core RISC-V processor is used to perform security verification on the data source corresponding to the data to be stored, and after successful verification, the DMA module communicates with the PCIE port of the FPGA to receive the data to be stored. The PCIE port receives the data to be stored from an external device.
[0032] S2, using a multi-core RISC-V processor to classify the data to be stored using a data classification strategy, so as to divide the data to be stored into different security storage levels, and using a DMA module to store the data to be stored at different security storage levels into the corresponding address space in the DDR device. Among them, the encryption level, access rights and management methods of the data in different address spaces are all different.
[0033] Specifically, first, after receiving the data to be stored, the data classification strategy can be used to classify the data to be stored so as to divide the data to be stored into different security storage levels, and then the DMA module is used to store the data to be stored at different security storage levels into the corresponding address space in the DDR device. Among them, the encryption level, access rights and management methods of the data in different address spaces are all different.
[0034] It is understandable that the higher the security storage level of data, the more critical the corresponding data is. Therefore, a higher encryption level is required to encrypt it, the data management method is more stringent, and correspondingly, the access rights are also set higher.
[0035] In one embodiment of the present invention, Figure 2 As shown, a data classification strategy is used to classify the data to be stored so as to divide the data to be stored into different security storage levels, including the following steps:
[0036] S21, obtaining the collection time of the data to be stored in the external device, the collector information corresponding to the data to be stored, and the historical browsing record.
[0037] S22, obtaining a first coefficient of the data to be stored according to the collection time, obtaining a second coefficient of the data to be stored according to the collector information, and obtaining a third coefficient of the data to be stored according to the historical browsing record.
[0038] In one embodiment of the present invention, the first coefficient is calculated according to the acquisition time using the following formula:
[0039]
[0040] Among them, m1 is the first coefficient, T1 is the current time, T2 is the acquisition time, c1 and d1 are set values. Among them, c1 and d1 can be calibrated according to actual conditions.
[0041] Specifically, the longer the data collection time is from the current time, the greater the probability that the data is outdated, and the smaller the corresponding first coefficient is.
[0042] In one embodiment of the present invention, the second coefficient is calculated using the following formula according to the collector information:
[0043]
[0044] Wherein, m2 is the second coefficient, L1 is the collector level, K is a set value greater than 1, and c2 is a set value. K and c2 can be calibrated according to actual conditions.
[0045] Specifically, the higher the level of the data collector, the more critical the corresponding data is, and therefore, the larger the corresponding second coefficient is.
[0046] In one embodiment of the present invention, obtaining a third coefficient of the data to be stored according to the historical browsing records includes: dividing the time from the acquisition time to the current time into multiple time ranges, and setting a corresponding first proportional factor for each time range; obtaining the historical browsing frequency in each time range; and calculating the third coefficient according to the first proportional factor and the historical browsing frequency of each time range by the following formula:
[0047]
[0048] Among them, m3 is the third coefficient, α is the adjustment coefficient, χ i is the first proportional factor corresponding to the i-th time range, f i is the historical browsing frequency corresponding to the i-th time range, and n is the number of divided time ranges. Among them, α and n can be calibrated according to actual conditions.
[0049] In one embodiment of the present invention, the historical browsing record includes the historical browsing frequency of the data to be stored, and the third coefficient of the data to be stored is obtained according to the historical browsing record by the following formula:
[0050] m3=c3*f,
[0051] Wherein, m3 is the third coefficient, c3 is the set value, and f is the historical browsing frequency of the data to be stored. Wherein, c3 can be calibrated according to the actual situation.
[0052] Specifically, the more frequently the data is browsed, the more critical the corresponding data is, and therefore, the larger the corresponding third coefficient is.
[0053] S23, obtaining a critical coefficient of the data to be stored according to the first coefficient, the second coefficient and the third coefficient.
[0054] In one embodiment of the present invention, the criticality coefficient is calculated according to the first coefficient, the second coefficient and the third coefficient by the following formula:
[0055] m=β1*m1+β2*m2+β3*m3,
[0056] Among them, β1 is the proportional factor corresponding to the first coefficient, β2 is the proportional factor corresponding to the second coefficient, β3 is the proportional factor corresponding to the third coefficient, and m is the critical coefficient.
[0057] Among them, β1, β2 and β3 can be calibrated according to actual conditions.
[0058] S24, dividing the data to be stored into different security storage levels according to the criticality coefficient.
[0059] Specifically, each secure storage level may correspond to a criticality coefficient range, and the secure storage level may be determined based on the criticality coefficient range of the data to be stored. a , the corresponding security storage level is level 1; if m a <m≤m b , the corresponding security storage level is level 2; if m b <m, the corresponding security storage level is level three.
[0060] In summary, according to the FPGA hardware-based real-time classification storage method for unstructured data in an embodiment of the present invention, when a data storage request instruction sent by an enterprise user is received, a multi-core RISC-V processor is used to perform security verification on the data source corresponding to the corresponding data to be stored, and after the verification is successful, the DMA module is used to communicate with the PCIE port of the FPGA to receive the data to be stored, wherein the PCIE port receives the data to be stored from an external device, and the multi-core RISC-V processor adopts a data classification strategy to classify the data to be stored, so as to divide the data to be stored into different security storage levels, and the DMA module is used to store the data to be stored of different security storage levels in the corresponding address space in the DDR device, wherein the encryption level, access permission and management method of the data in different address spaces are different.
[0061] In the description of the present invention, “plurality” means two or more than two, unless otherwise clearly and specifically defined.
[0062] In the present invention, unless otherwise clearly specified and limited, terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integrated connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements.
[0063] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.
[0064] The execution order of each step shown in the flowchart is a preferred implementation mode. In other embodiments of the present invention, the steps may also be adjusted according to the functions involved in each step, for example, they may be executed simultaneously or in reverse order.
[0065] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing the logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transmit a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0066] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0067] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
Claims
1. A real-time classification storage method for unstructured data based on FPGA hardware. It is characterized in that A multi-core RISC-V processor is provided on the FPGA, and the real-time classification storage method of unstructured data includes: When receiving a data storage request instruction sent by an enterprise user, the multi-core RISC-V processor is used to perform security verification on the data source corresponding to the corresponding data to be stored, and after successful verification, the DMA module is used to communicate with the PCIE port of the FPGA to receive the data to be stored, wherein the PCIE port receives the data to be stored from an external device; The data to be stored is classified by the multi-core RISC-V processor using a data classification strategy to divide the data to be stored into different security storage levels, and the DMA module is used to store the data to be stored at different security storage levels in the corresponding address space of the DDR device, wherein the encryption level, access permission and management method of the data in different address spaces are different; wherein the data to be stored is classified by using the data classification strategy to divide the data to be stored into different security storage levels, including the following steps: Acquire the collection time of the data to be stored in the external device, the collector information corresponding to the data to be stored, and the historical browsing record; Acquire a first coefficient of the data to be stored according to the collection time, acquire a second coefficient of the data to be stored according to the collector information, and acquire a third coefficient of the data to be stored according to the historical browsing record; Acquire a critical coefficient of the data to be stored according to the first coefficient, the second coefficient and the third coefficient; The data to be stored is divided into different security storage levels according to the criticality coefficient.
2. The method for real-time classification and storage of unstructured data based on FPGA hardware according to claim 1, It is characterized in that The first coefficient is calculated using the following formula according to the acquisition time: Among them, m 1 is the first coefficient, T 1 is the current moment, T 2 is the acquisition time, c 1 and d 1 is the set value.
3. The method for real-time classification and storage of unstructured data based on FPGA hardware according to claim 2, It is characterized in that The second coefficient is calculated using the following formula according to the collector information: Among them, m 2 is the second coefficient, L 1 is the collector level, K is a set value greater than 1, c 2 is the set value.
4. The method for real-time classification and storage of unstructured data based on FPGA hardware according to claim 3, It is characterized in that Acquiring the third coefficient of the data to be stored according to the historical browsing record includes: Divide the time between the acquisition time and the current time into multiple time ranges, and set a corresponding first proportional factor for each time range; Get the historical browsing frequency in each time range; The third coefficient is calculated according to the first proportional factor and the historical browsing frequency of each time range by the following formula: Among them, m 3 is the third coefficient, α is the adjustment coefficient, χi is the first proportional factor corresponding to the i-th time range, and f i is the historical browsing frequency corresponding to the i-th time range, and n is the number of divided time ranges.
5. The method for real-time classification and storage of unstructured data based on FPGA hardware according to claim 3, It is characterized in that The historical browsing record includes the historical browsing frequency of the data to be stored, and the third coefficient of the data to be stored is obtained according to the historical browsing record by the following formula: m 3 =c 3 *f Among them, m 3 is the third coefficient, c 3 is a set value, and f is the historical browsing frequency of the data to be stored.
6. The method for real-time classification and storage of unstructured data based on FPGA hardware according to claim 4 or 5, It is characterized in that The criticality coefficient is calculated according to the first coefficient, the second coefficient and the third coefficient by the following formula: m=β 1 *m 1 +b 2 *m 2 +b 3 *m 3 , Among them, β 1 is the proportional factor corresponding to the first coefficient, β 2 is the proportional factor corresponding to the second coefficient, β 3 is the proportional factor corresponding to the third coefficient, and m is the critical coefficient.
