Unstructured data dynamic sampling storage method and device
By designing an unstructured data dynamic sampling storage method, the inconspicuous sampling rules and security problems are solved, the objectivity and security of sampling results are achieved, and the integrity and reproducibility of data are ensured.
Patent Information
- Application Number
- CN202411122831.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-03
AI Technical Summary
The sampling rules in the existing unstructured data sampling methods are not specific or fixed, resulting in the inability to reproduce the first sampling and the security of the sampling results and description information cannot be guaranteed.
By designing an unstructured data dynamic sampling storage method, the basic data structure and sampling data storage structure are adopted, including encrypted storage of sampling description information and sampling results, and sampling algorithms and encryption algorithms are used to ensure the objectivity and security of sampling results.
The sampling results are more objective and efficient, which facilitates first sampling and reproduction, and the security of sampling results and description information is improved through encryption algorithms to prevent data tampering.
Smart Images

Figure CN119066212A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for dynamically sampling and storing unstructured data. Background Art
[0002] Unstructured data sampling is to non-continuously select a subset from an unstructured data set as a sample to represent the characteristics and information of the entire data set. Unstructured data usually refers to data without a predefined pattern or format, such as text, images, audio, and video. Common methods for unstructured data sampling include: simple random sampling, systematic sampling, stratified sampling, expert sampling, cluster sampling, sequence sampling, and importance sampling.
[0003] However, the inventors of this application found that the above-mentioned prior art has at least the following technical problems in the process of implementing the technical solutions in the embodiments of this application:
[0004] The unstructured data sampling method in the prior art has the following technical problems: the sampling rules are not specific or fixed, the sampling description information and the sampling results are not encrypted and stored, which leads to the inability to reproduce the first sampling and the security of the sampling results and the sampling description information cannot be guaranteed. Summary of the invention
[0005] The embodiment of the present invention provides a method and device for dynamic sampling and storage of unstructured data, which is used to solve the technical problems that the sampling rules of unstructured data sampling methods in the prior art are not specific or fixed, and the sampling description information and sampling results are not encrypted and stored, which leads to the inability to reproduce the first sampling and the security of the sampling results and sampling description information cannot be guaranteed. The technical effects of more objective sampling results, high sampling efficiency, easy first sampling reproduction, and data tampering prevention are achieved, and the security of sampling results and sampling description information is improved through encryption algorithms.
[0006] In view of the above problems, an embodiment of the present application is proposed to provide a method and device for dynamic sampling and storage of unstructured data.
[0007] In a first aspect, the present invention provides a method for dynamic sampling and storage of unstructured data, the method comprising: reading unstructured data to be sampled; designing a structure of the sampled data according to a communication protocol, including a basic data structure and a sampled data storage structure:
[0008] The basic data structure is a sampling description information structure, and the sampling description information specifically includes: sampling data size, sampling data file name, sampling data file type, sampling data file extension, sampling intensity p, sampling length, sampling algorithm, sampling starting point, sampling interval, sample encryption algorithm and description block encryption algorithm;
[0009] The sampling data storage structure includes two parts: a description block and a data block, which are stored in the sampling data storage object.
[0010] Among them, the basic data structure BscDtStruct (Basic data structure) is defined as:
[0011] BscDtStruct::=D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 (1)
[0012] in,
[0013] D1::=<Sampling data size> (2)
[0014] D2::=<Sampling data file name> (3)
[0015] D3::=<Sampling data file type> (4)
[0016] D4::=<Extension of sampling data file> (5)
[0017] D5::=<Sampling intensity> (6)
[0018] D6::=<sampling length> (7)
[0019] D7::=<Sampling Algorithm> (8)
[0020] D8::=<Sampling start point> (9)
[0021] D9::=<Sampling interval> (10)
[0022] D10::=<Sample encryption algorithm> (11)
[0023] D11::=<Description of block encryption algorithm> (12)
[0024] The corresponding relationship between sampling intensity p and sampling rate q SampIntenToRate can be defined as:
[0025] SampIntenToRate::=(<sampling intensity p> <sampling rate q>) + (13)
[0026] In formula (13), the sampling intensity p and the sampling rate q are in a one-to-one correspondence.
[0027] The sampling data storage structure SampStoreStruct (Sampling data storage structure) is defined as:
[0028] SampStoreStruct::=<Description block (SampDesc)><Data block (SampData)> (14)
[0029] <Description block (sampDesc)>::=BscDtStruct (15)
[0030] <data block (sampData)> = <digit> + (16)
[0031] The description block (SampDesc) is used to store sampling description information, and the data block (SampData) is used to store sampling results.
[0032] Calculate the total capacity of the unstructured data to be sampled; select a sampling algorithm based on the total capacity of the unstructured data to be sampled and the sampling requirements; obtain the sampling rate of this sampling based on the sampling requirements; calculate the sampling length based on the total capacity of the unstructured data to be sampled and the sampling rate; input the read unstructured data to be sampled and the sampling length into the sampling algorithm, sample the unstructured data to be sampled, and obtain a sampling result; select a sample encryption algorithm based on the size of the sampling result; encrypt the sampling result through the sample encryption algorithm to obtain an encrypted sampling result; obtain a sampling data storage object, wherein the sampling data storage object includes a description block and a data block; store the sampling description information in the description block of the sampling data storage object, and store the encrypted sampling result in the data block.
[0033] Preferably, the step of storing the sampling description information in the description block of the sampling data storage object comprises:
[0034] Selecting a description block encryption algorithm for the sample data storage object, and obtaining the sample description information, and storing it in the description block of the sample data storage object;
[0035] The description block of the sample data storage object is encrypted using the description block encryption algorithm.
[0036] Preferably, the sampling requirements include:
[0037] The first sampling requirement is that each sample unit has an equal chance of being selected and does not depend on any subjective factors or prior selection;
[0038] The second sampling requirement is to select samples uniformly from the population according to a fixed interval or step size.
[0039] Preferably, the overall capacity of the sampled unstructured data is combined with the sampling requirements to select a sampling algorithm, including:
[0040] Determining the overall capacity of the unstructured data to be sampled;
[0041] If the total volume of the unstructured data to be sampled is small, a full sampling algorithm is selected;
[0042] If the overall capacity of the unstructured data to be sampled is large, and the sampling requirement is the first sampling requirement, a hash sampling algorithm is selected;
[0043] If the overall capacity of the unstructured data to be sampled is relatively large, and the sampling requirement is the second sampling requirement, an equidistant sampling algorithm is selected.
[0044] Preferably, obtaining the sampling rate of this sampling according to the sampling requirement includes:
[0045] Determine whether the sampling algorithm is full sampling;
[0046] If the sampling algorithm is not full sampling, the sampling rates corresponding to the sampling intensities at each level are configured according to the sampling requirements to determine the sampling intensity of this sampling;
[0047] According to the corresponding relationship between the sampling intensity and the sampling rate, the sampling rate of the current sampling is obtained.
[0048] The sampling intensity is divided into ten levels, and the correspondence between the sampling intensity and the sampling rate is: the first-level sampling intensity p1 corresponds to the sampling rate q1, the second-level sampling intensity p2 corresponds to the sampling rate q2, the third-level sampling intensity p3 corresponds to the sampling rate q3, the fourth-level sampling intensity p4 corresponds to the sampling rate q4, the fifth-level sampling intensity p5 corresponds to the sampling rate q5, the sixth-level sampling intensity p6 corresponds to the sampling rate q6, the seventh-level sampling intensity p7 corresponds to the sampling rate q7, the eighth-level sampling intensity p8 corresponds to the sampling rate q8, the ninth-level sampling intensity p9 corresponds to the sampling rate q9 and the tenth-level sampling intensity p10 corresponds to the sampling rate q10.
[0049] Preferably, the sampling length is calculated according to the overall capacity of the unstructured data to be sampled and the sampling rate, including:
[0050]
[0051] in, It is rounded up.
[0052] In a second aspect, the present invention provides a device for dynamically sampling and storing unstructured data, the device comprising:
[0053] A first reading unit, wherein the first reading unit is used to read unstructured data to be sampled;
[0054] A first calculation unit, the first calculation unit is used to calculate the overall capacity of the unstructured data to be sampled;
[0055] A first selection unit, the first selection unit is used to select a sampling algorithm according to the overall capacity of the unstructured data to be sampled and in combination with sampling requirements;
[0056] A second obtaining unit, the second obtaining unit is used to obtain a sampling rate of this sampling according to the sampling requirement;
[0057] A second calculation unit, the second calculation unit is used to calculate the sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate;
[0058] a third obtaining unit, the third obtaining unit being used to input the read unstructured data to be sampled and the sampling length into the sampling algorithm, sampling the unstructured data to be sampled, and obtaining a sampling result;
[0059] A second selection unit, the second selection unit is used to select a sample encryption algorithm according to the size of the sampling result;
[0060] A fourth obtaining unit, the fourth obtaining unit is used to encrypt the sampling result by using the sample encryption algorithm to obtain an encrypted sampling result;
[0061] A fifth obtaining unit, the fifth obtaining unit is used to obtain a sample data storage object, wherein the sample data storage object includes a description block and a data block;
[0062] The first storage unit is used to store the sampling description information in the description block of the sampling data storage object, and the encrypted sampling result is stored in the data block.
[0063] Preferably, the first storage unit includes:
[0064] a third selection unit, the third selection unit being used to select an encryption algorithm for a description block of the sample data storage object, and obtain the sampling description information, and store it in a description block of the sample data storage object;
[0065] A first encryption unit is used to encrypt the description block of the sample data storage object by using the description block encryption algorithm.
[0066] Preferably, the sampling requirements include:
[0067] The first sampling requirement is that each sample unit has an equal chance of being selected and does not depend on any subjective factors or prior selection;
[0068] The second sampling requirement is to select samples uniformly from the population according to a fixed interval or step size.
[0069] Preferably, the first selection unit includes:
[0070] A first judgment unit, the first judgment unit is used to judge the overall capacity of the unstructured data to be sampled;
[0071] a fourth selection unit, the fourth selection unit being configured to select a full sampling algorithm if the overall capacity of the unstructured data to be sampled is small;
[0072] A fifth selection unit, the fifth selection unit is used to select a hash sampling algorithm if the overall capacity of the unstructured data to be sampled is large and the sampling requirement is the first sampling requirement;
[0073] The sixth selection unit is used to select an equidistant sampling algorithm if the overall capacity of the unstructured data to be sampled is large and the sampling requirement is the second sampling requirement.
[0074] Preferably, the second obtaining unit comprises:
[0075] A second judgment unit, the second judgment unit is used to judge whether the sampling algorithm is full sampling;
[0076] A first determining unit, the first determining unit being used to configure the sampling rates corresponding to the sampling intensities at various levels according to the sampling requirements if the sampling algorithm is not full sampling, and determine the sampling intensity of this sampling;
[0077] A sixth obtaining unit, wherein the sixth obtaining unit is used to obtain the sampling rate according to the corresponding relationship between the sampling intensity and the sampling rate.
[0078] Preferably, the sampling intensity is divided into ten levels, and the correspondence between the sampling intensity and the sampling rate is: the first-level sampling intensity p1 corresponds to the sampling rate q1, the second-level sampling intensity p2 corresponds to the sampling rate q2, the third-level sampling intensity p3 corresponds to the sampling rate q3, the fourth-level sampling intensity p4 corresponds to the sampling rate q4, the fifth-level sampling intensity p5 corresponds to the sampling rate q5, the sixth-level sampling intensity p6 corresponds to the sampling rate q6, the seventh-level sampling intensity p7 corresponds to the sampling rate q7, the eighth-level sampling intensity p8 corresponds to the sampling rate q8, the ninth-level sampling intensity p9 corresponds to the sampling rate q9 and the tenth-level sampling intensity p10 corresponds to the sampling rate q10.
[0079] Preferably, the second calculation unit includes:
[0080]
[0081] in, It is rounded up.
[0082] In a third aspect, the present invention provides a dynamic sampling storage device for unstructured data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:
[0083] Read the unstructured data to be sampled; calculate the overall capacity of the unstructured data to be sampled; select a sampling algorithm based on the overall capacity of the unstructured data to be sampled and the sampling requirements; obtain the sampling rate of this sampling based on the sampling requirements; calculate the sampling length based on the overall capacity of the unstructured data to be sampled and the sampling rate; input the read unstructured data to be sampled and the sampling length into the sampling algorithm, sample the unstructured data to be sampled, and obtain a sampling result; select a sample encryption algorithm based on the size of the sampling result; encrypt the sampling result through the sample encryption algorithm to obtain an encrypted sampling result; obtain a sampling data storage object, wherein the sampling data storage object includes a description block and a data block; store the sampling description information in the description block of the sampling data storage object, and store the encrypted sampling result in the data block.
[0084] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:
[0085] Read the unstructured data to be sampled; calculate the overall capacity of the unstructured data to be sampled; select a sampling algorithm based on the overall capacity of the unstructured data to be sampled and the sampling requirements; obtain the sampling rate of this sampling based on the sampling requirements; calculate the sampling length based on the overall capacity of the unstructured data to be sampled and the sampling rate; input the read unstructured data to be sampled and the sampling length into the sampling algorithm, sample the unstructured data to be sampled, and obtain a sampling result; select a sample encryption algorithm based on the size of the sampling result; encrypt the sampling result through the sample encryption algorithm to obtain an encrypted sampling result; obtain a sampling data storage object, wherein the sampling data storage object includes a description block and a data block; store the sampling description information in the description block of the sampling data storage object, and store the encrypted sampling result in the data block.
[0086] The above one or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:
[0087] An embodiment of the present invention provides a method and device for dynamic sampling and storage of unstructured data, the method comprising: reading unstructured data to be sampled; calculating the overall capacity of the unstructured data to be sampled; selecting a sampling algorithm according to the overall capacity of the unstructured data to be sampled and in combination with sampling requirements; obtaining a sampling rate for this sampling according to the sampling requirements; calculating a sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate; inputting the read unstructured data to be sampled and the sampling length into the sampling algorithm, sampling the unstructured data to be sampled, and obtaining a sampling result; selecting a sample encryption algorithm according to the size of the sampling result; encrypting the sampling result by the sample encryption algorithm to obtain an encrypted sampling result; obtaining a sampling data storage object, wherein the sampling data storage object comprises a description block and a data block; storing sampling description information in the description block of the sampling data storage object, and storing the encrypted sampling result in the data block. It is used to solve the technical problems in the existing unstructured data sampling methods, such as the sampling rules are not specific or fixed, the sampling description information and sampling results are not encrypted and stored, which leads to the inability to reproduce the first sampling and the security of the sampling results and sampling description information cannot be guaranteed. The technical effects of more objective sampling results, high sampling efficiency, easy first sampling reproduction, and data tampering prevention are achieved, and the security of sampling results and sampling description information is improved through encryption algorithms.
[0088] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0089] Figure 1 A schematic diagram of a flow chart of a method for dynamic sampling and storage of unstructured data in an embodiment of the present invention;
[0090] Figure 2 Schematic diagram of the structure of a dynamic sampling storage device for unstructured data in an embodiment of the present invention;
[0091] Figure 3 Another structural schematic diagram of a device for dynamic sampling and storage of unstructured data in an embodiment of the present invention;
[0092] Figure 4 A screenshot of the unstructured text data to be sampled provided in an embodiment of the present invention;
[0093] Figure 5 A sampling result diagram of the unstructured text data to be sampled provided in an embodiment of the present invention;
[0094] Explanation of the accompanying drawings: first reading unit 1, first calculating unit 2, first selecting unit 3, second obtaining unit 4, second calculating unit 5, third obtaining unit 6, second selecting unit 7, fourth obtaining unit 8, fifth obtaining unit 9, first storage unit 10, bus 300, receiver 301, processor 302, transmitter 303, memory 304, bus interface 306. DETAILED DESCRIPTION
[0095] The embodiments of the present invention provide a method and device for dynamic sampling and storage of unstructured data, which solves the technical problems in the unstructured data sampling method in the prior art that the sampling rules are not specific or fixed, the sampling description information and the sampling results are not encrypted and stored, resulting in the inability to reproduce the first sampling and the security of the sampling results and the sampling description information cannot be guaranteed.
[0096] The overall idea of the technical solution provided by the present invention is as follows: read the unstructured data to be sampled; calculate the overall capacity of the unstructured data to be sampled; select a sampling algorithm based on the overall capacity of the unstructured data to be sampled and the sampling requirements; obtain the sampling rate of this sampling based on the sampling requirements; calculate the sampling length based on the overall capacity of the unstructured data to be sampled and the sampling rate; input the read unstructured data to be sampled and the sampling length into the sampling algorithm, sample the unstructured data to be sampled, and obtain a sampling result; select a sample encryption algorithm based on the size of the sampling result; encrypt the sampling result through the sample encryption algorithm to obtain an encrypted sampling result; obtain a sampling data storage object, wherein the sampling data storage object includes a description block and a data block; store the sampling description information in the description block of the sampling data storage object, and store the encrypted sampling result in the data block. The technical effects of making the sampling results more objective, the sampling efficiency higher, facilitating the reproduction of the first sampling, preventing data from being tampered with, and improving the security of the sampling results and sampling description information through encryption algorithms are achieved.
[0097] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.
[0098] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0099] Embodiment 1
[0100] Figure 1 FIG. 1 is a flow chart of a method for dynamic sampling and storage of unstructured data in an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a method for dynamic sampling and storage of unstructured data, the method comprising:
[0101] Step s1: read the unstructured data to be sampled;
[0102] Specifically, the unstructured data to be sampled may be text, image, audio, video and other data. In the embodiment of the present application, text data is selected as an example of unstructured data for explanation, and other types of unstructured data are similar.
[0103] Design the structure of sampling data according to the communication protocol, including the basic data structure and the sampling data storage structure:
[0104] The basic data structure is a sampling description information structure, and the sampling description information specifically includes: sampling data size, sampling data file name, sampling data file type, sampling data file extension, sampling intensity p, sampling length, sampling algorithm, sampling starting point, sampling interval, sample encryption algorithm and description block encryption algorithm;
[0105] The sampling data storage structure includes two parts: a description block and a data block, which are stored in the sampling data storage object.
[0106] Among them, the basic data structure BscDtStruct (Basic data structure) is defined as:
[0107] BscDtStruct::=D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 (1)
[0108] in,
[0109] D1::=<Sampling data size>=2349904 (2)
[0110] D2::=<Sample data file name>=《Information technology data quality evaluation indicators》 (3)
[0111] D3::=<Sampling data file type>=Text(4)
[0112] D4::=<Extension of sampling data file>=pdf (5)
[0113] D5::=<sampling intensity>=p1 (6)
[0114] D6::=<Sampling length>=23500 (7)
[0115] D7::=<Sampling Algorithm>=Hash Sampling (8)
[0116] D8::=<Sampling start point>=8 (9)
[0117] D9::=<Sampling interval>=-1 (10)
[0118] D10::=<Sample encryption algorithm>=RSA and AES hybrid encryption (11)
[0119] D11::=<Description of block encryption algorithm>=RSA algorithm (12)
[0120] The sampling interval D9 is -1, indicating that the sampling interval of the hash sampling algorithm adopted in this embodiment is not fixed.
[0121] The corresponding relationship between sampling intensity p and sampling rate q SampIntenToRate can be defined as:
[0122] SampIntenToRate::=(<sampling intensity p> <sampling rate q>) + (13)
[0123] In formula (13), the sampling intensity p and the sampling rate q are in a one-to-one correspondence.
[0124] The sampling data storage structure SampStoreStruct (Sampling data storage structure) is defined as:
[0125] SampStoreStruct::=<Description block (SampDesc)><Data block (SampData)> (14)
[0126] <Description block (sampDesc)>::=BscDtStruct (15)
[0127] <data block (sampData)> = <digit> + (16)
[0128] The description block (SampDesc) is used to store sampling description information, and the data block (SampData) is used to store sampling results.
[0129] Step s2: calculating the overall capacity of the unstructured data to be sampled;
[0130] Step s3: selecting a sampling algorithm according to the overall capacity of the unstructured data to be sampled and the sampling requirements;
[0131] Step s4: according to the sampling requirement, obtaining the sampling rate of this sampling;
[0132] Step s5: Calculate the sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate;
[0133] Step s6: inputting the read unstructured data to be sampled and the sampling length into the sampling algorithm, sampling the unstructured data to be sampled, and obtaining a sampling result;
[0134] Step s7: Select a sample encryption algorithm according to the size of the sampling result;
[0135] Step s8: encrypting the sampling result by the sample encryption algorithm to obtain an encrypted sampling result;
[0136] Step s9: obtaining a sample data storage object, wherein the sample data storage object includes a description block and a data block;
[0137] Step s10: The sampling description information is stored in the description block of the sampling data storage object, and the encrypted sampling result is stored in the data block.
[0138] Furthermore, the step of storing the sampling description information in the description block of the sampling data storage object includes:
[0139] Selecting a description block encryption algorithm for the sample data storage object, and obtaining the sample description information, and storing it in the description block of the sample data storage object;
[0140] The description block of the sample data storage object is encrypted using the description block encryption algorithm.
[0141] Specifically, firstly, the overall capacity of the unstructured text data to be sampled is calculated, and the corresponding sampling algorithm is selected in combination with the specific sampling requirements; then, according to the sampling requirements, the sampling rates corresponding to the sampling intensities at various levels are configured to determine the sampling intensity p of this sampling, and the sampling intensity p of this sampling is converted into the corresponding sampling rate q according to the corresponding relationship between the sampling intensity p and the sampling rate q; then, the overall capacity of the unstructured data to be sampled and the sampling rate q are substituted into formula (17) to calculate the sampling length:
[0142]
[0143] in, It is rounded up.
[0144] The read unstructured data to be sampled and the sampling length are input into the sampling algorithm to complete the sampling of the unstructured data to be sampled and obtain a sampling result.
[0145] In step s3, the overall capacity of the sampled unstructured data is combined with the sampling requirements to select a sampling algorithm, which specifically includes:
[0146] Determining the overall capacity of the unstructured data to be sampled;
[0147] If the total volume of the unstructured data to be sampled is small, a full sampling algorithm is selected;
[0148] If the overall capacity of the unstructured data to be sampled is large, and the sampling requirement is the first sampling requirement, a hash sampling algorithm is selected;
[0149] If the overall capacity of the unstructured data to be sampled is relatively large, and the sampling requirement is the second sampling requirement, an equidistant sampling algorithm is selected.
[0150] Furthermore, the sampling requirements include:
[0151] The first sampling requirement is that each sample unit has an equal chance of being selected and does not depend on any subjective factors or prior selection;
[0152] The second sampling requirement is to select samples uniformly from the population according to a fixed interval or step size.
[0153] The sampling algorithm and execution logic of the present invention are specifically as follows:
[0154] (1) Full sampling: Since full sampling is to sample all the unstructured data to be sampled, all the unstructured data to be sampled that are read are the sampling results.
[0155] (2) Hash sampling, the specific implementation process is as follows:
[0156] Step 1: Initialize a list for storing sampling results.
[0157] Step 2: Loop through each byte in the unstructured data to be sampled.
[0158] Step 3: Pass the byte data read each time into the predefined hash function for calculation, so as to obtain the hash value corresponding to each byte of the unstructured data to be sampled.
[0159] Step 4: Compare the hash value obtained in step 3 with the sampling rate q to determine whether to add the current byte to the list storing the sampling results.
[0160] Step 5: Calculate the length of the list storing the sampling results and compare it with the sampling length. If the length of the list storing the sampling results is equal to the sampling length, the loop ends. Return the list storing the sampling results, otherwise, continue the loop.
[0161] (3) Equidistant sampling, the specific implementation process is as follows:
[0162] Step 1: Configure the equidistant sampling starting point according to specific sampling requirements to ensure that the equidistant sampling starting point is within the range of zero to the overall capacity of the unstructured data to be sampled.
[0163] Step 2: Configure the sampling interval of equidistant sampling according to specific sampling requirements.
[0164] Step 3: Starting from the sampling starting point of the unstructured data to be sampled configured in the first step, perform equidistant sampling on the unstructured data to be sampled by bytes until the sampling length of data is sampled or the reading of the entire unstructured data to be sampled is completed, and the sampling result of equidistant sampling is obtained.
[0165] Specifically, in step s3, the overall capacity of the unstructured text data to be sampled according to the embodiment of the present application is calculated, such as Figure 4 As shown, it is determined that the overall capacity of the unstructured text data to be sampled is 2349904 bytes, so that a comprehensive analysis is unrealistic or too costly, and the sampling requirement is the first sampling requirement, that is, each sampling unit has an equal chance of being selected and is not affected by any subjective factors. It is further determined to select a hash sampling algorithm to sample the unstructured text data to be sampled.
[0166] In step s4, obtaining a sampling rate according to the sampling requirement includes:
[0167] Determine whether the sampling algorithm is full sampling;
[0168] If the sampling algorithm is not full sampling, the sampling rates corresponding to the sampling intensities at each level are configured according to the sampling requirements to determine the sampling intensity of this sampling;
[0169] According to the corresponding relationship between the sampling intensity and the sampling rate, the sampling rate of the current sampling is obtained.
[0170] Furthermore, the sampling intensity is divided into ten levels, and the correspondence between the sampling intensity p and the sampling rate q is: the first-level sampling intensity p1 corresponds to the sampling rate q1, the second-level sampling intensity p2 corresponds to the sampling rate q2, the third-level sampling intensity p3 corresponds to the sampling rate q3, the fourth-level sampling intensity p4 corresponds to the sampling rate q4, the fifth-level sampling intensity p5 corresponds to the sampling rate q5, the sixth-level sampling intensity p6 corresponds to the sampling rate q6, the seventh-level sampling intensity p7 corresponds to the sampling rate q7, the eighth-level sampling intensity p8 corresponds to the sampling rate q8, the ninth-level sampling intensity p9 corresponds to the sampling rate q9 and the tenth-level sampling intensity p10 corresponds to the sampling rate q10.
[0171] Specifically, in order to improve the sampling efficiency and verify the feasibility of the unstructured dynamic sampling method proposed in this patent, the sampling intensity p of this embodiment is determined to be a first-level sampling intensity. Wherein, the sampling intensity is divided into ten levels, and the corresponding relationship between the sampling intensity p and the sampling rate q is: the first-level sampling intensity p1 corresponds to the sampling rate q1 of 1%, the second-level sampling intensity p2 corresponds to the sampling rate q2 of 2%, the third-level sampling intensity p3 corresponds to the sampling rate q3 of 5%, the fourth-level sampling intensity p4 corresponds to the sampling rate q4 of 10%, the fifth-level sampling intensity p5 corresponds to the sampling rate q5 of 15%, the sixth-level sampling intensity p6 corresponds to the sampling rate q6 of 25%, the seventh-level sampling intensity p7 corresponds to the sampling rate q7 of 30%, the eighth-level sampling intensity p8 corresponds to the sampling rate q8 of 40%, the ninth-level sampling intensity p9 corresponds to the sampling rate q9 of 50%, and the tenth-level sampling intensity p10 corresponds to the sampling rate q10 of 60%.
[0172] First, according to the corresponding relationship between the sampling intensity p and the sampling rate q mentioned above, the first-level sampling intensity p1 is converted into the corresponding sampling rate q1=1%; then, according to the total capacity of the unstructured text data to be sampled of 2349904 bytes and the sampling rate q1=1%, the sampling length is calculated to be 23500 bytes using formula (17); finally, the read unstructured text data to be sampled and the sampling length of 23500 bytes are input into the selected hash sampling algorithm, thereby completing the sampling of the unstructured text data to be sampled. The sampling result SampRes is as follows: Figure 5 As shown, the technical effect of more objective sampling results and higher sampling efficiency is further achieved.
[0173] In step s6, the unstructured data to be sampled and the sampling length are input into the sampling algorithm, and the unstructured data to be sampled is sampled to obtain a sampling result;
[0174] The specific implementation process of hash sampling of the unstructured text data to be sampled in this embodiment is as follows:
[0175] Step 1: Initialize a list for storing sampling results to be empty.
[0176] Step 2: Loop through each byte in the unstructured text data to be sampled.
[0177] Step 3: Pass the byte data read each time into the predefined hash function for calculation, so as to obtain the hash value corresponding to each byte of the text data to be sampled. The specific execution process of the third step is as follows:
[0178] Step 1: Calculate the SHA-256 hash value of each byte data read.
[0179] Step 2: Convert the SHA-256 hash value of each byte data read in Step 1 into a hexadecimal string.
[0180] Step 3: Convert the hexadecimal string obtained in Step 2 into an integer.
[0181] Step 4: Compare the hash value integer corresponding to the current byte data obtained in step 4 with int(2^256*sampling rate q1) to determine whether to add the current byte data to the list storing the sampling results.
[0182] Step 5: Calculate the length of the list storing the sampling results and compare it with the sampling length. If the length of the list storing the sampling results is greater than or equal to the sampling length, the loop ends and returns to the list storing the sampling results. Otherwise, continue the loop.
[0183] In step s7, a sample encryption algorithm is selected according to the size of the sampling result; since the encryption target is text byte data and the data volume is relatively large, and combined with the security, availability and performance of the commonly used encryption algorithms, it is determined that the sampling result SampRes of this embodiment uses the RSA and AES hybrid encryption algorithm for data encryption. Further, the RSA and AES hybrid encryption execution process in step s7 is as follows:
[0184] Step 1: The sampling result acquisition party uses the RSA algorithm to generate a public key and private key pair. The public key is used to encrypt data, and the private key is used to decrypt data.
[0185] Step 2: The sampling result acquisition party passes the RSA public key generated in the first step to the sampling encryption party;
[0186] Step 3: Sample the encryption party and generate a random AES key with a length of 256 bits;
[0187] Step 4: The sampling encryption party uses the AES algorithm and the AES key in step 3 to encrypt the sampling results;
[0188] Step 5: Sample the encryption party and encrypt the AES key using the RSA public key received in step 2.
[0189] In step s10, if Figure 4 As shown, the embodiment of the present application takes unstructured text data as an example, and the sampled text size: 2349904 bytes, the sampling file name: "Information Technology Data Quality Evaluation Indicators", the sampling file type: text, the sampling file extension: pdf, the sampling intensity: p1, the sampling length: 23500 bytes, the sampling algorithm: hash sampling, the sampling starting point: 8, the sampling interval: -1, the sample encryption algorithm: RSA and AES mixed encryption, the description block encryption algorithm: RSA algorithm is stored in the description block of the sampling data storage object, and the text sampling result SampRes after RSA and AES mixed encryption is stored in the data block of the sampling data storage object. Among them, the sampling data storage object adopted in the example of the present application includes two parts: a description block SampDesc and a data block SampData. The description block SampDesc is used to store the description information of the sampling, including the sampling data size (occupying n1 Byte), the sampling data file name (occupying n2 Byte), the sampling data file type (occupying n3 Byte), the sampling data file extension (occupying n4 Byte), the sampling intensity (occupying n5 Byte), the sampling length (occupying n6 Byte), the sampling algorithm (occupying n7 Byte), the sampling starting point (occupying n8 Byte), the sampling interval (occupying n9 Byte), the sample encryption algorithm (occupying n10 Byte) and the description block encryption algorithm (occupying n11 Byte); the data block SampData (occupying n12 Byte) is used to store the encrypted sampling results. Among them, (n1, n2, n3, n4, n5, n6, n7, n8, n9, n10, n11, n12), the reference values are n1=8, n2=64, n3=16, n4=4, n5=8, n6=8, n7=16, n8=8, n9=8, n10=32, n11=32, n12=1860000.
[0190] In order to prevent the leakage of sampling description information, the embodiment of the present application selects the asymmetric encryption algorithm RSA as the encryption algorithm of the description block of the sampling data storage object, and stores the asymmetric encryption algorithm RSA in the encryption algorithm position of the description block of the sampling data storage object.
[0191] The description block of the sample data storage object is encrypted using the asymmetric encryption algorithm RSA. In step s10, the specific steps of encrypting the description block of the sample data storage object using the RSA algorithm are as follows:
[0192] Step 1: The sampling result acquirer uses the RSA algorithm to generate a pair of public and private keys. The public key is used to encrypt data, and the private key is used to decrypt data.
[0193] Step 2: The sampling result acquisition party passes the RSA public key generated in the first step to the sampling encryption party.
[0194] Step 3: The sampling encryption party converts the description block of the sampling data storage object into a character stream suitable for encryption.
[0195] Step 4: The sampling encryption party encrypts the description block of the sampling data storage object using the RSA public key received in the second step.
[0196] Step 5: The sampling encryption party transmits the encrypted sampling data storage object and the AES key encrypted by RSA when encrypting the sample data to the sampling result acquisition party.
[0197] Step 6: The sampling result acquirer first uses the RSA private key generated in the first step to decrypt the description block of the sampling data storage object, then uses the RSA private key generated when encrypting the sampling result to decrypt the AES key encrypted by RSA, and finally uses the decrypted AES key to decrypt the sampling result encrypted by AES to obtain the sampling result.
[0198] Embodiment 2
[0199] Based on the same inventive concept as the unstructured data dynamic sampling storage method in the aforementioned embodiment, the present invention also provides an unstructured data dynamic sampling storage device, such as Figure 2 As shown, the device comprises:
[0200] A first reading unit 1, the first reading unit is used to read the unstructured data to be sampled;
[0201] A first calculation unit 2, the first calculation unit is used to calculate the overall capacity of the unstructured data to be sampled;
[0202] A first selection unit 3, the first selection unit is used to select a sampling algorithm according to the overall capacity of the unstructured data to be sampled and in combination with sampling requirements;
[0203] A second obtaining unit 4, the second obtaining unit is used to obtain a sampling rate of this sampling according to the sampling requirement;
[0204] A second calculation unit 5, the second calculation unit is used to calculate the sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate;
[0205] A third obtaining unit 6, the third obtaining unit is used to input the read unstructured data to be sampled and the sampling length into the sampling algorithm, sample the unstructured data to be sampled, and obtain a sampling result;
[0206] A second selection unit 7, the second selection unit is used to select a sample encryption algorithm according to the size of the sampling result;
[0207] A fourth obtaining unit 8, the fourth obtaining unit is used to encrypt the sampling result by using the sample encryption algorithm to obtain an encrypted sampling result;
[0208] A fifth obtaining unit 9, the fifth obtaining unit is used to obtain a sample data storage object, wherein the sample data storage object includes a description block and a data block;
[0209] The first storage unit 10 is used to store the sampling description information in the description block of the sampling data storage object, and the encrypted sampling result is stored in the data block.
[0210] Further, the sampling description information includes: sampling data size, sampling data file name, sampling data file type, sampling data file extension, sampling intensity p, sampling length, sampling algorithm, sampling starting point, sampling interval, sample encryption algorithm and description block encryption algorithm;
[0211] Furthermore, the first storage unit 10 includes:
[0212] a third selection unit, the third selection unit being used to select an encryption algorithm for a description block of the sample data storage object, and obtain the sampling description information, and store it in a description block of the sample data storage object;
[0213] A first encryption unit is used to encrypt the description block of the sample data storage object by using the description block encryption algorithm.
[0214] Furthermore, the sampling requirements include:
[0215] The first sampling requirement is that each sample unit has an equal chance of being selected and does not depend on any subjective factors or prior selection;
[0216] The second sampling requirement is to select samples uniformly from the population according to a fixed interval or step size.
[0217] Furthermore, the first selection unit 3 includes:
[0218] A first judgment unit, the first judgment unit is used to judge the overall capacity of the unstructured data to be sampled;
[0219] a fourth selection unit, the fourth selection unit being configured to select a full sampling algorithm if the overall capacity of the unstructured data to be sampled is small;
[0220] A fifth selection unit, the fifth selection unit is used to select a hash sampling algorithm if the overall capacity of the unstructured data to be sampled is large and the sampling requirement is the first sampling requirement;
[0221] The sixth selection unit is used to select an equidistant sampling algorithm if the overall capacity of the unstructured data to be sampled is large and the sampling requirement is the second sampling requirement.
[0222] Furthermore, the second obtaining unit 4 includes:
[0223] A second judgment unit, the second judgment unit is used to judge whether the sampling algorithm is full sampling;
[0224] A first determining unit, the first determining unit being used to configure the sampling rates corresponding to the sampling intensities at various levels according to the sampling requirements if the sampling algorithm is not full sampling, and determine the sampling intensity of this sampling;
[0225] A sixth obtaining unit, wherein the sixth obtaining unit is used to obtain the sampling rate according to the corresponding relationship between the sampling intensity and the sampling rate.
[0226] Furthermore, the sampling intensity is divided into ten levels, and the correspondence between the sampling intensity p and the sampling rate q is: the first-level sampling intensity p1 corresponds to the sampling rate q1, the second-level sampling intensity p2 corresponds to the sampling rate q2, the third-level sampling intensity p3 corresponds to the sampling rate q3, the fourth-level sampling intensity p4 corresponds to the sampling rate q4, the fifth-level sampling intensity p5 corresponds to the sampling rate q5, the sixth-level sampling intensity p6 corresponds to the sampling rate q6, the seventh-level sampling intensity p7 corresponds to the sampling rate q7, the eighth-level sampling intensity p8 corresponds to the sampling rate q8, the ninth-level sampling intensity p9 corresponds to the sampling rate q9 and the tenth-level sampling intensity p10 corresponds to the sampling rate q10.
[0227] Furthermore, the second computing unit includes:
[0228]
[0229] in, It is rounded up.
[0230] The foregoing Figure 1 The various variations and specific examples of the unstructured data dynamic sampling storage method in Example 1 are also applicable to an unstructured data dynamic sampling storage device in this embodiment. Through the above detailed description of an unstructured data dynamic sampling storage method, those skilled in the art can clearly know the implementation method of an unstructured data dynamic sampling storage device in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here.
[0231] Embodiment 3
[0232] Based on the same inventive concept as the unstructured data dynamic sampling storage method in the aforementioned embodiment, the present invention also provides an unstructured data dynamic sampling storage device, on which a computer program is stored, and when the program is executed by a processor, the steps of any method of the unstructured data dynamic sampling storage method described above are implemented.
[0233] Among them, Figure 3 In the embodiment of the present invention, a bus architecture (represented by bus 300) is shown, which may include any number of interconnected buses and bridges, and bus 300 links various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are not further described herein. Bus interface 306 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, namely a transceiver, which provides a unit for communicating with various other devices over a transmission medium.
[0234] The processor 302 is responsible for managing the bus 300 and general processing, while the memory 304 may be used to store data used by the processor 302 when performing operations.
[0235] Embodiment 4
[0236] Based on the same inventive concept as the method for dynamic sampling and storage of unstructured data in the aforementioned embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the following steps when executed by a processor:
[0237] Read the unstructured data to be sampled; calculate the overall capacity of the unstructured data to be sampled; select a sampling algorithm based on the overall capacity of the unstructured data to be sampled and the sampling requirements; obtain the sampling rate of this sampling based on the sampling requirements; calculate the sampling length based on the overall capacity of the unstructured data to be sampled and the sampling rate; input the read unstructured data to be sampled and the sampling length into the sampling algorithm, sample the unstructured data to be sampled, and obtain a sampling result; select a sample encryption algorithm based on the size of the sampling result; encrypt the sampling result through the sample encryption algorithm to obtain an encrypted sampling result; obtain a sampling data storage object, wherein the sampling data storage object includes a description block and a data block; store the sampling description information in the description block of the sampling data storage object, and store the encrypted sampling result in the data block.
[0238] In the specific implementation process, when the program is executed by the processor, any method step in the first embodiment can also be implemented.
[0239] The above one or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:
[0240] An embodiment of the present invention provides a method and device for dynamic sampling and storage of unstructured data, the method comprising: reading unstructured data to be sampled; calculating the overall capacity of the unstructured data to be sampled; selecting a sampling algorithm according to the overall capacity of the unstructured data to be sampled and in combination with sampling requirements; obtaining a sampling rate for this sampling according to the sampling requirements; calculating a sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate; inputting the read unstructured data to be sampled and the sampling length into the sampling algorithm, sampling the unstructured data to be sampled, and obtaining a sampling result; selecting a sample encryption algorithm according to the size of the sampling result; encrypting the sampling result by the sample encryption algorithm to obtain an encrypted sampling result; obtaining a sampling data storage object, wherein the sampling data storage object comprises a description block and a data block; storing sampling description information in the description block of the sampling data storage object, and storing the encrypted sampling result in the data block. It is used to solve the technical problems in the existing unstructured data sampling methods, such as the sampling rules are not specific or fixed, the sampling description information and sampling results are not encrypted and stored, which leads to the inability to reproduce the first sampling and the security of the sampling results and sampling description information cannot be guaranteed. The technical effects of more objective sampling results, high sampling efficiency, easy first sampling reproduction, and data tampering prevention are achieved, and the security of sampling results and sampling description information is improved through encryption algorithms.
[0241] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0242] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0243] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0244] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0245] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for dynamic sampling and storage of unstructured data, characterized in that: The method comprises: Read unstructured data to be sampled; Design the structure of sampling data according to the communication protocol, including the basic data structure and the sampling data storage structure: The basic data structure is a sampling description information structure, and the sampling description information specifically includes: sampling data size, sampling data file name, sampling data file type, sampling data file extension, sampling intensity p, sampling length, sampling algorithm, sampling starting point, sampling interval, sample encryption algorithm and description block encryption algorithm; The sampling data storage structure includes two parts: a description block and a data block, which are stored in the sampling data storage object; Among them, the basic data structure BscDtStruct (Basic data structure) is defined as: BscDtStruct::=D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 (1) in, D1::=<Sampling data size>(2) D2::=<Sampling data file name>(3) D3::=<Sampling data file type>(4) D4::=<Extension of sampling data file>(5) D5::=<Sampling intensity>(6) D6::=<sampling length>(7) D7::=<Sampling algorithm>(8) D8::=<Sampling start point>(9) D9::=<sampling interval>(10) D10::=<Sample encryption algorithm>(11) D11::=<Description of block encryption algorithm>(12) The corresponding relationship between sampling intensity p and sampling rate q SampIntenToRate can be defined as: SampIntenToRate::=(<sampling intensity p> <sampling rate q>)+ (13) In formula (13), the sampling intensity p and the sampling rate q are in a one-to-one correspondence; The sampling data storage structure SampStoreStruct (Sampling data storage structure) is defined as: SampStoreStruct::=<Description block (SampDesc)><Data block (SampData)> (14) <Description block (sampDesc)>::=BscDtStruct (15) <data block (sampData)> = <digit> + (16) Among them, the description block (SampDesc) is used to store sampling description information, and the data block (SampData) is used to store sampling results; Calculating the overall capacity of the unstructured data to be sampled; Selecting a sampling algorithm according to the overall capacity of the unstructured data to be sampled and in combination with sampling requirements; According to the sampling requirements, the sampling rate of this sampling is obtained; Calculating a sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate; Inputting the read unstructured data to be sampled and the sampling length into the sampling algorithm, sampling the unstructured data to be sampled, and obtaining a sampling result; Selecting a sample encryption algorithm according to the size of the sampling result; Encrypting the sampling result by the sample encryption algorithm to obtain an encrypted sampling result; Obtaining a sample data storage object, wherein the sample data storage object includes a description block and a data block; The sampling description information is stored in the description block of the sampling data storage object, and the encrypted sampling result is stored in the data block.
2. The method according to claim 1, characterized in that The description block for storing the sampling description information in the sampling data storage object includes: Selecting a description block encryption algorithm for the sample data storage object, and obtaining the sample description information, and storing it in the description block of the sample data storage object; The description block of the sample data storage object is encrypted using the description block encryption algorithm.
3. The method according to claim 1, characterized in that The sampling requirements include: The first sampling requirement is that each sample unit has an equal chance of being selected and does not depend on any subjective factors or prior selection; The second sampling requirement is to select samples uniformly from the population according to a fixed interval or step size.
4. The method according to claim 3, characterized in that According to the overall capacity of the unstructured data to be sampled and the sampling requirements, a sampling algorithm is selected, including: Determining the overall capacity of the unstructured data to be sampled; If the total volume of the unstructured data to be sampled is small, a full sampling algorithm is selected; If the overall capacity of the unstructured data to be sampled is large, and the sampling requirement is the first sampling requirement, a hash sampling algorithm is selected; If the overall capacity of the unstructured data to be sampled is relatively large, and the sampling requirement is the second sampling requirement, an equidistant sampling algorithm is selected.
5. The method according to claim 4, characterized in that The step of obtaining the sampling rate of the current sampling according to the sampling requirement includes: Determine whether the sampling algorithm is full sampling; If the sampling algorithm is not full sampling, the sampling rates corresponding to the sampling intensities at each level are configured according to the sampling requirements to determine the sampling intensity of this sampling; According to the corresponding relationship between the sampling intensity and the sampling rate, the sampling rate of the current sampling is obtained; The sampling intensity is divided into ten levels, and the correspondence between the sampling intensity and the sampling rate is: the first-level sampling intensity p1 corresponds to the sampling rate q1, the second-level sampling intensity p2 corresponds to the sampling rate q2, the third-level sampling intensity p3 corresponds to the sampling rate q3, the fourth-level sampling intensity p4 corresponds to the sampling rate q4, the fifth-level sampling intensity p5 corresponds to the sampling rate q5, the sixth-level sampling intensity p6 corresponds to the sampling rate q6, the seventh-level sampling intensity p7 corresponds to the sampling rate q7, the eighth-level sampling intensity p8 corresponds to the sampling rate q8, the ninth-level sampling intensity p9 corresponds to the sampling rate q9 and the tenth-level sampling intensity p10 corresponds to the sampling rate q10.
6. The method according to claim 5, characterized in that The sampling length is calculated based on the total capacity of the unstructured data to be sampled and the sampling rate, including: in, It is rounded up.
7. A dynamic sampling storage device for unstructured data, characterized in that: The device comprises: A first reading unit, wherein the first reading unit is used to read unstructured data to be sampled; A first calculation unit, the first calculation unit is used to calculate the overall capacity of the unstructured data to be sampled; A first selection unit, the first selection unit is used to select a sampling algorithm according to the overall capacity of the unstructured data to be sampled and in combination with sampling requirements; A second obtaining unit, the second obtaining unit is used to obtain a sampling rate of this sampling according to the sampling requirement; A second calculation unit, the second calculation unit is used to calculate the sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate; a third obtaining unit, the third obtaining unit being used to input the read unstructured data to be sampled and the sampling length into the sampling algorithm, sampling the unstructured data to be sampled, and obtaining a sampling result; A second selection unit, the second selection unit is used to select a sample encryption algorithm according to the size of the sampling result; A fourth obtaining unit, the fourth obtaining unit is used to encrypt the sampling result by using the sample encryption algorithm to obtain an encrypted sampling result; A fifth obtaining unit, the fifth obtaining unit is used to obtain a sample data storage object, wherein the sample data storage object includes a description block and a data block; The first storage unit is used to store the sampling description information in the description block of the sampling data storage object, and the encrypted sampling result is stored in the data block.
8. A dynamic sampling storage device for unstructured data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the following steps are implemented: Read unstructured data to be sampled; Calculating the overall capacity of the unstructured data to be sampled; Selecting a sampling algorithm according to the overall capacity of the unstructured data to be sampled and in combination with sampling requirements; According to the sampling requirements, the sampling rate of this sampling is obtained; Calculating a sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate; Inputting the read unstructured data to be sampled and the sampling length into the sampling algorithm, sampling the unstructured data to be sampled, and obtaining a sampling result; Selecting a sample encryption algorithm according to the size of the sampling result; Encrypting the sampling result by the sample encryption algorithm to obtain an encrypted sampling result; Obtaining a sample data storage object, wherein the sample data storage object includes a description block and a data block; The sampling description information is stored in the description block of the sampling data storage object, and the encrypted sampling result is stored in the data block.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the following steps are implemented: Read unstructured data to be sampled; Calculating the overall capacity of the unstructured data to be sampled; Selecting a sampling algorithm according to the overall capacity of the unstructured data to be sampled and in combination with sampling requirements; According to the sampling requirements, the sampling rate of this sampling is obtained; Calculating a sampling length according to the overall capacity of the unstructured data to be sampled and the sampling rate; Inputting the read unstructured data to be sampled and the sampling length into the sampling algorithm, sampling the unstructured data to be sampled, and obtaining a sampling result; Selecting a sample encryption algorithm according to the size of the sampling result; Encrypting the sampling result by the sample encryption algorithm to obtain an encrypted sampling result; Obtaining a sample data storage object, wherein the sample data storage object includes a description block and a data block; The sampling description information is stored in the description block of the sampling data storage object, and the encrypted sampling result is stored in the data block.
