Reversible judgment method and device based on data element similarity calculation

Abstract

The invention provides a data element similarity calculation-based reversibility judgment method and device, and the method carries out the reversibility judgment of a data element based on the generalization of a feature field in the data element, and comprises the steps: S10, carrying out the similarity calculation of the feature field in the data element and corresponding original data; and S20, carrying out reversibility judgment on the data element based on a similarity calculation result. The method focuses on the data element, limits irreversible and reversible auditing scenes, is based on a standardized data structure, namely the data element, modal and combined state data elements in the data element define a main body field and a feature field, and can perform similarity matching on the feature field of the data element and original data in a very targeted manner; the scene of the data element is further limited, and the method is more accurate and safer.

Description

Technical Field

[0001] The present invention relates to the technical field of data resources, and in particular to a reversible determination method and device based on data element similarity calculation. Background Art

[0002] The reversibility of data components means that the data component model should ensure that the data component results cannot be reversed back to the data resource, ensuring that the data user cannot restore the original data based on the information provided by the data component to avoid leakage of the original data. Otherwise, it is called data reversibility.

[0003] At present, in the traditional information security field, symmetric encryption, asymmetric encryption, and homomorphic encryption technologies are mainly used to ensure that data cannot be reversible. Only certain entities holding keys and private keys can reversibly restore data. Therefore, if keys and private agreements are leaked, certain entities holding keys and private agreements can reversibly restore data.

[0004] Regarding reversible audit technology: Currently, encryption algorithms in the traditional information security field cannot be effectively audited. Once the key is leaked, the data can be reversible at any time. In addition, there are currently common text similarity algorithms such as edit distance, text vector cosine similarity, and Hamming distance. However, the above algorithms are not currently used in the field of reversible auditing, but are mainly used in the fields of information retrieval and text duplication detection. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to efficiently and reliably determine the reversibility of data elements. The present invention proposes a reversibility determination method and device based on data element similarity calculation.

[0006] According to a reversibility determination method based on data element similarity calculation according to an embodiment of the present invention, the method determines the reversibility of a data element based on the generalization of a feature field in the data element, and the method includes:

[0007] S10, calculating the similarity between the feature field in the data element and the corresponding original data;

[0008] S20, determining the reversibility of the data element based on the similarity calculation result.

[0009] According to some embodiments of the present invention, step S10 specifically includes:

[0010] S11, creating an inverted index for the original data;

[0011] S12, performing query matching between the feature fields in the data element and the original data based on the established inverted index, and performing similarity calculation.

[0012] In some embodiments of the present invention, in step S11, the types of the inverted index created include: text type, numerical type, and string type.

[0013] According to some embodiments of the present invention, in step S11, when establishing an inverted index, the word segmentation strategy adopted includes:

[0014] For text types, use the text tokenizer to perform tokenization;

[0015] For numeric types, tokenize based on commas and stop words between numeric values;

[0016] For string types, the N-gram model is used for word segmentation.

[0017] In some embodiments of the present invention, in step S12, before query matching, prefix and suffix truncation and word segmentation operations are performed on the feature field to be queried, including:

[0018] For text type feature fields, use word segmentation algorithm to split them;

[0019] For string type feature fields, the prefix and suffix of the preset fixed length are used for splitting;

[0020] For numeric type feature fields, remove the decimal point.

[0021] According to some embodiments of the present invention, in step S12, the similarity is calculated by one or more of an edit distance similarity algorithm, a Hamming distance, and a text vector cosine similarity.

[0022] In some embodiments of the present invention, before the similarity calculation is performed in step S10, the method further includes:

[0023] The data elements containing the same main field are merged, and the characteristic data fields with the same main field are merged before similarity calculation.

[0024] According to an embodiment of the present invention, a reversibility determination device based on data element similarity calculation performs reversibility determination of a data element based on the generalization of a feature field in the data element. The device includes:

[0025] A similarity calculation module calculates the similarity between the feature fields in the data element and the corresponding original data;

[0026] The determination result output module determines the reversibility of the data element based on the similarity calculation result.

[0027] According to the reversible determination device based on data element similarity calculation according to an embodiment of the present invention, the device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program implements the steps of the method described above when executed by the processor.

[0028] According to the computer storage medium of the embodiment of the present invention, a computer program is stored on the computer storage medium, and when the computer program is executed by a processor, the steps of the reversible determination method based on data element similarity calculation as described above are implemented.

[0029] The present invention has the following beneficial effects:

[0030] Traditional irreversible means, because they focus on the data itself, can only be reversible through encryption and desensitization, which makes it impossible to effectively carry out audit means. There are too many scenarios, and it is difficult to audit through a unified means, or even difficult to audit. The present invention focuses on data elements, limits irreversible and reversible audit scenarios, and is based on a standardized data structure - data elements. The modal and combined data elements in it define the main fields and feature fields, and can very specifically match the similarity between the feature fields of data elements and the original data, further limiting the scenarios of data elements, making it more accurate and secure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a structural diagram of a data element according to an embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of a data structure of a data element according to an embodiment of the present invention;

[0033] Figure 3 The flowchart of the reversible determination method based on data element similarity calculation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the present invention is described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0035] The description of the method flow in the specification of the present invention and the steps of the flowchart in the drawings of the specification of the present invention do not have to be strictly executed according to the step numbers, and the method steps can be executed in a different order. Moreover, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0036] In order to facilitate the understanding of the solution of the present invention, the technical terms involved in the present invention are first explained as follows:

[0037] Data component

[0038] It has a certain theme. After the data resources are desensitized, a data set is formed by several related fields as needed, or data features are formed by modeling the associated fields of data resources.

[0039] Data component modeldata componentmodel:

[0040] A data model constructed by certain rules, algorithms, etc. used in the process of developing data components using data resources. A means or method of processing data resources into data components.

[0041] composed data component:

[0042] A data set consisting of several related fields after data resources are desensitized.

[0043] modal data component:

[0044] Data features are formed by modeling the associated fields of data resources.

[0045] combinatorial data component:

[0046] A data set or data feature that is a combination of configurational and modal data elements.

[0047] The scheme of the present invention is described in detail below:

[0048] First of all, this method is based on the reversibility review of data components, and the structure of the component itself can ensure that the data is irreversible. To facilitate the understanding of the solution, we first introduce how the reversibility of data components is formed, and then explain how to conduct reversibility review. The structure of the data component is as follows:

[0049] like Figure 1 As shown in the figure, the structure of the data element includes the following four parts:

[0050] Data element type: mainly described from the morphological characteristics, including configuration data element, modal data element, combination data element and heterogeneous data element;

[0051] Naming rules: mainly described from the naming elements and naming methods;

[0052] Identifier compilation rules: mainly described from the identifier composition and identifier form;

[0053] Core metadata: mainly includes content, attributes and description.

[0054] Among them, the correlation calculation is based only on modal data elements and combined data elements. The data structures of these two data elements are as follows: Figure 2 shown.

[0055] Among them, the subject field represents the identification of a subject, such as user ID, ID number, mobile phone number, etc., and the feature field is the feature value calculated by the generalization model, not the original value. A subject field can be included in multiple data elements.

[0056] Combined with data elements, we can draw a preliminary conclusion that, through the processing of data elements, the original data associated with the subject identification is generalized, thereby protecting the privacy information of the subject. Then, the reversibility audit is aimed at whether its characteristic values ​​are truly generalized.

[0057] If the component developer mixes the original data when developing the feature field, such as adding prefixes, suffixes, or shuffling the text order, then the data may become reversible.

[0058] Therefore, similarity review mainly involves comparing feature fields with original data and determining reversibility through similarity algorithms.

[0059] According to the reversibility determination method based on data element similarity calculation according to an embodiment of the present invention, the method determines the reversibility of the data element based on the generalization of the characteristic field in the data element, and the method includes:

[0060] S10, calculating the similarity between the feature field in the data element and the corresponding original data, specifically including:

[0061] S11, creating an inverted index for the original data, where the types of the created inverted index include: text type, value type, and string type.

[0062] S12, performing query matching between the feature fields in the data element and the original data based on the established inverted index, and performing similarity calculation;

[0063] S20, determining the reversibility of the data element based on the similarity calculation result.

[0064] According to some embodiments of the present invention, in step S11, when establishing an inverted index, the word segmentation strategy adopted includes:

[0065] For text types, use the text tokenizer to perform tokenization;

[0066] For numeric types, tokenize based on commas and stop words between numeric values;

[0067] For string types, the N-gram model is used for word segmentation.

[0068] In some embodiments of the present invention, in step S12, before query matching, prefix and suffix truncation and word segmentation operations are performed on the feature field to be queried, including:

[0069] For text type feature fields, use word segmentation algorithm to split them;

[0070] For string type feature fields, use the preset fixed length prefix and suffix truncation to split them;

[0071] For numeric type feature fields, remove the decimal point.

[0072] According to some embodiments of the present invention, in step S12, the similarity is calculated by one or more of an edit distance similarity algorithm, a Hamming distance, and a text vector cosine similarity.

[0073] In some embodiments of the present invention, before the similarity calculation is performed in step S10, the method further includes: merging data elements containing the same main field, merging feature data fields of the same main field and then performing similarity calculation. Figure 3 As shown, each data element may correspond to an independent two-dimensional table. When there are multiple data elements, there may be multiple identical main fields in the multiple data elements. The characteristic data of the same main field in different data elements may be the same or different. Before similarity calculation, the characteristic data fields of the same main field are merged and then the similarity calculation is performed. This can increase the amount of information in the characteristic data, thereby facilitating the calculation of reversible data elements.

[0074] According to the reversibility determination device based on data element similarity calculation according to an embodiment of the present invention, the device determines the reversibility of the data element based on the generalization of the characteristic field in the data element, and the device includes: a similarity calculation module and a determination result output module.

[0075] Among them, the similarity calculation module performs similarity calculation on the feature fields in the data element and the corresponding original data;

[0076] The determination result output module determines the reversibility of the data element based on the similarity calculation result.

[0077] According to the reversible determination device based on data element similarity calculation according to an embodiment of the present invention, the device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the method described above are implemented.

[0078] According to the computer storage medium of the embodiment of the present invention, a computer program is stored on the computer storage medium, and when the computer program is executed by a processor, the steps of the reversible determination method based on data element similarity calculation as described above are implemented.

[0079] Refer to the following Figure 3 The similarity review process of the present invention is described with a specific embodiment. It should be understood that the following description is only an exemplary description and should not be construed as a specific limitation of the present invention.

[0080] A column of raw data can be regarded as a document, and the identifier of the document is the column. First, create an inverted index for these documents. The word segmentation strategy used in the index is as follows:

[0081] Text type (if it contains Chinese characters or English contains spaces, it is text type), use the ordinary text tokenizer;

[0082] For numeric types, commas, stop words, and other similar characters are used to connect the values, and words can be separated between the values.

[0083] For string type, since the query statement may use prefixes, suffixes, or strings, it is recommended to use N-gram for word segmentation to increase the probability of segmenting the hit word.

[0084] After the inverted index is established, each value of the feature field of the component is used as a query word to match the inverted index. If a match is found, the data component is considered to be reversible.

[0085] In addition, before searching, you need to perform prefix and suffix truncation and word segmentation on the query so that the query word is split into shorter strings, which makes it easier to find possible results. The query expansion strategy is as follows:

[0086] Text type: Split using word segmentation algorithm;

[0087] String type: split by truncating the prefix and suffix of a fixed length;

[0088] Number type: decimal point can be removed;

[0089] When matching index queries, corresponding extensions can also be made. For example, vector similarity calculation can be expanded to use edit distance similarity algorithm, Hamming distance, etc. Any similarity algorithm can be tried. The focus is on components and the above matching process.

[0090] In addition, it should be noted that for data reversibility review, data similarity can also be used to determine whether there is a reversible risk. First, the characteristic field data of the component is compared with the original data one by one, and the similarity is found through the text similarity algorithm. If they are similar, it is determined that this data component has a reversible risk.

[0091] In some cases, the data is directly encrypted, and the structure of the data cannot be directly obtained, and its main information and detailed data are unclear, and the reversibility cannot be determined by data similarity. The above method of the present invention is not limited by the data itself, and focuses on the data element to realize the reversibility determination of the data element.

[0092] The present invention has the following beneficial effects:

[0093] Traditional irreversible means, because they focus on the data itself, can only be reversible through encryption and desensitization, which makes it impossible to effectively carry out audit means. There are too many scenarios, and it is difficult to audit through a unified means, or even difficult to audit. The present invention focuses on data elements, limits irreversible and reversible audit scenarios, and is based on a standardized data structure - data elements. The modal and combined data elements in it define the main fields and feature fields, and can very specifically match the similarity between the feature fields of data elements and the original data, further limiting the scenarios of data elements, making it more accurate and secure.

[0094] Through the description of the specific implementation methods, a deeper and more specific understanding of the technical means and effects adopted by the present invention to achieve the predetermined purpose should be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not intended to limit the present invention.

Claims

1. A reversible determination method based on data element similarity calculation, characterized in that: The method determines the reversibility of the data element based on the generalization of the characteristic field in the data element, and the method includes: S10, calculating the similarity between the feature field in the data element and the corresponding original data; S20, determining the reversibility of the data element based on the similarity calculation result.

2. The reversible determination method based on data element similarity calculation according to claim 1, characterized in that: Step S10 specifically includes: S11, creating an inverted index for the original data; S12, performing query matching between the feature fields in the data element and the original data based on the established inverted index, and performing similarity calculation.

3. The reversible determination method based on data element similarity calculation according to claim 2, characterized in that: In step S11, the types of inverted indexes created include: text type, numerical type, and string type.

4. The reversible determination method based on data element similarity calculation according to claim 3, characterized in that: In step S11, when establishing the inverted index, the word segmentation strategies adopted include: For text types, use the text tokenizer to perform tokenization; For numeric types, word segmentation is performed based on commas and stop words between numeric values; For string types, the N-gram model is used for word segmentation.

5. The reversible determination method based on data element similarity calculation according to claim 3, characterized in that: In step S12, before query matching, prefix and suffix truncation and word segmentation operations are performed on the feature field to be queried, including: For text type feature fields, use word segmentation algorithm to split them; For string type feature fields, the prefix and suffix of the preset fixed length are used for splitting; For numeric type feature fields, remove the decimal point.

6. The reversible determination method based on data element similarity calculation according to claim 1, characterized in that: In step S12, the similarity is calculated by one or more of the edit distance similarity algorithm, Hamming distance, and text vector cosine similarity.

7. The reversible determination method based on data element similarity calculation according to claim 1, characterized in that: Before performing similarity calculation in step S10, the method further includes: The data elements containing the same main field are merged, and the characteristic data fields with the same main field are merged before similarity calculation.

8. A reversible determination device based on data element similarity calculation, characterized in that: The device determines the reversibility of the data element based on the generalization of the characteristic field in the data element, and the device includes: A similarity calculation module calculates the similarity between the feature fields in the data element and the corresponding original data; The determination result output module determines the reversibility of the data element based on the similarity calculation result.

9. A reversible determination device based on data element similarity calculation, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the reversible determination method based on data element similarity calculation as claimed in any one of claims 1 to 7.

Images