Method and system for privacy data protection in big data analysis process

Abstract

The invention discloses a method and a system for privacy data protection in a big data analysis process, and relates to the technical field of data protection. Comprising the steps of 1, carrying out metadata desensitization processing, 2, carrying out desensitization on an original problem of a user, and 3, carrying out privacy restoration on the desensitized user problem. According to the method, directional replacement and restoration of the privacy data are performed between the GBI and the large language model, sensitive data of an enterprise domain are replaced in the mode, and data privacy is protected.

Description

Technical Field

[0001] The present invention discloses a method and system for protecting privacy data in a big data analysis process, and relates to the technical field of data protection. Background Art

[0002] In recent years, large language models have made rapid progress in the field of applied technology. Using these models for data analysis not only lowers the technical threshold, but also enables people familiar with the business to quickly obtain the required analysis results. This has become a mainstream application trend. However, in this process, the protection of privacy data is still an unresolved issue. For example, the data analysis and mining system built on large language model technology is called GBI. Its characteristics are that it uses large language models to understand the natural language input by users and builds it into SQL. After executing queries on the database, it returns the results and renders them into charts. The current GBI implementation method is to directly pass user input to the large language model. Currently, MaaS is usually used directly for data input, but when MaaS is used, enterprise data will be out of the domain, which poses a risk of privacy exposure. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides a method and system for protecting privacy data in a big data analysis process, thereby solving the problem of privacy protection of a GBI system in the data analysis process.

[0004] The specific scheme proposed by the present invention is:

[0005] The present invention provides a privacy data protection method in a big data analysis process, comprising:

[0006] Step 1: Metadata desensitization processing:

[0007] Step 11: Generate the corresponding desensitized words based on the metadata information in the metadata vector storage of the GBI system.

[0008] Step 12: Create a corresponding table between desensitized words and metadata and store it in the metadata desensitization library.

[0009] Step 2: Desensitize the user's original question:

[0010] Step 21: Deploy a small parameter model locally and receive the user's original question through the small parameter model.

[0011] Step 22: Use a small parameter model to identify private words in the user's original question.

[0012] Step 23: Obtain the same or semantically similar field names from the metadata vector storage based on the privacy words, then retrieve the desensitized field names corresponding to the field names from the metadata desensitization library, generate a desensitized user question and desensitized word correspondence table and send it to the desensitized word cache.

[0013] Step 24: Send the desensitized user questions to the big model cloud service to generate the corresponding SQL.

[0014] Step 3: Restore privacy of desensitized user questions:

[0015] Step 31: Receive the desensitized word correspondence table through the desensitized word cache,

[0016] Step 32: Use the small parameter model to receive the SQL generated by the large model cloud service.

[0017] Step 33: Restore the user question after desensitization based on the desensitized word correspondence table in the desensitized word cache and the SQL generated by the large model cloud service.

[0018] Step 34: Generate SQL based on the restored user question and send it to the GBI system. After the GBI system executes it, it returns the data to the user.

[0019] Furthermore, in step 12 of the privacy data protection method for a big data analysis process, a corresponding table of desensitized words and metadata is established, including:

[0020] Preset the data structure of the corresponding table, the data structure includes field meaning, field name, data type and enumeration value;

[0021] In step 12, an open source relational database is used as the metadata desensitization library.

[0022] Furthermore, in step 12 of the method for protecting privacy data in a big data analysis process, a corresponding table of desensitized words and metadata is established, which further includes:

[0023] Establish the generation rules of the correspondence table:

[0024] Convert English field names to Chinese field names;

[0025] For industry-sensitive information, new names are generated based on organizational prompts.

[0026] Furthermore, step 23 of the method for protecting privacy data in a big data analysis process generates a desensitized word correspondence table, including: organizing the desensitized word correspondence table into a json format.

[0027] The present invention also provides a privacy data protection system for a big data analysis process, including a metadata desensitization module, a question desensitization module and a question restoration module.

[0028] The metadata desensitization module performs metadata desensitization processing:

[0029] Step 11: Generate the corresponding desensitized words based on the metadata information in the metadata vector storage of the GBI system.

[0030] Step 12: Create a corresponding table between desensitized words and metadata and store it in the metadata desensitization library.

[0031] The question desensitization module desensitizes the user's original question:

[0032] Step 21: Deploy a small parameter model locally and receive the user's original question through the small parameter model.

[0033] Step 22: Use a small parameter model to identify private words in the user's original question.

[0034] Step 23: Obtain the same or semantically similar field names from the metadata vector storage based on the privacy words, then retrieve the desensitized field names corresponding to the field names from the metadata desensitization library, generate a desensitized user question and desensitized word correspondence table, and send it to the desensitized word cache together with the privacy words.

[0035] Step 24: Send the desensitized user questions to the big model cloud service to generate the corresponding SQL.

[0036] The question restoration module restores the privacy of desensitized user questions:

[0037] Step 31: Receive the desensitized word correspondence table through the desensitized word cache,

[0038] Step 32: Use the small parameter model to receive the SQL generated by the large model cloud service.

[0039] Step 33: Restore the user question after desensitization based on the desensitized word correspondence table in the desensitized word cache and the SQL generated by the large model cloud service.

[0040] Step 34: Generate SQL based on the restored user question and send it to the GBI system. After the GBI system executes it, it returns the data to the user.

[0041] Furthermore, when the metadata desensitization module of the privacy data protection system for a big data analysis process executes step 12 to establish a corresponding table of desensitized words and metadata, it includes:

[0042] Preset the data structure of the corresponding table, the data structure includes field meaning, field name, data type and enumeration value;

[0043] An open source relational database is used as the metadata desensitizing library.

[0044] Furthermore, when the metadata desensitization module of the privacy data protection system for a big data analysis process executes step 12 to establish a correspondence table between desensitized words and metadata, it also includes:

[0045] Establish the generation rule of the correspondence table:

[0046] Convert English field names to Chinese field names;

[0047] For industry-sensitive information, new names are generated based on organizational prompts.

[0048] Furthermore, when the problem desensitizing module of the privacy data protection system of the big data analysis process executes step 23 to generate a desensitized word correspondence table, it includes: organizing the desensitized word correspondence table into a json format.

[0049] The benefits of the present invention are:

[0050] The present invention provides an intermediate layer to perform targeted replacement and restoration of private data between GBI and the large language model, and uses this method to replace the sensitive data of the enterprise out of the domain to protect data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the working process of the GBI system in the prior art.

[0052] Figure 2 It is a schematic diagram of the working process of the GBI system after applying the method of the present invention.

[0053] Figure 3 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0055] Example 1

[0056] GBI is a data analysis and mining system based on large language model technology. The general working process of the GBI system can be found in the attached Figure 1 The main function of step A is to send the user's original question and the library table (including fields) information retrieved from the metadata vector storage to the model cloud service, and require the model cloud service to return executable SQL.

[0057] Large model cloud service: a large language model deployed on a public cloud, providing a pay-as-you-go usage model. Metadata mainly refers to the database library, table, and field information, or the data set defined in the BI tool and the indicators and dimension information in the data set. Small parameter model: generally refers to a large language model with a model parameter volume of less than 7B as a small parameter model.

[0058] The method of the present invention deploys a small parameter model between step A and the model cloud service, and performs privacy replacement and privacy restoration through the small parameter model. For example, under the hardware conditions of 16 logical cores and 64GB memory, the small parameter model chatglm-6b can be run using chatglm.cpp to obtain an effect of 10-12 tokens per second.

[0059] The method of the present invention may include:

[0060] Step 1: Metadata desensitization processing:

[0061] Step 11: Generate the corresponding desensitized words based on the metadata information in the metadata vector storage of the GBI system. This process shortens the response time of step A and splits the data transmitted to the large model cloud service into two parts for processing. The largest proportion of the retrieved data set / library table, indicator / field information, that is, the metadata information part, generates desensitized words in advance.

[0062] Step 12: Create a correspondence table between desensitized words and metadata and store it in the metadata desensitization library.

[0063] The establishment of a correspondence table between desensitized words and metadata may further include:

[0064] The data structure of the corresponding table is preset, and the data structure includes field meaning, field name, data type and enumeration value; an open source relational database is used as the metadata desensitization library, such as Mysql, and the data structure can be referred to Table 1.

[0065] Table 1

[0066]

[0067] Step 2: Desensitize the user's original question:

[0068] Step 21: Deploy a small parameter model locally and receive the user's original question through the small parameter model.

[0069] Step 22: Use a small parameter model to identify private words in the user's original question.

[0070] Step 23: Obtain the same or semantically similar field names from the metadata vector storage based on the privacy words, then retrieve the desensitized field names corresponding to the field names from the metadata desensitization library, generate a desensitized user question and desensitized word correspondence table and send it to the desensitized word cache.

[0071] Step 24: Send the desensitized user questions to the big model cloud service to generate the corresponding SQL.

[0072] Step 3: Restore privacy of desensitized user questions:

[0073] Step 31: Receive the desensitized word correspondence table through the desensitized word cache,

[0074] Step 32: Use the small parameter model to receive the SQL generated by the large model cloud service.

[0075] Step 33: Restore the user question after desensitization based on the desensitized word correspondence table in the desensitized word cache and the SQL generated by the large model cloud service.

[0076] Step 34: Generate SQL based on the restored user question and send it to the GBI system. After the GBI system executes it, it returns the data to the user.

[0077] Example 2

[0078] For example, there is a metadata summary table in the database of the GBI system, as shown in Table 2.

[0079] Table 2

[0080] Fields Field Name Id Retail Account Number Addr Retailer Address Date Sales time Type Product Type ItemId Product Number Amount Sales quantity

[0081] Design an original user question: Query the total cigarette sales volume of Dingdang Convenience Store in DL City (number: 112124091601) in August 24.

[0082] The privacy data protection process can be referred to as follows:

[0083] Step 1: Metadata desensitization processing:

[0084] Step 11: Generate corresponding desensitized words based on the metadata information in the metadata vector storage of the GBI system.

[0085] Step 12: Establish a correspondence table between desensitized words and metadata and store it in the metadata desensitization library. This may further include: establishing a generation rule for the correspondence table:

[0086] Convert English field names into Chinese field names; for industry-sensitive information, organize prompt words to generate new names.

[0087] The correspondence between the generated desensitized words and metadata is shown in Table 3.

[0088] Table 3

[0089] Original field name Desensitized field name Id Customer Number Addr Customer Address Date Sales time Type Product Type ItemId Product Number Amount Sales quantity

[0090] The corresponding table is stored in the metadata desensitization library after uniqueness check.

[0091] Step 2: Desensitize the user's original question:

[0092] Step 21: Deploy a small parameter model locally and receive the user's original question through the small parameter model.

[0093] Step 22: Use a small parameter model to identify private words in the user's original question.

[0094] Step 23: Obtain the same or semantically similar field names from the metadata vector storage based on the privacy words, and then take out the desensitized field names corresponding to the field names from the metadata desensitization library, generate a desensitized user question and desensitized word correspondence table and send it to the desensitized word cache. For example, replace consecutive numbers with random numbers of the same number of digits, replace store names and product names with other names with practical meanings that do not have industry characteristics, and do not include tobacco in the names. Output two parts of content, the first part is the desensitized user question, and the second part is the correspondence between sensitive words and replacement words.

[0095] Generate a desensitized user question: "Query the total sales volume of daily necessities in DL City Convenience Store (number: 543216789012) in August 24." The generated sensitive word correspondence table is shown in Table 4. After this table is organized into json format, it is sent to the desensitized word cache.

[0096] Table 4

[0097] Sensitive words Corresponding words Dingdang Convenience Store in Dalian Dalian Convenience Store 112124091601 543216789012 cigarette Daily necessities

[0098] Step 24: Send the desensitized user question to the big model cloud service to generate the corresponding SQL. The user question sent to the big model cloud service is "Query the total sales value of daily necessities in DL City Convenience Home Supermarket (number: 543216789012) in August 24." The reference table is: Summary Table, with the fields: Customer Number, Customer Address, Sales Time, Product Type, Product Number, Total Amount.

[0099] The SQL replied by the large model cloud service can be: SELECT SUM(sales quantity) FROM summary table WHERE customer number = '543216789012' AND sales time BETWEEN '2024-08-01' AND '2024-09-01'.

[0100] Step 3: Restore privacy of desensitized user questions:

[0101] Step 31: Receive the desensitized word correspondence table through the desensitized word cache,

[0102] Step 32: Use the small parameter model to receive the SQL generated by the large model cloud service.

[0103] Step 33: According to the desensitized word corresponding table 4 in the desensitized word cache and Table 3, the SQL generated by the large model cloud service is restored to the user question after desensitization. The restored SQL is as follows: SELECT SUM(Amount)FROM summary table WHERE Id = '112124091601' AND Date BETWEEN '2024-08-01' AND '2024-09-01'.

[0104] Step 34: Generate SQL based on the restored user question and send it to the GBI system. After the GBI system executes it, it returns the data to the user.

[0105] The present invention adopts a combination of a small-parameter and large-language model on the intranet and a large-model cloud service running on the public network to perform intelligent data analysis, replace and restore sensitive words, protect data privacy, and meet the needs of low-investment sensitive enterprises.

[0106] Example 3

[0107] The present invention also provides a privacy data protection system for a big data analysis process, including a metadata desensitization module, a question desensitization module and a question restoration module.

[0108] The metadata desensitization module performs metadata desensitization processing:

[0109] Step 11: Generate the corresponding desensitized words based on the metadata information in the metadata vector storage of the GBI system.

[0110] Step 12: Create a corresponding table between desensitized words and metadata and store it in the metadata desensitization library.

[0111] The question desensitization module desensitizes the user's original question:

[0112] Step 21: Deploy a small parameter model locally and receive the user's original question through the small parameter model.

[0113] Step 22: Use a small parameter model to identify private words in the user's original question.

[0114] Step 23: Obtain the same or semantically similar field names from the metadata vector storage based on the privacy words, then retrieve the desensitized field names corresponding to the field names from the metadata desensitization library, generate a desensitized user question and desensitized word correspondence table, and send it to the desensitized word cache together with the privacy words.

[0115] Step 24: Send the desensitized user questions to the big model cloud service to generate the corresponding SQL.

[0116] The question restoration module restores the privacy of desensitized user questions:

[0117] Step 31: Receive the desensitized word correspondence table through the desensitized word cache,

[0118] Step 32: Use the small parameter model to receive the SQL generated by the large model cloud service.

[0119] Step 33: Restore the user question after desensitization based on the desensitized word correspondence table in the desensitized word cache and the SQL generated by the large model cloud service.

[0120] Step 34: Generate SQL based on the restored user question and send it to the GBI system. After the GBI system executes it, it returns the data to the user.

[0121] As the information interaction and execution process between the modules of the above-mentioned system are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0122] Similarly, the system of the present invention provides an intermediate layer to perform targeted replacement and restoration of privacy data between GBI and the large language model, thereby replacing sensitive data out of the enterprise domain and protecting data privacy.

[0123] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0124] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A privacy data protection method for a big data analysis process, characterized by: include: Step 1: Metadata desensitization processing: Step 11: Generate the corresponding desensitized words based on the metadata information in the metadata vector storage of the GBI system. Step 12: Create a corresponding table between desensitized words and metadata and store it in the metadata desensitization library. Step 2: Desensitize the user's original question: Step 21: Deploy a small parameter model locally and receive the user's original question through the small parameter model. Step 22: Use a small parameter model to identify private words in the user's original question. Step 23: Obtain the same or semantically similar field names from the metadata vector storage based on the privacy words, then retrieve the desensitized field names corresponding to the field names from the metadata desensitization library, generate a desensitized user question and desensitized word correspondence table and send it to the desensitized word cache. Step 24: Send the desensitized user questions to the big model cloud service to generate the corresponding SQL. Step 3: Restore privacy of desensitized user questions: Step 31: Receive the desensitized word correspondence table through the desensitized word cache, Step 32: Use the small parameter model to receive the SQL generated by the large model cloud service. Step 33: Restore the user question after desensitization based on the desensitized word correspondence table in the desensitized word cache and the SQL generated by the large model cloud service. Step 34: Generate SQL based on the restored user question and send it to the GBI system. After the GBI system executes it, it returns the data to the user.

2. According to the method for protecting privacy data in a big data analysis process according to claim 1, it is characterized by: In step 12, a correspondence table between desensitized words and metadata is established, including: Preset the data structure of the corresponding table, the data structure includes field meaning, field name, data type and enumeration value; In step 12, an open source relational database is used as the metadata desensitization library.

3. According to the method for protecting privacy data in a big data analysis process as described in claim 1, it is characterized by: The step 12 of establishing a correspondence table between desensitized words and metadata also includes: Establish the generation rules of the correspondence table: Convert English field names to Chinese field names; For industry-sensitive information, new names are generated based on organizational prompts.

4. The method for protecting privacy data in a big data analysis process according to claim 1 is characterized in that Step 23 generates a desensitized word correspondence table, including: organizing the desensitized word correspondence table into a json format.

5. A privacy data protection system for a big data analysis process, characterized by: It includes metadata desensitization module, question desensitization module and question restoration module. The metadata desensitization module performs metadata desensitization processing: Step 11: Generate the corresponding desensitized words based on the metadata information in the metadata vector storage of the GBI system. Step 12: Create a corresponding table between desensitized words and metadata and store it in the metadata desensitization library. The question desensitization module desensitizes the user's original question: Step 21: Deploy a small parameter model locally and receive the user's original question through the small parameter model. Step 22: Use a small parameter model to identify private words in the user's original question. Step 23: Obtain the same or semantically similar field names from the metadata vector storage based on the privacy words, then retrieve the desensitized field names corresponding to the field names from the metadata desensitization library, generate a desensitized user question and desensitized word correspondence table, and send it to the desensitized word cache together with the privacy words. Step 24: Send the desensitized user questions to the big model cloud service to generate the corresponding SQL. The question restoration module restores the privacy of desensitized user questions: Step 31: Receive the desensitized word correspondence table through the desensitized word cache, Step 32: Use the small parameter model to receive the SQL generated by the large model cloud service. Step 33: Restore the user question after desensitization based on the desensitized word correspondence table in the desensitized word cache and the SQL generated by the large model cloud service. Step 34: Generate SQL based on the restored user question and send it to the GBI system. After the GBI system executes it, it returns the data to the user.

6. A privacy data protection system for big data analysis process according to claim 5, characterized in that When the metadata desensitization module executes step 12 to establish a correspondence table between desensitized words and metadata, it includes: Preset the data structure of the corresponding table, the data structure includes field meaning, field name, data type and enumeration value; An open source relational database is used as the metadata desensitizing library.

7. The privacy data protection system for big data analysis process according to claim 5 is characterized by: When the metadata desensitization module executes step 12 to establish a correspondence table between desensitized words and metadata, it also includes: Establish the generation rule of the correspondence table: Convert English field names to Chinese field names; For industry-sensitive information, new names are generated based on organizational prompts.

8. According to claim 5, a privacy data protection system for big data analysis process is characterized in that the problem When the desensitizing module executes step 23 to generate a desensitized word correspondence table, it includes: organizing the desensitized word correspondence table into a json format.

Images