Data processing method and device, equipment, storage medium and product
By identifying and perturbing sensitive data on search text uploaded by users, efficient data privacy protection for large-scale model forgetting is achieved, avoiding the problems of high computing costs and degradation of model performance in the prior art.
Patent Information
- Application Number
- CN202411648698.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-02-25
AI Technical Summary
Existing large-model forgetting schemes require updating model parameters and retraining the model, resulting in high computational costs and degradation of model performance.
The preset quality evaluation model uses the user's sensitive data to identify the search text uploaded by the user. If sensitive data exists, the target perturbation function is used to perturb the identified search text to obtain the damaged search text, thereby achieving forgetting without updating the model Parameters or retrain the model.
It avoids the high computational cost and degradation of model performance caused by updating model parameters and retraining the model, and achieves efficient data privacy protection.
Smart Images
Figure CN119513097A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment, storage medium and product. Background Art
[0002] In recent years, the status of data elements as a national basic strategic resource has been continuously highlighted. Data elements have become the core engine, key driving force and core resource for the deepening development of the digital economy and the construction of Digital China. Data governance is a data management concept involving the entire organization. Through data governance, the ability to ensure high data quality throughout the entire life cycle of data is also a support for business goals. The key focus areas of data governance include availability, consistency, data integrity and data security, as well as the establishment of processes to ensure effective data management throughout the enterprise and ultimately improve data quality. However, due to differences in infrastructure, R&D capabilities, standards, etc. among different organizations and departments, the quality of data produced varies greatly, and less high-quality data is produced. How to intelligently, efficiently and accurately detect and optimize the quality of big data, and ensure that privacy data is not leaked during data query and evaluation, is a major problem.
[0003] In related technologies, data privacy is protected through machine unlearning technology. However, existing large model unlearning solutions usually require updating model parameters and retraining the model, which leads to high computational costs and degraded model performance. Summary of the invention
[0004] The main purpose of this application is to provide a data processing method, device, equipment, storage medium and product, which aims to solve the technical problems of existing large model forgetting solutions, which usually require updating model parameters and retraining models, resulting in high computing costs and reduced model performance.
[0005] To achieve the above purpose, the present application proposes a data processing method, which includes:
[0006] Get the search text uploaded by the user;
[0007] The search text is subjected to sensitive data identification through a preset quality assessment model. If the sensitive data is identified, the identified search text is perturbed through a target perturbation function to obtain a damaged search text, wherein the damaged search text needs to be forgotten.
[0008] In one embodiment, the step of identifying sensitive data of the search text by using a preset quality assessment model, and if the sensitive data is identified, perturbing the identified search text by using a target perturbation function to obtain the damaged search text comprises:
[0009] Dividing the search text containing the sensitive data into at least one information field;
[0010] Converting the information field into a corresponding embedding vector;
[0011] A preset number of the embedding vectors are selected from the embedding vectors, and the preset number of the embedding vectors are perturbed by a target perturbation function to obtain the damaged search text.
[0012] In one embodiment, the step of obtaining the search text uploaded by the user includes:
[0013] Obtaining an initial classification sub-model and a sample text for training the initial classification sub-model, wherein the initial classification sub-model is based on an initial classifier and an initial perturbation function, and initial parameters of the initial perturbation function are randomly selected;
[0014] Using the data types corresponding to the sample texts as sample labels for each of the sample texts, wherein the data types include sensitive data types and non-sensitive data types;
[0015] Inputting the sample text and the sample label into the initial classifier for training to obtain a preset classifier that meets the accuracy requirement;
[0016] Calculating the vector distance between the module corresponding function composed of the preset classifier and the initial perturbation function and the corresponding function of the initial classifier, and obtaining the target perturbation function that meets the distance requirement based on the vector distance;
[0017] Based on the preset classifier and the target disturbance function, a classification sub-model is determined.
[0018] In one embodiment, after the step of obtaining the search text uploaded by the user, the method further includes:
[0019] If the classification sub-model detects that the sensitive data does not exist in the search text, then based on the search text, a target library table for quality assessment is determined;
[0020] Extracting features from the search text using a preset quality assessment model to obtain target features;
[0021] Based on the target feature, determining a first rule for quality assessment, wherein the first rule includes a meta-information quality rule and a volatility detection rule;
[0022] Obtaining a second rule in the search text;
[0023] Based on the first rule and the second rule, a quality assessment is performed on the target library table to obtain a quality assessment result.
[0024] In one embodiment, the step of extracting features from the search text using a preset quality assessment model to obtain target features includes:
[0025] Based on the field type of the library table in the data warehouse, determine the field detection rule, and establish a mapping relationship between the meta information of the field in the library table and the field detection rule;
[0026] Acquire an initial quality assessment sub-model, and input the mapping relationship into the initial quality assessment sub-model for rule learning to obtain the learned initial quality assessment sub-model;
[0027] Obtaining sample library tables of different layers extracted from preset layers based on the data warehouse, and quality health scores corresponding to the sample library tables;
[0028] Using the quality and health score as the sample label of the corresponding sample library table;
[0029] Inputting the sample library table and the sample labels into the learned initial quality assessment sub-model to obtain an initial quality assessment result;
[0030] Inputting the initial quality assessment result and the sample label into a preset reward model to obtain a reward score, and optimizing the initial quality assessment sub-model based on the reward score to obtain a target quality assessment sub-model that meets the accuracy requirements;
[0031] Based on the target quality assessment sub-model and the classification sub-model, a preset quality assessment model is determined.
[0032] In one embodiment, the step of performing quality assessment on the target library table based on the first rule and the second rule to obtain a quality assessment result includes:
[0033] Based on the quality assessment result, determining the problem type of the problematic data in the library table;
[0034] Determining relevant factors associated with the problem type through an association rule algorithm;
[0035] Generate a visualization chart based on the problem type and the related factors;
[0036] Analyze the data lineage of the problematic data to determine the problematic link;
[0037] Based on the problem link and the visual chart, a repair suggestion is obtained, and based on the repair suggestion, the problem library table corresponding to the problem data is repaired.
[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:
[0039] The acquisition module is used to obtain the search text uploaded by the user;
[0040] The perturbation module is used to identify sensitive data of the search text through a preset quality assessment model. If the sensitive data is identified, the identified search text is perturbed through a target perturbation function to obtain a damaged search text, wherein the damaged search text needs to be forgotten.
[0041] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.
[0042] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.
[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.
[0044] One or more technical solutions proposed in this application have at least the following technical effects:
[0045] Compared with the related art, in which data privacy is protected by machine unlearning technology, the existing large-model forgetting scheme usually requires updating model parameters and retraining the model, which will result in high computational cost and decreased model performance. The present application obtains the search text uploaded by the user; through a preset quality assessment model, the search text is identified as sensitive data; if the sensitive data is identified, the identified search text is perturbed by a target perturbation function to obtain the damaged search text, wherein the damaged search text needs to be forgotten. It can be understood that the present application identifies sensitive data through a classification sub-model; if sensitive data exists, the search text that needs to be forgotten is perturbed by a target perturbation function to obtain the damaged search text after perturbation; there is no need to update model parameters or retrain the model, and the problem of high computational cost and decreased model performance caused by updating model parameters and retraining the model can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0048] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;
[0049] Figure 2 This is the overall architecture diagram of the data processing method of this application;
[0050] Figure 3 This is a flowchart of the model forgetting operation of the data processing method of this application;
[0051] Figure 4 This is a diagram of the model optimization mechanism of the data processing method of this application;
[0052] Figure 5 A flowchart of the second embodiment of the data processing method of this application is provided;
[0053] Figure 6 A quality assessment flow chart of the data processing method for this application;
[0054] Figure 7 This is a model operation architecture diagram of the data processing method of this application;
[0055] Figure 8 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;
[0056] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.
[0057] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0058] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0059] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0060] The main solution of the embodiment of the present application is: to obtain the search text uploaded by the user; to identify sensitive data of the search text through a preset quality assessment model, and if the sensitive data is identified, to perturb the identified search text through a target perturbation function to obtain the damaged search text, wherein the damaged search text needs to be forgotten.
[0061] In related technologies, data privacy is protected through machine unlearning technology. However, existing large model unlearning solutions usually require updating model parameters and retraining the model, which leads to high computational costs and degraded model performance.
[0062] This application identifies sensitive data through a classification sub-model. If sensitive data exists, the search text that needs to be forgotten is perturbed through a target perturbation function to obtain the damaged search text after the perturbation. There is no need to update the model parameters or retrain the model, which can avoid the problem of high computational cost and degraded model performance due to updating the model parameters and retraining the model.
[0063] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes a data processing device as an example to illustrate this embodiment and the following embodiments.
[0064] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the data processing method of the present application.
[0065] In this embodiment, refer to Figure 2 , Figure 2 The data processing method includes steps S100 to S200:
[0066] Step S100, obtaining the search text uploaded by the user;
[0067] It should be noted that the data processing device is provided with a search page. The data processing device obtains the search text uploaded by the user on the search page.
[0068] Step S200, using a preset quality assessment model to identify sensitive data in the search text, if the sensitive data is identified, perturbing the identified search text through a target perturbation function to obtain a damaged search text, wherein the damaged search text needs to be forgotten.
[0069] It is understandable that the data processing device is also provided with a component (Text2SQL). After obtaining the search text, the data processing device will convert the natural language search text input into an SQL query statement (prompt) through Text2SQL. The Text2SQL component allows users to ask questions in simple language through natural language processing (NLP) technology, and then automatically converts them into SQL commands that the database can understand and execute, which greatly reduces the threshold for ordinary users to access complex data warehouses.
[0070] It should be noted that sensitive data is data including private information such as salary information. The target perturbation function is used to damage the user's prompt, thereby forgetting the prompt containing sensitive data. If the data processing device recognizes that there is sensitive data in the prompt, the prompt is perturbed by the target perturbation function used to damage the user's prompt, thereby forgetting the prompt containing sensitive data, to obtain a damaged prompt after the perturbation.
[0071] Furthermore, after detecting sensitive data, the classification sub-model of the data processing device will damage the prompt. After the damage, the preset quality assessment model will not analyze based on the damaged prompt to obtain the quality analysis result, thereby ensuring data security.
[0072] In a feasible implementation, the search text is subjected to sensitive data identification through a preset quality assessment model. If the sensitive data is identified, the identified search text is perturbed through a target perturbation function to obtain the damaged search text, which includes the following steps:
[0073] Dividing the search text containing the sensitive data into at least one information field;
[0074] It is understandable that the information field can be a word, a phrase or a sentence, depending on the requirements of the application scenario. The data processing device divides the search text of the sensitive data type into a number of information fields (tokens).
[0075] Specifically, x is the original input prompt, which can be represented as a sequence of multiple tokens: x = {x1, x2, ..., x T}, where T is the number of tokens, indicating a prompt text with T tokens.
[0076] Converting the information field into a corresponding embedding vector;
[0077] It should be noted that the data processing device will not modify x in the token space, but will damage it in the embedding space, so the information field in the token space is converted into an embedding vector in the embedding space.
[0078] Specifically, the information field is embedded in the function E: X→R d , each tokenx i Mapping to the d-dimensional real vector space R d In the above example, we get each tokenx i The corresponding embedding vector e i , e i ∈R d , that is, e i =E(x i ), where the embedding vector e can be expressed as e=E(x)={e1,e2,...,e T}, each e i is x i Representation in embedding space.
[0079] A preset number of the embedding vectors are selected from the embedding vectors, and the preset number of the embedding vectors are perturbed by a target perturbation function to obtain the damaged search text.
[0080] It can be understood that the initial perturbation function is the damage function Corrupt: εxS→ε, where ε represents the vector space where the embedding vector is located, σ∈S, This function accepts an embedding vector e and a damage intensity parameter σ and returns a damaged embedding vector, where the damage intensity parameter σ represents the strength of the damage and controls the degree of damage. The data processing device randomly selects a preset number of embedding vectors and sets their values to 0 or other random values. Through the target perturbation function and the damage intensity parameter σ, the preset number of embedding vectors are modified to achieve the purpose of destroying the prompt and forgetting the relevant information. Please refer to Figure 3 , Figure 3 Run the flowchart for the model forgetting. The resulting embedding corruption prompt is represented as:
[0081]
[0082] In a feasible implementation manner, the following steps are included before step S100:
[0083] Obtaining an initial classification sub-model and a sample text for training the initial classification sub-model, wherein the initial classification sub-model is based on an initial classifier and an initial perturbation function, and initial parameters of the initial perturbation function are randomly selected;
[0084] It is understandable that the initial classifier is the prompt classifier. The sample text is a data set containing private information and non-private information. The data processing device obtains the prompt classifier and the initial classification sub-model composed of the initial perturbation function Corrupt whose parameters are randomly selected and the data set containing private information and non-private information.
[0085] Using the data types corresponding to the sample texts as sample labels for each of the sample texts, wherein the data types include sensitive data types and non-sensitive data types;
[0086] It should be noted that the data processing device marks whether each text in the data set includes private data to obtain a sample label.
[0087] Inputting the sample text and the sample label into the initial classifier for training to obtain a preset classifier that meets the accuracy requirement;
[0088] It is understandable that the preset classifier can accurately distinguish between private information and non-private information. If the text is determined to contain private information, it will enter the forgetting process. The data processing device inputs the sample text and sample label into the initial classifier for training to obtain the preset classifier that can accurately distinguish between private information and non-private information.
[0089] Calculating the vector distance between the module corresponding function composed of the preset classifier and the initial perturbation function and the corresponding function of the initial classifier, and obtaining the target perturbation function that meets the distance requirement based on the vector distance;
[0090] It should be noted that in order to make the forgotten learning model behave like the retained model (i.e., the original data model without adding privacy filtering), the data processing device needs to define a distance function, which quantifies the gap between the two models for all m∈M, where M can include different configurations of the forgotten learning model and different configurations of the retained model, and m is a preset classification submodel. In order to minimize the distance function, an alternative metric value is used. To approximately preserve the performance of the model on M. A common distance metric is defined below:
[0091]
[0092] Among them, θ o are the parameters of the retained model, is a function that takes the embedding vector e and the parameters θ of the retention model o , and returns an intermediate result, m i is another function that accepts The result is returned as a metric value, which can be a quantification of the performance of the retained model on the input e, where M represents the number of data points in the dataset.
[0093] It is understandable that the data processing device calculates the optimal damage intensity parameter σ by the zero-order optimization method so that the gap index between the forgotten learning model and the retained model is minimized. The goal of the forgotten learning is to minimize the following formula:
[0094]
[0095] Specifically, the data processing device sets the step size ηη and the smoothing parameter ∈ (also called the perturbation size), and calculates the zero-order gradient approximation through the randomly selected damage intensity parameter σ, and updates the damage intensity parameter σ according to the calculated zero-order gradient approximation, with the purpose of minimizing the distance d, and queries its final index gap during the optimization, and continuously iterates and calculates according to the index gap to obtain the final parameter σ as the parameter of the damage function Corrupt, please refer to Figure 4 , Figure 4 A model optimization mechanism diagram is provided. Through the above steps, specific information can be forgotten without retraining the large language model, while keeping other performance of the model unaffected.
[0096] Based on the preset classifier and the target disturbance function, a classification sub-model is determined.
[0097] It should be noted that the data processing device finally determines the classification sub-model according to the trained preset classifier and target perturbation function.
[0098] In this embodiment, the data processing device perturbs the input prompt by adding a perturbation function in the classification sub-model. There is no need to retrain the model to achieve forgetting, and other performance of the model is not affected, thereby saving a lot of computing resources and time.
[0099] The present application embodiment provides a data processing method, referring to Figure 5 , Figure 5 This is a flow chart of the second embodiment of the data processing method of the present application.
[0100] In this embodiment, refer to Figure 6 , Figure 6 A quality assessment flow chart is provided, wherein the data processing method comprises steps A100 to A500:
[0101] Step A100, if the classification sub-model detects that the sensitive data does not exist in the search text, then determine a target library table for quality assessment based on the search text;
[0102] It should be noted that the target library table is the library table stored in the data warehouse within the scope of quality assessment described by the user in the search text. The classification sub-model of the data processing device will identify the sensitive data in the search text. If there is no sensitive data, no disturbance will be generated. The library table stored in the data warehouse within the scope of quality assessment described in the search text will be directly determined through the preset quality assessment model.
[0103] Furthermore, the data processing device is also provided with a data synchronization module, which is responsible for synchronizing the data to be tested outside the system, such as data in databases or data warehouses such as MySQL, Oracle and Hive, into this system and performing formatting operations to prepare for subsequent quality assessment.
[0104] Step A200, extracting features from the search text using a preset quality assessment model to obtain target features;
[0105] It is understandable that the preset quality assessment model is based on the large language model (LLM), which responds to input questions in the ReAct (Reasoning and Acting) way. ReAct means that the model should not only think (reasoning) but also take actions (acting) when dealing with problems. The model is not limited to passively answering questions, but can also actively perform some operations, and provide external tools such as data synchronization, metadata management, data calculation and Text2SQL. The target features include table names, field names, and field types.
[0106] It should be noted that the preset quality assessment model of the data processing device can intelligently identify the meaning of the field based on features such as the table name, field name and field type, and extract relevant features in the search text to search for target features in the text.
[0107] Step A300, determining a first rule for quality assessment based on the target feature, wherein the first rule includes a meta-information quality rule and a volatility detection rule;
[0108] It should be noted that the first rule is a basic quality rule, such as non-empty detection of primary key fields, validity detection of gender fields, standardization detection of mobile phone numbers / ID numbers / license plate numbers, and valid range detection of numeric fields.
[0109] In particular, compared with conventional quality rules, the basic quality rules of data processing equipment additionally add meta-information quality and volatility detection to comprehensively evaluate data quality, and the evaluation results are more comprehensive;
[0110] Meta-information quality detection includes the following rules: table names must comply with the rules. Tables in data warehouses must comply with the specifications. Different organizations can have their own naming specifications, which are usually segmented naming in the form of {hierarchical identifier}_business domain_subject domain_table meaning_data granularity, such as dwd_cdn_download_bandwi dth_di; tables must have corresponding Chinese names or descriptions; field names must comply with the rules and be uniformly separated by underscores or camel case. Special characters are not allowed. Fields must have Chinese names and corresponding descriptions.
[0111] Volatility detection includes the following rules: year-on-year / month-on-month data volume fluctuations compared to yesterday / a week ago / a month ago cannot exceed the threshold. In addition to data volume, you can also customize indicators based on specific fields; the average fluctuation within a week / month cannot exceed the threshold.
[0112] It is understandable that after extracting features such as table name, field name and field type from the search text, the data processing device recommends basic quality rules that match the features, including volatility detection rules and meta-information quality detection rules.
[0113] Step A400, obtaining a second rule in the search text;
[0114] It should be noted that the second rule is specific to some business areas, for example:
[0115] phone_number is the mobile phone number, which must conform to the Chinese mobile phone number format;
[0116] latitude_logitude is the longitude and latitude information, which needs to conform to a specific location range;
[0117] The value of the data processing field is in the range of [aa, bb, cc, dd];
[0118] The department_id of the employee table corresponds one-to-one with the id of the department table;
[0119] The amount of consumption data collected daily in Nanjing is more than 100,000;
[0120] The daily data volume fluctuation of the data processing table partition shall not exceed 10%;
[0121] Table naming rule: {ods / dwd / dws / ads}_cdn_dowload_{entity}_{1d / di / df}.
[0122] It is understandable that when the data processing device is reasoning through the preset quality assessment model, it adds examples to the problem through the few-shot prompt method, tells the model some business field-specific rules, the latest standard examples of data quality, increases the accuracy of the model, and is more in line with expert experience, such as:
[0123] Task: Answer user questions about data quality
[0124] Example 1:
[0125] Question: What is the data quality of the testdb1.consumer table?
[0126] Constraint rules: The name and payment fields of the table cannot be empty, the timestamp format is "2024-07-0510:00:00", and the sum of aoumnt needs to be consistent with the sum of paymeng. Answer: Among the 12 fields in the testdb1.consumer table, the name and payment fields have empty values, the timestamp field has 100 format problems, sum(aoumount) and sum(payment) are inconsistent, the data fluctuation compared to the previous day exceeds 50%, and the partition data with a life cycle of 1 year but accessed less than 100 times 30 days ago. In summary, the overall quality health score of the table is 70 points.
[0127] Furthermore, when performing quality assessment, the data processing device generally uses rules stored in a preset quality assessment model without further limitation by the user.
[0128] Step A500: Based on the first rule and the second rule, a quality assessment is performed on the target library table to obtain a quality assessment result.
[0129] It is understandable that the quality assessment result is a quality score and a quality assessment report, or any one of them, and the corresponding result is output according to the user's needs. The preset quality assessment model of the data processing device analyzes and calculates the retrieved library table for quality assessment according to the quality assessment rules to obtain the quality assessment result.
[0130] For details, please refer to Figure 7 , Figure 7 A model operation architecture diagram is provided. The preset quality assessment model of the data processing device receives the search text uploaded by the user on the search interface: "I want to know the quality of the financial system data?" After receiving the search text uploaded by the user, the preset quality assessment model will take action (Action) to gradually call the tool calculation and return the quality assessment result. Since the user's demand is only about the quality of the financial system data, the preset quality assessment model only returns the quality score.
[0131] In a feasible implementation manner, the following steps are included before step A100:
[0132] Based on the field type of the library table in the data warehouse, determine the field detection rule, and establish a mapping relationship between the meta information of the field in the library table and the field detection rule;
[0133] It should be noted that the field detection rules are common field detection rules in the data warehouse. The data processing device extracts the common field detection rules in the data warehouse according to the number of field types in the database table in the data warehouse, and establishes a mapping relationship between the field meta-information and the field detection rules.
[0134] For example, there is a field called customer_age, and its meta information is as follows:
[0135] Field name: customer_age;
[0136] Field type: INT;
[0137] Field description: "Age of customer";
[0138] Field enumeration value: none;
[0139] You can define the following detection rules for it:
[0140] Null value check: Make sure customer_age is not null.
[0141] Value range check: Make sure customer_age is in the range [1,120].
[0142] Acquire an initial quality assessment sub-model, and input the mapping relationship into the initial quality assessment sub-model for rule learning to obtain the learned initial quality assessment sub-model;
[0143] It is understandable that the initial quality assessment sub-model is an untrained large language model. The data processing device obtains an untrained large language model, inputs the mapping relationship between the field meta-information and the field detection rule into the untrained large language model for rule learning, and the model can learn how to judge the quality of the data according to the specific situation of the field.
[0144] Obtaining sample library tables of different layers extracted from preset layers based on the data warehouse, and quality health scores corresponding to the sample library tables;
[0145] It should be noted that the preset layers include the detail layer (DWD), summary layer (DWS) and application layer (ADS), while the original data layer (ODS) is generally not suitable as a selected data set because it mainly contains unprocessed raw data. The sample library table is a high-quality data set that is used to adjust model parameters according to known quality standards so that it can identify and evaluate data quality. The data processing equipment selects libraries and tables of different quality levels from the detail layer (DWD), summary layer (DWS) and application layer (ADS) of the data warehouse to construct a selected data set. The data set is based on libraries and tables of different qualities manually evaluated by historical expert experience and their quality health scores. These library table data evaluated and scored by experts contain rich quality information and features, which can provide valuable materials and basis for subsequent model training and optimization.
[0146] Using the quality and health score as the sample label of the corresponding sample library table;
[0147] It is understandable that the data processing device uses the quality and health score corresponding to the sample library table as the sample label for model training.
[0148] Inputting the sample library table and the sample labels into the learned initial quality assessment sub-model to obtain an initial quality assessment result;
[0149] It should be noted that the data processing device inputs the sample library table and sample labels into the learned initial quality assessment sub-model for quality assessment to obtain an initial quality assessment result obtained by the model assessment.
[0150] Inputting the initial quality assessment result and the sample label into a preset reward model to obtain a reward score, and optimizing the initial quality assessment sub-model based on the reward score to obtain a target quality assessment sub-model that meets the accuracy requirements;
[0151] It is understandable that the preset reward model RW (Reward Modeling) learns how to evaluate the output quality of the model based on the scoring data provided by humans. The data processing device uses the reward model to calculate the difference between the result of each model evaluation and the true score as a reward score, and adopts the PPO (Proximal Policy Optimization) algorithm to update the parameters of the model, where the PPO algorithm aims to minimize the uncertainty of the interaction between the policy network and the environment, while maintaining the rationality of the update step size to prevent overfitting, and then iterates the training according to the reward score, and each iteration adjusts the SFT model according to the new reward signal until the performance index is reached.
[0152] Specifically, refer to Figure 7The data processing equipment uses SFT technology (Supervised Fine-Tuning) to supervise the fine-tuning of the GPT pre-training model. After completing the initial fine-tuning of the SFT model, it further carries out the human feedback reinforcement learning RLHF (Reinforcement Learning from Human Feedback) process.
[0153] Furthermore, the preset reward model is trained based on a data set containing multiple quality scores of the same data table by humans. The reward model will learn the standards and logic of human quality scoring for the table, so that it can evaluate the results output by the SFT model and give reward feedback. Then, the PPO (Proximal Policy Optimization) algorithm is used to further fine-tune the SFT model based on the reward feedback of the RW model.
[0154] Based on the target quality assessment sub-model and the classification sub-model, a preset quality assessment model is determined.
[0155] It should be noted that the data processing device determines the preset quality assessment model based on the trained target quality assessment sub-model and classification sub-model.
[0156] In a feasible implementation manner, the following steps are included after step A500:
[0157] Based on the quality assessment result, determining the problem data with problems in the library table, and classifying and aggregating the problem data to obtain the problem data of different problem types after classification and aggregating;
[0158] It is understood that the problematic data is the data showing quality problems in the evaluation results. The data processing equipment uses statistical analysis methods to classify and summarize the data showing quality problems in the evaluation results to find the part with the biggest problems.
[0159] Determining relevant factors associated with the problem type through an association rule algorithm;
[0160] It should be noted that the relevant factors represent the potential relationship between data quality problems and factors such as data sources and data processing procedures. The data processing equipment uses the association rule mining algorithm in data mining technology to explore the potential relationship between data quality problems and factors such as data sources and data processing procedures to determine the main factors that cause data quality problems.
[0161] Generate a visualization chart based on the problem type and the related factors;
[0162] It is understandable that visualization charts include bar charts, line charts, and heat maps, which make the problem presentation more intuitive and facilitate in-depth analysis of the causes and patterns of the problem. Data processing equipment uses visualization technology to present the distribution and related factors of data quality problems in the form of charts, such as bar charts, line charts, and heat maps, making the problem presentation more intuitive and facilitate in-depth analysis of the causes and patterns of the problem.
[0163] Analyze the data lineage of the problematic data to determine the problematic link;
[0164] It should be noted that data lineage records the link relationship of data generation, which is crucial for understanding the source, flow path and usage of data. Data processing equipment uses the meta-information management function in the data analysis module to analyze data lineage, trace the source of data with quality problems, and trace back to each upstream and downstream table for fields with data quality problems. Correlation analysis is performed to draw inferences from one example and find out all related problem points. Combined with data log records, check the detailed information such as data processing time, processing personnel, processing operations, and production tasks in each link, so as to accurately locate the specific links and nodes where data quality problems occur.
[0165] Based on the problem link and the visual chart, a repair suggestion is obtained, and based on the repair suggestion, the problem library table corresponding to the problem data is repaired.
[0166] It should be noted that the repair suggestions can be specific scripts, algorithms or operation guidelines. The data processing device provides repair suggestions for each specific problem point based on the problem links obtained by the visualization chart and lineage tracing, so that the problem database table corresponding to the problem data can be repaired according to the repair suggestions.
[0167] Furthermore, the data processing equipment is also provided with a quality monitoring mechanism to monitor the quality of the repaired data in real time to ensure that the data quality is always within a controllable range.
[0168] Furthermore, after repairing the problem, the data processing equipment will continue to observe changes in data quality and adjust the repair plan based on actual conditions.
[0169] In this embodiment, the data processing device considers more evaluation dimensions to comprehensively evaluate the data quality, including metadata dimensions and data fluctuation dimensions, which is more comprehensive; the data quality evaluation rules are intelligently learned through a large model, which has better scalability.
[0170] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0171] This application also provides a data processing device, please refer to Figure 8 , the data processing device comprises:
[0172] The acquisition module 10 is used to acquire the search text uploaded by the user;
[0173] The perturbation module 20 is used to identify sensitive data of the search text through a preset quality assessment model. If the sensitive data is identified, the identified search text is perturbed through a target perturbation function to obtain a damaged search text, wherein the damaged search text needs to be forgotten.
[0174] Optionally, the disturbance module includes:
[0175] A conversion submodule is used to divide the search text containing the sensitive data into at least one information field; convert the information field into a corresponding embedding vector; select a preset number of the embedding vectors from the embedding vectors, and perturb the preset number of the embedding vectors through a target perturbation function to obtain the damaged search text.
[0176] Optionally, the acquisition module includes:
[0177] A parameter optimization submodule is used to obtain an initial classification submodel and sample text for training the initial classification submodel, wherein the initial classification submodel is based on an initial classifier and an initial perturbation function, and the initial parameters of the initial perturbation function are randomly selected; the data types corresponding to the sample texts are used as sample labels for each of the sample texts, wherein the data types include sensitive data types and non-sensitive data types; the sample texts and the sample labels are input into the initial classifier for training to obtain a preset classifier that meets the accuracy requirements; the vector distance between the corresponding function of the module composed of the preset classifier and the initial perturbation function and the corresponding function of the initial classifier is calculated, and based on the vector distance, a target perturbation function that meets the distance requirements is obtained; and the classification submodel is determined based on the preset classifier and the target perturbation function.
[0178] The quality assessment submodule is used to extract features from the search text through a preset quality assessment model to obtain target features; based on the target features, determine a first rule for quality assessment, wherein the first rule includes a meta-information quality rule and a volatility detection rule; obtain a second rule in the search text; if the classification submodel detects that the sensitive data does not exist in the search text, determine a target library table for quality assessment based on the search text; based on the first rule and the second rule, perform quality assessment on the target library table to obtain a quality assessment result.
[0179] Optionally, the quality assessment submodule includes:
[0180] A rule learning unit is used to determine field detection rules based on the field types of library tables in the data warehouse, and establish a mapping relationship between the metadata of the fields in the library tables and the field detection rules; obtain an initial quality assessment sub-model, and input the mapping relationship into the initial quality assessment sub-model for rule learning to obtain the learned initial quality assessment sub-model; obtain sample library tables of different layers extracted from the preset layers of the data warehouse, and the quality health scores corresponding to the sample library tables; use the quality health scores as the sample labels of the corresponding sample library tables; input the sample library tables and the sample labels into the learned initial quality assessment sub-model to obtain initial quality assessment results; input the initial quality assessment results and the sample labels into a preset reward model to obtain reward scores, and optimize the initial quality assessment sub-model based on the reward scores to obtain a target quality assessment sub-model that meets the accuracy requirements; determine a preset quality assessment model based on the target quality assessment sub-model and the classification sub-model.
[0181] A repair unit is used to determine the problem type of the problem data with problems in the library table based on the quality assessment result; determine the relevant factors associated with the problem type through an association rule algorithm; generate a visualization chart based on the problem type and the relevant factors; analyze the data lineage of the problem data to determine the problem link; obtain repair suggestions based on the problem link and the visualization chart, and repair the problem library table corresponding to the problem data based on the repair suggestions.
[0182] The data processing device provided by the present application adopts the data processing method in the above embodiment to solve the technical problem of data processing. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be described in detail here.
[0183] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.
[0184] Reference below Fig. 9, which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, tablet computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0185] like Fig. 9 As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0186] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0187] The data processing device provided by the present application adopts the data processing method in the above embodiment to solve the technical problem of data processing. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0188] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0189] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0190] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.
[0191] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0192] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.
[0193] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data processing device, the data processing device: obtains the search text uploaded by the user; identifies sensitive data of the search text through a preset quality assessment model, and if the sensitive data is identified, perturbs the identified search text through a target perturbation function to obtain the damaged search text, wherein the damaged search text needs to be forgotten.
[0194] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0195] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0196] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0197] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problems of data processing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be repeated here.
[0198] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.
[0199] The computer program product provided by this application can solve the technical problem of data processing. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data processing method provided by the above embodiment, which will not be repeated here.
[0200] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A data processing method, characterized in that: The data processing method comprises: Get the search text uploaded by the user; The search text is subjected to sensitive data identification through a preset quality assessment model. If the sensitive data is identified, the identified search text is perturbed through a target perturbation function to obtain a damaged search text, wherein the damaged search text needs to be forgotten.
2. The data processing method according to claim 1, characterized in that: The step of identifying sensitive data in the search text by using a preset quality assessment model, and if the sensitive data is identified, perturbing the identified search text by using a target perturbation function to obtain the damaged search text comprises: Dividing the search text containing the sensitive data into at least one information field; Converting the information field into a corresponding embedding vector; A preset number of the embedding vectors are selected from the embedding vectors, and the preset number of the embedding vectors are perturbed by a target perturbation function to obtain the damaged search text.
3. The data processing method according to claim 1, characterized in that: The step of obtaining the search text uploaded by the user includes: Obtaining an initial classification sub-model and a sample text for training the initial classification sub-model, wherein the initial classification sub-model is based on an initial classifier and an initial perturbation function, and initial parameters of the initial perturbation function are randomly selected; Using the data types corresponding to the sample texts as sample labels for each of the sample texts, wherein the data types include sensitive data types and non-sensitive data types; Inputting the sample text and the sample label into the initial classifier for training to obtain a preset classifier that meets the accuracy requirement; Calculating the vector distance between the module corresponding function composed of the preset classifier and the initial perturbation function and the corresponding function of the initial classifier, and optimizing the initial parameters of the initial perturbation function based on the vector distance to obtain a target perturbation function that meets the distance requirement; Based on the preset classifier and the target disturbance function, a classification sub-model is determined.
4. The data processing method according to claim 3, characterized in that: After the step of obtaining the search text uploaded by the user, the following steps are further included: If the classification sub-model detects that the sensitive data does not exist in the search text, then based on the search text, a target library table for quality assessment is determined; Extracting features from the search text using a preset quality assessment model to obtain target features; Based on the target feature, determining a first rule for quality assessment, wherein the first rule includes a meta-information quality rule and a volatility detection rule; Obtaining a second rule in the search text; Based on the first rule and the second rule, a quality assessment is performed on the target library table to obtain a quality assessment result.
5. The data processing method according to claim 4, characterized in that: The step of extracting features from the search text using a preset quality assessment model to obtain target features includes: Based on the field type of the library table in the data warehouse, determine the field detection rule, and establish a mapping relationship between the meta information of the field in the library table and the field detection rule; Acquire an initial quality assessment sub-model, and input the mapping relationship into the initial quality assessment sub-model for rule learning to obtain the learned initial quality assessment sub-model; Obtaining sample library tables of different layers extracted from preset layers based on the data warehouse, and quality health scores corresponding to the sample library tables; Using the quality and health score as the sample label of the corresponding sample library table; Inputting the sample library table and the sample labels into the learned initial quality assessment sub-model to obtain an initial quality assessment result; Inputting the initial quality assessment result and the sample label into a preset reward model to obtain a reward score, and optimizing the initial quality assessment sub-model based on the reward score to obtain a target quality assessment sub-model that meets the accuracy requirements; Based on the target quality assessment sub-model and the classification sub-model, a preset quality assessment model is determined.
6. The data processing method according to claim 4, characterized in that: The step of performing quality assessment on the target library table based on the first rule and the second rule to obtain a quality assessment result comprises: Based on the quality assessment result, determining the problem type of the problematic data in the library table; Determining relevant factors associated with the problem type through an association rule algorithm; Generate a visualization chart based on the problem type and the related factors; Analyze the data lineage of the problematic data to determine the problematic link; Based on the problem link and the visual chart, a repair suggestion is obtained, and based on the repair suggestion, the problem library table corresponding to the problem data is repaired.
7. A data processing device, characterized in that: The device comprises: The acquisition module is used to obtain the search text uploaded by the user; The perturbation module is used to identify sensitive data of the search text through a preset quality assessment model. If the sensitive data is identified, the identified search text is perturbed through a target perturbation function to obtain a damaged search text, wherein the damaged search text needs to be forgotten.
8. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 6.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.