Data management method and device based on ChatGLM model and storage medium
Through the data governance method based on the ChatGLM model, the problem of low efficiency in traditional data governance methods when processing massive data is solved, efficient encoding and filling of multi-source data is achieved, risk data is generated and privacy protection is enhanced, and modern business is satisfied with the real-time Requirements for data analysis.
Patent Information
- Application Number
- CN202411049129.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-22
AI Technical Summary
Traditional data governance methods are inefficient in the face of massive data, unable to achieve rapid integration and unified view of data, difficult to meet the needs of modern businesses for real-time data analysis, and are insufficient to adapt to rapidly changing business needs and technical environments.
The data governance method based on the ChatGLM model is adopted to generate the governance data of the target enterprise by collecting multi-source data, matching data rules, self-coding processing, autoregressive filling and data desensitization, and improve the quality and availability of data.
It realizes efficient encoding and filling of multi-source data, identify and generate risk data, enhances data integrity and accuracy, and enhances privacy protection through data desensitization processing, improving the efficiency of data governance.
Smart Images

Figure CN119004397A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a data governance method, device and storage medium based on a ChatGLM model. Background Art
[0002] As information technology develops rapidly, the challenges of data growth faced by enterprises are becoming increasingly complex. Traditional data processing methods, which rely on relational databases and batch processing technologies, are unable to cope with massive amounts of data and are inefficient. These methods often fail to achieve rapid data integration and unified views, resulting in delayed decision-making and increased costs.
[0003] In addition, due to the lack of necessary real-time performance, traditional data governance methods are difficult to meet the needs of modern business for real-time data analysis. More importantly, these methods are obviously insufficient in adapting to rapidly changing business needs and technical environments. Therefore, seeking a new data governance method to improve data quality and availability has become a top priority for enterprises. Summary of the invention
[0004] The present invention provides a data governance method, device and storage medium based on the ChatGLM model, the main purpose of which is to solve the problem of low efficiency in data governance.
[0005] To achieve the above purpose, the present invention provides a data governance method based on the ChatGLM model, comprising:
[0006] Collect multi-source data of target enterprises;
[0007] Performing data rule matching on the multi-source data, and determining that the successfully matched multi-source data is model data of a preset ChatGLM model;
[0008] Using the preset ChatGLM model to perform self-encoding processing on the model data to obtain the encoded text of the model data;
[0009] Performing autoregressive filling on the coded text, and generating risk data in the multi-source data according to the coded text after the autoregressive filling;
[0010] The risk data is desensitized to obtain the governance data of the target enterprise.
[0011] Optionally, performing data rule matching on the multi-source data and determining that the successfully matched multi-source data is model data of a preset ChatGLM model includes:
[0012] Extracting a data structure of the multi-source data;
[0013] Performing outlier correction on the multi-source data, and calculating a matching degree between the multi-source data after outlier correction and the data structure;
[0014] When the matching degree is greater than a preset matching threshold, model data of the preset ChatGLM model is generated according to the multi-source data corresponding to the matching degree.
[0015] Optionally, generating model data of the preset ChatGLM model according to the multi-source data corresponding to the matching degree includes:
[0016] Determine the multi-source data corresponding to the matching degree as initial data, and perform text-to-numeric attribute conversion on the initial data;
[0017] Determine the value range of the initial data according to the data type in the data structure;
[0018] The initial data after the text-to-numeric attribute conversion is numerically scaled according to the numerical range to obtain model data of the preset ChatGLM model.
[0019] Optionally, the using the preset ChatGLM model to perform self-encoding processing on the model data to obtain the encoded text of the model data includes:
[0020] Performing vector transformation on the model data using the ChatGLM model to obtain a data vector of the model data;
[0021] The model data is data-encoded according to the data vector, and the encoded model data is used to generate an encoded text of the model data.
[0022] Optionally, the performing data encoding on the model data according to the data vector comprises:
[0023] Determine the row position and column position of the model data according to the data vector;
[0024] Performing sinusoidal coding on the row position to obtain a row code of the model data;
[0025] Performing cosine coding on the column positions to obtain column coding of the model data;
[0026] Generate a two-dimensional position coding matrix of the model data according to the row coding and the column coding;
[0027] Vector concatenation is performed on the two-dimensional position coding matrix and the data vector to obtain coding data of the model data.
[0028] Optionally, performing autoregressive padding on the encoded text includes:
[0029] Determining a text unit to be predicted of the encoded text and a text order of the text unit to be predicted;
[0030] Perform text prediction on the text units to be predicted in sequence according to the text sequence to obtain predicted text of the encoded text;
[0031] The predicted text is used to perform text filling on the encoded text to obtain the filled text of the encoded text.
[0032] Optionally, the performing text prediction on the to-be-predicted text units in sequence according to the text order to obtain predicted text of the encoded text includes:
[0033] S21, determining the first text unit to be predicted in the encoded text according to the text order;
[0034] S22, generating the first unit text of the first text unit to be predicted;
[0035] S23, determining the adjacent text unit to be predicted of the first text unit to be predicted according to the text order;
[0036] S24, generating adjacent unit texts of the adjacent text unit to be predicted according to the first unit text, and repeating steps S23-S24 until the last unit text of the last text unit to be predicted in the encoded text is generated;
[0037] S25, collecting all generated first unit texts, adjacent unit texts and last unit texts as predicted text of the encoded text.
[0038] Optionally, generating risk data in the multi-source data according to the encoded text after the autoregressive filling includes:
[0039] Extracting key features from the encoded text after the autoregressive padding;
[0040] Performing feature classification on the key features to obtain classification features of the key features;
[0041] generating a risk score for the classification feature according to the feature category corresponding to the classification feature;
[0042] A risk report of the key feature is determined according to the risk score and a preset risk threshold, and risk data in the multi-source data is determined according to the risk report.
[0043] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:
[0044] at least one processor; and,
[0045] a memory communicatively connected to the at least one processor; wherein,
[0046] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data governance method based on the ChatGLM model described above.
[0047] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned data governance method based on the ChatGLM model.
[0048] The embodiment of the present invention utilizes the ChatGLM model to automatically encode and fill multi-source data, thereby being able to efficiently identify and generate risk data. The autoregressive filling technology can effectively fill in the gaps or errors in the data and improve the integrity and accuracy of the data. In the process of generating risk data, sensitive information can be identified and desensitized to enhance data privacy protection. Therefore, the data governance method, device and storage medium based on the ChatGLM model proposed in the present invention can solve the problem of low efficiency in data governance. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 A flowchart of a data governance method based on the ChatGLM model provided in one embodiment of the present invention;
[0050] Figure 2 A schematic diagram of the structure of an electronic device for implementing the data governance method based on the ChatGLM model provided in one embodiment of the present invention.
[0051] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0053] The embodiment of the present application provides a data governance method based on the ChatGLM model. The execution subject of the data governance method based on the ChatGLM model includes but is not limited to at least one of the electronic devices such as the server, the terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the data governance method based on the ChatGLM model can be executed by software or hardware installed on the terminal device or the server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
[0054] Reference Figure 1 FIG. 1 is a flow chart of a data governance method based on the ChatGLM model provided in accordance with an embodiment of the present invention. In this embodiment, the data governance method based on the ChatGLM model includes:
[0055] S1. Collect multi-source data of the target enterprise.
[0056] In the embodiment of the present invention, the acquisition of multi-source data of the target enterprise refers to collecting data from multiple data sources inside and outside the enterprise through a data acquisition tool or platform.
[0057] Furthermore, these data sources may include relational databases (such as Oracle, DB2, Microsoft SQLServer, Microsoft Access, MySQL, etc.), non-relational databases (such as NoSQL, Cloudant, MongoDB, Redis, HBase, etc.), and unstructured document data (such as application logs, operating system logs, etc.).
[0058] In detail, through data acquisition, enterprises can obtain data from different systems, applications and data stores, and unify this data into a centralized or distributed data store for further analysis, reporting and decision support.
[0059] In detail, by collecting multi-source data from target companies, companies can more comprehensively and accurately understand their business conditions, customer needs, market trends and other information, providing stronger support for business decisions. This comprehensive data collection is crucial for data-driven decision-making and business development of enterprises.
[0060] S2. Perform data rule matching on the multi-source data, and determine that the successfully matched multi-source data is model data of a preset ChatGLM model.
[0061] In an embodiment of the present invention, performing data rule matching on the multi-source data and determining that the successfully matched multi-source data is model data of a preset ChatGLM model includes:
[0062] Extracting a data structure of the multi-source data;
[0063] Performing outlier correction on the multi-source data, and calculating a matching degree between the multi-source data after outlier correction and the data structure;
[0064] When the matching degree is greater than a preset matching threshold, model data of the preset ChatGLM model is generated according to the multi-source data corresponding to the matching degree.
[0065] In detail, the data structure includes: field name, data type, length, and constraints of the multi-source data.
[0066] In detail, extracting the data structure of the multi-source data refers to the process of obtaining the data structure from the multi-source data, which means identifying and recording the various components of the data, such as fields, data types, lengths, constraints, etc. For example: for a set of sales data, the extracted data structure may include information such as sales date, product number, sales quantity, sales amount, etc.
[0067] In detail, the outlier correction of the multi-source data refers to identifying and correcting outliers or outliers in the data. Outliers may be caused by data entry errors or other reasons. For example, if sales suddenly become abnormally high in a certain month, it may be caused by data entry errors and needs to be corrected.
[0068] In detail, the Euclidean distance algorithm can be used to evaluate the matching degree between the corrected multi-source data and the data structure.
[0069] In detail, the calculation of the matching degree between the multi-source data after the outlier correction and the data structure refers to determining the similarity or consistency between the multi-source data and the preset data structure. In this step, each component of the data needs to match the expected data structure. For example, if the date field in the sales data is inconsistent with the expected date format, further processing may be required to ensure the match.
[0070] In detail, the preset matching threshold refers to a threshold set during the matching process, which is used to determine whether the degree of matching between the multi-source data and the expected data structure meets the requirements. If the matching degree exceeds the preset threshold, the model data of the ChatGLM model can be generated based on the multi-source data.
[0071] In general, the process of extracting data structure from multi-source data, correcting and matching data, and preliminarily determining which data can be used as input data for the ChatGLM model based on preset conditions can help improve the quality and availability of data, thereby supporting data-driven decision-making and business development.
[0072] In detail, the generating of the model data of the preset ChatGLM model according to the multi-source data corresponding to the matching degree includes:
[0073] Determine the multi-source data corresponding to the matching degree as initial data, and perform text-to-numeric attribute conversion on the initial data;
[0074] Determine the value range of the initial data according to the data type in the data structure;
[0075] The initial data after the text-to-numeric attribute conversion is numerically scaled according to the numerical range to obtain model data of the preset ChatGLM model.
[0076] In detail, first, the multi-source data corresponding to the matching degree is determined as the initial data. Then, the initial data is subjected to text-to-numeric attribute conversion, which means converting the attribute value of the text type into a numeric type so that the computer can better understand and process the data, for example, converting text descriptions such as "high", "medium", and "low" into numeric types 1, 2, and 3.
[0077] Example: Suppose there is a product sales dataset in which the sales levels of the products are described in text, such as "best-selling", "average", and "slow-selling". These text descriptions need to be converted into numerical types, such as 1 for "best-selling", 2 for "average", and 3 for "slow-selling".
[0078] Furthermore, text-to-numeric attribute conversion can convert non-numeric data into numerical data, so that these data can be processed and analyzed by mathematical models, thereby better performing data mining and modeling analysis.
[0079] In detail, the numerical range is determined according to the data type, and determining the numerical range can help normalize and standardize the data for subsequent numerical scaling and model training. For example, determining the maximum and minimum values of numerical data.
[0080] Example: In sales data, the sales quantity may be a numeric attribute, and the minimum and maximum values of the sales quantity need to be determined.
[0081] In detail, determining the numerical range of the initial data according to the data type in the data structure refers to scaling the data after the text-to-numeric attribute conversion according to the numerical range, mapping the data to a specific range, usually between 0 and 1. Numerical scaling can avoid the impact of large differences in the value ranges of different attributes on model training and prediction, and can also speed up model convergence and improve model performance.
[0082] Example: Numerically scale sales quantities to map them to the range between 0 and 1.
[0083] In general, text-to-numeric attribute conversion can convert non-numeric data into numerical data. Determining the numerical range can help normalize the data, while numerical scaling helps eliminate the differences in value ranges between different attributes to ensure data quality and consistency and enhance the stability and generalization ability of the model.
[0084] S3. Use the preset ChatGLM model to perform self-encoding processing on the model data to obtain the encoded text of the model data.
[0085] In an embodiment of the present invention, the self-encoding process of the model data using the preset ChatGLM model to obtain the encoded text of the model data includes:
[0086] Performing vector transformation on the model data using the ChatGLM model to obtain a data vector of the model data;
[0087] The model data is data-encoded according to the data vector, and the encoded model data is used to generate an encoded text of the model data.
[0088] In detail, the ChatGLM model is a pre-trained model for generating natural language text, which has the ability to understand and generate natural language. The ChatGLM model is usually trained based on a large-scale corpus and can generate fluent and coherent text.
[0089] In detail, autoencoding is a technique that encodes input data into a more compact representation. Here, model data is encoded into a compact text representation. Autoencoding can reduce the dimension of the data and extract important features of the data, which helps to reduce the complexity of the data and better understand the meaning of the data.
[0090] In detail, using the ChatGLM model to vectorize the model data refers to the process of converting the model data into a vector representation so that the computer can better understand and process the data.
[0091] In general, it is necessary to use the pre-trained ChatGLM model to self-encode the model data, convert the model data into a compact text representation, and perform vector conversion and data encoding as needed. The implementation of these steps can help understand and process the model data and extract useful information and features from it, providing a more reliable data foundation for subsequent data analysis and modeling.
[0092] In detail, the data encoding the model data according to the data vector includes:
[0093] Determine the row position and column position of the model data according to the data vector;
[0094] Performing sinusoidal coding on the row position to obtain a row code of the model data;
[0095] Performing cosine coding on the column positions to obtain column coding of the model data;
[0096] Generate a two-dimensional position coding matrix of the model data according to the row coding and the column coding;
[0097] Vector concatenation is performed on the two-dimensional position coding matrix and the data vector to obtain coding data of the model data.
[0098] In detail, the row and column position of each data vector in the model data is determined. The model data is usually a matrix, and each data vector corresponds to an element in the matrix. Therefore, the row and column position of each data vector needs to be determined.
[0099] For example, for a 2x3 matrix, the first data vector may correspond to the elements in the first row and first column, the second data vector may correspond to the elements in the first row and second column, and so on.
[0100] In detail, the row position information is encoded into a sinusoidal coding vector using a sine function. Sine coding maps continuous position information into a fixed-length vector space through a sine function to capture the periodic relationship between positions.
[0101] For example: For row positions 1, 2, and 3, sinusoidal encoding is performed to obtain the corresponding vector representation.
[0102] In detail, the column position information is encoded into a cosine-encoded vector using a cosine function. Cosine encoding is a method that maps continuous position information into a fixed-length vector space to capture the relationship between positions.
[0103] For example, for column positions 1, 2, and 3, cosine encoding is performed to obtain the corresponding vector representation.
[0104] In detail, the two-dimensional position coding matrix is a matrix composed of row codes and column codes according to position information, wherein each position corresponds to a unique row code and column code.
[0105] In detail, vector concatenation is to connect two vectors together in a certain way to form a new vector, which can be operated using the vector concatenation function in the programming language.
[0106] In detail, by concatenating the position code with the original data vector, each data vector contains its corresponding position information, thereby being able to better represent the structure and relationship of the model data.
[0107] In detail, for the two-dimensional position encoding matrix, consider encoding the row and column position information of the text data separately to represent the position relationship of different words in the text, and merge the generated two-dimensional position encoding matrix with the word vector representation to obtain the word vector representation containing the position information.
[0108] In general, the data encoding process centered around two-dimensional position encoding can help the model better capture the positional relationship in text data and improve its ability to understand the text structure, thereby improving the model's performance in natural language processing tasks.
[0109] S4. Perform autoregressive filling on the coded text, and generate risk data in the multi-source data according to the coded text after the autoregressive filling.
[0110] In an embodiment of the present invention, the step of performing autoregressive padding on the encoded text includes:
[0111] Determining a text unit to be predicted of the encoded text and a text order of the text unit to be predicted;
[0112] Perform text prediction on the text units to be predicted in sequence according to the text sequence to obtain predicted text of the encoded text;
[0113] The predicted text is used to perform text filling on the encoded text to obtain the filled text of the encoded text.
[0114] In detail, autoregressive filling is a technique in time series analysis to fill missing data points. In time series data, missing values may occur, which may affect further analysis and prediction. Autoregressive filling fills the missing values by exploiting the autocorrelation within the time series data.
[0115] Specifically, autoregressive filling uses known data points to predict missing data points by inferring future observations based on past observations in the time series data.
[0116] In detail, determining the text units to be predicted and the text order refers to determining the text units that need to be autoregressively filled in the encoded text and determining their text order, wherein the text units to be predicted refer to the text fragments that need to be predicted based on the previously known text, and the text order refers to the order of the text units to be predicted in the encoded text.
[0117] For example, if the encoded text is an article, the text unit to be predicted may be a sentence in the article, and the text order is the order of the sentences in the article.
[0118] In detail, determining the text units to be predicted and the text order is the premise of autoregressive filling, which determines the text content and order that the model should focus on when filling.
[0119] In detail, performing text prediction on the text unit to be predicted refers to performing text prediction on the text units to be predicted in sequence according to the known encoded text to obtain the predicted text, wherein text prediction refers to the model predicting the content of the text unit to be predicted based on the previously known text content. Text prediction is the core step of autoregressive filling, which completes the filling by generating new text related to the known text, thereby maintaining the coherence and consistency of the text.
[0120] For example: If the text unit to be predicted is the beginning of a sentence, then the next word or phrase can be predicted based on the previous context.
[0121] In detail, using predicted text for text filling refers to using the generated predicted text and inserting it into the corresponding position in the encoded text to complete text filling, wherein text filling refers to embedding the generated predicted text into the original encoded text to fill the position of the text unit to be predicted, so that the entire text sequence is expanded and supplemented.
[0122] For example: If the text unit to be predicted is the end of a sentence, the predicted text generated by the model can be directly inserted into the end of the original sentence to complete the filling.
[0123] In general, through the above steps, the model can automatically predict and fill in new text content based on the known encoded text content, realizing autoregressive filling of the encoded text, thereby providing richer and more complete data for subsequent tasks.
[0124] In detail, the step of performing text prediction on the text units to be predicted in sequence according to the text order to obtain the predicted text of the encoded text includes:
[0125] S21, determining the first text unit to be predicted in the encoded text according to the text order;
[0126] S22, generating the first unit text of the first text unit to be predicted;
[0127] S23, determining the adjacent text unit to be predicted of the first text unit to be predicted according to the text order;
[0128] S24, generating adjacent unit texts of the adjacent text unit to be predicted according to the first unit text, and repeating steps S23-S24 until the last unit text of the last text unit to be predicted in the encoded text is generated;
[0129] S25, collecting all generated first unit texts, adjacent unit texts and last unit texts as predicted text of the encoded text.
[0130] In detail, determining the first text unit to be predicted in the encoded text according to the text order refers to determining the first text unit to be predicted in the encoded text that needs to be predicted according to the specified text order, wherein the text unit to be predicted refers to a text segment in the encoded text that needs to be predicted according to the context content. Determining the first text unit to be predicted is the starting point of the entire autoregressive filling process, which determines the order of subsequent predictions and filling.
[0131] For example, if the encoded text is a conversation, the first unit of text to be predicted may be the first sentence in the conversation; if the encoded text is a sentence, the first unit of text generated may be the first few words or phrases of the sentence.
[0132] In detail, generating the first unit text of the first text unit to be predicted refers to generating the predicted text of the first text unit to be predicted based on the known encoded text, wherein the first unit text refers to the predicted content of the text unit to be predicted generated based on the known text.
[0133] In detail, determining the adjacent text units to be predicted of the first text unit to be predicted according to the text order refers to determining the adjacent text units of the first text unit to be predicted according to the text order so as to perform the next step of text prediction, wherein the adjacent text units to be predicted refer to the text units to be predicted that immediately follow the first text unit to be predicted, and determining the adjacent text units to be predicted is to gradually advance the prediction and filling process in the entire text sequence.
[0134] For example, if the first text unit to be predicted is a sentence, then the adjacent text unit to be predicted may be the next sentence in the sentence.
[0135] In detail, generating adjacent unit texts of adjacent text units to be predicted based on the first unit text, and repeating the steps until the last unit text of the last text unit to be predicted in the encoded text is generated means generating adjacent unit texts of adjacent text units to be predicted in sequence based on the generated first unit text, until the last unit text of the last text unit to be predicted in the encoded text is generated. This step is repeated to cover all text units to be predicted in the entire encoded text, ensuring that each text unit is correctly predicted and filled. By cyclically generating the predicted text of each text unit to be predicted, the coherence and consistency of the entire encoded text are ensured, and no part will be missed during the filling process.
[0136] In detail, aggregating all generated first unit texts, adjacent unit texts and last unit texts into predicted text of the coded text means aggregating all generated unit texts in order to form predicted text of the coded text, wherein the aggregation operation integrates all predicted unit texts together to form a complete predicted text sequence, and by aggregating all predicted unit texts, a complete predicted result of the coded text is obtained, providing a usable data basis for subsequent tasks.
[0137] In detail, the method of using the predicted text to fill the coded text to obtain the filled text of the coded text includes:
[0138] Determining position labels of text units in the predicted text;
[0139] The encoded text is sequentially filled according to the position tags and the text units to obtain the filled text of the encoded text.
[0140] Specifically, the position labels of the text units in each predicted text are determined so that the encoded text can be subsequently filled in according to these labels, wherein the position label refers to an identifier used to mark the position information of each text unit in the predicted text, usually a number or a symbol.
[0141] For example, you can use numbers to represent the order of each text unit in the predicted text, such as 1 represents the first text unit, 2 represents the second text unit, and so on. A common practice is to use an index or counter to assign a position label to each text unit to ensure uniqueness and order.
[0142] In detail, determining the position tags can help us uniquely identify and locate each text unit, so that we can operate accurately in sequence during the filling process.
[0143] In detail, sequentially filling the encoded text according to the position tags and text units means filling the text units into the encoded text in sequence according to the position tags and corresponding text units in the predicted text to generate the filling text.
[0144] In detail, a text unit refers to a text segment in the predicted text, which may be a word, a phrase, or a sentence.
[0145] For example, if the position label and text unit in the predicted text are (1, "Hello") and (2, "world") respectively, then "Hello" and "world" are sequentially filled into the encoded text based on this information.
[0146] In the embodiment of the present invention, the step of generating risk data in the multi-source data according to the encoded text after the autoregressive filling includes:
[0147] Extracting key features from the encoded text after the autoregressive padding;
[0148] Performing feature classification on the key features to obtain classification features of the key features;
[0149] generating a risk score for the classification feature according to the feature category corresponding to the classification feature;
[0150] A risk report of the key feature is determined according to the risk score and a preset risk threshold, and risk data in the multi-source data is determined according to the risk report.
[0151] In detail, the key features refer to decisive or significant words, phrases or patterns extracted from the coded text, which are crucial for understanding the text content and identifying potential risks; classification features are the result of classifying key features into predefined categories, which may include different risk types or business areas; risk score is a quantitative indicator that represents the degree of potential risk assessed based on the classification features, and the high or low risk score is usually associated with the severity and urgency of the risk; risk threshold is a preset limit value used to determine whether the risk score reaches a level that requires action; risk report is a detailed document generated based on risk score and risk threshold, which records relevant information of the risk, such as risk type, scope of impact, possible consequences, etc.
[0152] In detail, extracting key features involves natural language processing technology, such as keyword extraction algorithms, to identify important risk-related information from text; feature classification of key features can be performed through machine learning classification models to assign the extracted features to pre-defined risk categories, such as "credit risk", "market risk", etc.; generating risk scores can utilize quantitative models, such as logistic regression or decision trees, to evaluate the severity of each risk event based on classification features and give a score; determining risk reports can combine risk scores and preset thresholds to formulate risk response strategies and form formal report documents for decision makers' reference.
[0153] For example: A financial institution is processing data from multiple sources (such as transaction records, customer complaints, external market data, etc.) with the goal of identifying the risk of sensitive information leakage.
[0154] First, extract key features from the encoded text after autoregressive filling, which may include specific financial terms, amounts, account information, transaction frequency, etc. For example, keywords such as "unauthorized transfers" or "abnormal account balances" are extracted from customer complaint data.
[0155] Next, the extracted key features are classified into predefined categories, such as "suspicious transactions", "personal information leakage", etc. For example, "unauthorized transfers" are classified as "suspicious transactions", while "personal account information being inquired" is classified as "personal information leakage".
[0156] Secondly, a risk score is generated according to the feature category corresponding to the classification feature, which may be based on historical data analysis, model prediction, etc. For example, if the "personal account information is asked" event is often associated with information leakage in past data, the system may give such events a high risk score.
[0157] Finally, the risk score and the preset risk threshold are combined to determine whether a risk report needs to be generated. For example, if the risk score exceeds the preset threshold (such as 70 / 100), a risk report is generated to point out the potential risk of information leakage.
[0158] Or, a healthcare organization is processing data from multiple sources (such as patient medical records, drug sales records, medical research reports, etc.) with the goal of identifying data that may contain risk in order to comply with data protection regulations and protect patient privacy.
[0159] In detail,
[0160] Extract key features from the encoded text after autoregressive filling, which may include specific medical terms, patient name, contact information, diagnosis information, etc. For example, keywords such as "patient name", "diagnosis result", and "contact information" are extracted from the patient's medical record.
[0161] Furthermore, the extracted key features are classified into predefined categories, such as “personal identification information”, “health condition”, etc. For example, “patient name” and “contact information” are classified as “personal identification information”, while “diagnosis result” is classified as “health condition”.
[0162] Then. In this scenario, the risk score can be generated based on the sensitivity of the feature, for example, data that directly identifies an individual has a higher sensitivity. For example, since "patient name" and "contact information" are directly personally identifiable information, the system may give such information a high sensitivity score.
[0163] Finally, the risk score and the preset risk threshold are combined to determine whether a risk report needs to be generated. For example, if the risk score exceeds the preset threshold (such as 70 / 100), a risk report is generated to point out the potential risk of data leakage and recommend further investigation.
[0164] Furthermore, when processing data, special attention should be paid to identifying information involving personal privacy, such as name, ID number, contact information, diagnosis information, etc. For example, if a medical research report contains the patient's full name and contact information, which is not necessary information, the system should mark the report as containing sensitive data.
[0165] S5. Desensitize the risk data to obtain the governance data of the target enterprise.
[0166] In the embodiment of the present invention, the risk data is desensitized to obtain the governance data of the target enterprise in order to ensure that the sensitive information in its risk data is properly handled, while the generated governance data maintains the integrity and availability of the data.
[0167] In detail, first, it is necessary to accurately identify sensitive information in risky data. This may include personal identity information, financial data, health records, etc. For example, sensitive information such as account number, transaction amount, personal contact information, etc. can be identified from customer transaction records.
[0168] Furthermore, the identified sensitive data is classified to determine which information needs to be desensitized. For example, personal contact information and account numbers are classified as direct personal identification information and need to be desensitized first.
[0169] In detail, select the appropriate desensitization method according to the type and sensitivity of the data. Common desensitization methods include masking, pseudo data generation, hash encryption, randomization, etc. For example, for account numbers, you can use the method of masking part of the numbers; for personal contact information, you can use a randomly generated phone number to replace the real number.
[0170] Furthermore, the selected desensitization methods are implemented for sensitive information in risky data to ensure that the data retains its original structure and function while removing or replacing sensitive parts. For example, the account number in the customer transaction record is displayed as "******8912" and the contact information is replaced with a randomly generated phone number.
[0171] Furthermore, the desensitized data is verified to ensure that all sensitive information has been effectively processed and the data's usefulness is not affected. For example, internal audits or third-party assessments can be used to confirm the validity and compliance of desensitized data.
[0172] In detail, the desensitized risk data is converted into the governance data of the enterprise, which can be used for business analysis, decision support, etc. under the premise of complying with data protection regulations. For example, desensitized customer transaction records are used to analyze consumption patterns without disclosing customer privacy. The embodiment of the present invention can efficiently identify and generate risk data by automatically encoding and filling multi-source data using the ChatGLM model. Through the autoregressive filling technology, the gaps or errors in the data can be effectively filled to improve the integrity and accuracy of the data. In the process of generating risk data, sensitive information can be identified and desensitized to enhance data privacy protection. Therefore, the data governance method based on the ChatGLM model proposed in the present invention can solve the problem of low efficiency in data governance.
[0173] like Figure 2 , which is a schematic diagram of the structure of an electronic device for implementing a data governance method based on a ChatGLM model provided by an embodiment of the present invention.
[0174] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a data management program based on the ChatGLM model.
[0175] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a data management program based on the ChatGLM model, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.
[0176] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device. Further, the memory 11 can also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the data management program based on the ChatGLM model, but also can be used to temporarily store data that has been output or is to be output.
[0177] The communication bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.
[0178] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.
[0179] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0180] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.
[0181] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.
[0182] The data management program based on the ChatGLM model stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:
[0183] Collect multi-source data of target enterprises;
[0184] Performing data rule matching on the multi-source data, and determining that the successfully matched multi-source data is model data of a preset ChatGLM model;
[0185] Using the preset ChatGLM model to perform self-encoding processing on the model data to obtain the encoded text of the model data;
[0186] Performing autoregressive filling on the coded text, and generating risk data in the multi-source data according to the coded text after the autoregressive filling;
[0187] The risk data is desensitized to obtain the governance data of the target enterprise.
[0188] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, which will not be repeated here.
[0189] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0190] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:
[0191] Collect multi-source data of target enterprises;
[0192] Performing data rule matching on the multi-source data, and determining that the successfully matched multi-source data is model data of a preset ChatGLM model;
[0193] Using the preset ChatGLM model to perform self-encoding processing on the model data to obtain the encoded text of the model data;
[0194] Performing autoregressive filling on the coded text, and generating risk data in the multi-source data according to the coded text after the autoregressive filling;
[0195] The risk data is desensitized to obtain the governance data of the target enterprise.
[0196] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways.
[0197] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0198] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0199] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0200] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0201] The embodiments of the present application can acquire and process relevant data based on big data technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A data governance method based on the ChatGLM model, characterized in that: The method comprises: Collect multi-source data of target enterprises; Performing data rule matching on the multi-source data, and determining that the successfully matched multi-source data is model data of a preset ChatGLM model; Using the preset ChatGLM model to perform self-encoding processing on the model data to obtain the encoded text of the model data; Performing autoregressive filling on the coded text, and generating risk data in the multi-source data according to the coded text after the autoregressive filling; The risk data is desensitized to obtain the governance data of the target enterprise.
2. The data governance method based on the ChatGLM model as claimed in claim 1, characterized in that: The performing data rule matching on the multi-source data and determining that the successfully matched multi-source data is model data of a preset ChatGLM model includes: Extracting a data structure of the multi-source data; Performing outlier correction on the multi-source data, and calculating a matching degree between the multi-source data after outlier correction and the data structure; When the matching degree is greater than a preset matching threshold, model data of the preset ChatGLM model is generated according to the multi-source data corresponding to the matching degree.
3. The data governance method based on the ChatGLM model as claimed in claim 2, characterized in that: The generating of the model data of the preset ChatGLM model according to the multi-source data corresponding to the matching degree includes: Determine the multi-source data corresponding to the matching degree as initial data, and perform text-to-numeric attribute conversion on the initial data; Determine the value range of the initial data according to the data type in the data structure; The initial data after the text-to-numeric attribute conversion is numerically scaled according to the numerical range to obtain model data of the preset ChatGLM model.
4. The data governance method based on the ChatGLM model as claimed in claim 1, characterized in that: The using the preset ChatGLM model to perform self-encoding processing on the model data to obtain the encoded text of the model data includes: Performing vector transformation on the model data using the ChatGLM model to obtain a data vector of the model data; The model data is data-encoded according to the data vector, and the encoded model data is used to generate an encoded text of the model data.
5. The data governance method based on the ChatGLM model as claimed in claim 4, characterized in that: The step of encoding the model data according to the data vector comprises: Determine the row position and column position of the model data according to the data vector; Performing sinusoidal coding on the row position to obtain a row code of the model data; Performing cosine coding on the column positions to obtain column coding of the model data; Generate a two-dimensional position coding matrix of the model data according to the row coding and the column coding; Vector concatenation is performed on the two-dimensional position coding matrix and the data vector to obtain coding data of the model data.
6. The data governance method based on the ChatGLM model as claimed in claim 1, characterized in that: The step of performing autoregressive filling on the encoded text comprises: Determining a text unit to be predicted of the encoded text and a text order of the text unit to be predicted; Perform text prediction on the text units to be predicted in sequence according to the text sequence to obtain predicted text of the encoded text; The predicted text is used to perform text filling on the encoded text to obtain the filled text of the encoded text.
7. The data governance method based on the ChatGLM model as claimed in claim 6, characterized in that: The step of performing text prediction on the text units to be predicted in sequence according to the text order to obtain predicted text of the encoded text includes: S21, determining the first text unit to be predicted in the encoded text according to the text order; S22, generating the first unit text of the first text unit to be predicted; S23, determining the adjacent text unit to be predicted of the first text unit to be predicted according to the text order; S24, generating adjacent unit texts of the adjacent text unit to be predicted according to the first unit text, and repeating steps S23-S24 until the last unit text of the last text unit to be predicted in the encoded text is generated; S25, collecting all generated first unit texts, adjacent unit texts and last unit texts as predicted text of the encoded text.
8. The data governance method based on the ChatGLM model as claimed in claim 1, characterized in that: The step of generating risk data in the multi-source data according to the encoded text after the autoregressive filling comprises: Extracting key features from the encoded text after the autoregressive padding; Performing feature classification on the key features to obtain classification features of the key features; generating a risk score for the classification feature according to the feature category corresponding to the classification feature; A risk report of the key feature is determined according to the risk score and a preset risk threshold, and risk data in the multi-source data is determined according to the risk report.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data governance method based on the ChatGLM model as described in any one of claims 1 to 8.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the data governance method based on the ChatGLM model as described in any one of claims 1 to 8.
