A collaborative management method for multi-tenant master data quality in smart grids

Abstract

The present invention relates to a multi-tenant master data quality collaborative management method for smart grids, comprising the following steps: S1: obtaining the original business data of each tenant's business system and performing preliminary cleaning to obtain a pre-cleaned master data set; S2: performing data quality testing on the pre-cleaned master data set to identify abnormal data sets; S3: summarizing the causes of abnormal data in the abnormal data set based on traceability analysis to obtain an abnormal attribution report and classified problem work orders; S4: using a policy recommendation model to recommend the optimal processing strategy based on the abnormal attribution report and classified problem work orders, and distributing work orders according to tenants and permissions to generate a collaborative governance task flow; S5: performing master data repair based on the optimal processing strategy and the collaborative governance task flow to obtain corrected master data. The present invention implements multi-tenant master data quality collaborative management for smart grids.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to a multi-tenant master data quality collaborative management method for smart grids. Background Art

[0002] With the rapid development of smart grids, a large number of heterogeneous business applications and data platforms have emerged in power systems. Various equipment, monitoring, marketing, and operations systems continuously generate and exchange massive amounts of master data (such as equipment profiles, user information, operations and maintenance work orders, and geographic locations), involving multiple tenants and departments. In this context, problems such as data silos, inconsistent standards, and complex data transfer chains have become increasingly prominent. Master data is prone to quality issues such as redundancy, omissions, errors, and inconsistencies during collection, transmission, and processing, seriously impacting the smooth operation of smart grid operations and intelligent decision-making support.

[0003] Traditional data quality management is often decentralized across systems, often lacking cross-departmental and cross-tenant collaboration mechanisms. Furthermore, the governance process relies on manual operations, resulting in long cycles and low efficiency. This makes it difficult to meet the new demands of "wide data access, efficient governance, and precise data transfer" in the smart grid. Furthermore, with the reform of the power market and business innovation, data quality issues are increasingly impacting business security, management compliance, and operational efficiency. A new, intelligent, and collaborative master data quality management approach is urgently needed to achieve full-process governance of multi-source, heterogeneous master data. Summary of the Invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide a multi-tenant master data quality collaborative management method for smart grids, thereby realizing multi-tenant master data quality collaborative management for smart grids.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A multi-tenant master data quality collaborative management method for smart grids includes the following steps:

[0007] S1: Obtain the original business data of each tenant's business system and perform preliminary cleaning to obtain the master data set after preliminary cleaning;

[0008] S2: Perform data quality testing on the primary data set after preliminary cleaning to identify abnormal data sets;

[0009] S3: Based on traceability analysis, the causes of abnormal data are summarized for abnormal data sets, and an abnormal attribution report and classified problem work orders are obtained;

[0010] S4: Based on the anomaly attribution report and classified problem tickets, the policy recommendation model is used to recommend the best handling strategy, and tickets are distributed according to tenants and permissions to generate a collaborative governance task flow;

[0011] S5: Repair the master data based on the optimal processing strategy and collaborative governance task flow to obtain the corrected master data.

[0012] Furthermore, the original business data of each tenant's business system is obtained and preliminarily cleaned to obtain the preliminarily cleaned master data set, as follows:

[0013] Each tenant's business system synchronizes the original business data to the master data platform through a set integration method, and accesses data partition storage based on tenant ID or tenant domain;

[0014] After receiving the original data from each tenant, the master data platform performs data standardization and structuring:

[0015] Map each tenant's fields to a unified standard field name, and convert date values ​​and string fields into a unified data format;

[0016] The text is uniformly converted into the character set specified by the platform to avoid garbled characters or storage errors caused by encoding differences;

[0017] Deconstruct the original unstructured or semi-structured data and flatten the data into a standard field structure that the database can receive and analyze;

[0018] And through the initial screening of dirty data, we get the master data set after preliminary cleaning.

[0019] Furthermore, the original unstructured or semi-structured data is structured and flattened into a standard field structure that the database can receive and analyze, as follows:

[0020] First, use the built-in data parser to identify whether the data uploaded by the business system contains nested objects, arrays, or irregular key-value pairs;

[0021] For each field in a nested structure, the platform uses the mapping expansion rule of parent field_child field to expand the nested content level by level;

[0022] List fields are adapted to specific business scenarios. For lists shorter than a preset value and with a fixed structure, they are expanded item by item using the "segment name_sequence number_subfield" method. For one-to-many lists with variable length or complex content, they are stored by creating a detail table, with each detail linked to the main table via the primary key.

[0023] The master data platform splits the processed results into a master table and a detail table. The master table stores one-to-one basic attributes, while the detail table specifically stores multiple pieces of information that originally belonged to nested arrays and corresponds to the master table through foreign keys.

[0024] During the disassembly process, the master data platform unifies the types and standardizes the formats of field contents, including converting all texts to UTF-8 encoding, converting dates and times to a standard format, and converting Boolean fields to a unified value range.

[0025] Furthermore, the initial screening of dirty data includes null value detection, format verification, validity and value range checks, uniqueness, and simple duplication detection, as follows:

[0026] The null value detection checks the key fields in all master data sets, including primary keys and required fields, for null values. Records with missing key fields are marked as dirty data and removed from the main process.

[0027] The format check uses a predefined template to check whether the preset fields conform to the specified format;

[0028] The legality and value range check, for fields with clear value ranges or value rules, checks whether the field content is within the allowed range;

[0029] The uniqueness and simple duplicate detection checks whether there are duplicates in the data of the preset business fields that are required to be unique. For the duplicate items detected, all are marked as suspicious dirty data and entered into the exception processing pool.

[0030] Furthermore, the data quality of the primary data set after preliminary cleaning is tested to identify abnormal data, as follows:

[0031] Use the classification model to classify and identify the primary data set after preliminary cleaning to identify regular abnormal data;

[0032] Use anomaly detection models to detect hidden abnormal data on the primary data set after preliminary cleaning;

[0033] According to the regular abnormal data and hidden abnormal data, the final abnormal data set is obtained.

[0034] Furthermore, the classification model is used to classify and identify the primary data set after preliminary cleaning, and identify regular abnormal data, as follows:

[0035] Extract historically labeled normal and abnormal data from the main data warehouse to form a training set; label definition:

[0036] ;

[0037] Extract data features to construct feature vectors, including field length features, missing markers, regularity check, business rule check, dictionary matching, primary and foreign key integrity, duplication, and time features; among them, the i-th data forms the feature vector X i ; yi is the label of the i-th data;

[0038] Use LightGBM to build a gradient boosting tree ensemble two-classification model. The model goal is to learn X i with y i The loss function uses the Logistic loss function:

[0039] ;

[0040] Among them, LOSS represents the loss function, N is the amount of training set data; is the predicted abnormal probability;

[0041] The LightGBM model generates logit scores through an additive tree structure :

[0042] ;

[0043] Among them, T k (X i ) is the k-th tree for sample X i The output of; K is the total number of trees;

[0044] Use Sigmoid function to convert to probability :

[0045] ;

[0046] σ represents the Sigmoid function;

[0047] Set the threshold θ for risk control:

[0048] Filter {i|P abn,i ≥θ}, and is summarized as the regular anomaly dataset E clf .

[0049] Furthermore, we used the anomaly detection model to detect hidden abnormal data on the primary cleaned data set, as follows:

[0050] For the eigenvector X i , perform normalization to obtain X' i ;Build an anomaly detection model based on the improved autoencoder, including encoder and decoder;

[0051] The encoder uses an L-layer perceptron structure and gradually reduces the dimension:

[0052] ;

[0053] Among them, σ L is the activation function of the L-th layer encoder, W(L) is the weight matrix of the L-th layer perceptron, b (L) is the bias vector of the L-th layer perceptron; z i is the low-dimensional representation of the encoder output; h (L-1) is the output of the L-1 layer perceptron;

[0054] The decoder is an L-layer structure symmetrical to the encoder, gradually restoring the dimensions:

[0055]

[0056] in, To reconstruct data; is the activation function of the L-th layer decoder; is the output of the L-1 layer of the decoder; 、 are the weight and bias of the Lth layer of the decoder respectively;

[0057] Anomaly score calculation:

[0058] Overall reconstruction error :

[0059] ;

[0060] Weighted reconstruction error :

[0061] ;

[0062] Among them, m represents the total number of features; w j represents the weight coefficient of the jth feature; represents the true value of the jth feature of the i-th sample; represents the reconstructed value of the jth feature of the i-th sample;

[0063] Field-level anomaly scores :

[0064] ;

[0065] Sparsity penalty :

[0066] ;

[0067] Final anomaly score :

[0068] ;

[0069] Where λ represents the balance coefficient;

[0070] For each data record, the final anomaly score is compared with the threshold, and those greater than the threshold are identified as hidden anomaly data.

[0071] Furthermore, for abnormal data sets, based on traceability analysis, the causes of abnormal data are summarized, and abnormal attribution reports and classified problem tickets are generated, as follows:

[0072] For the identified abnormal data sets, collect the details of the abnormal data one by one, including the associated metadata;

[0073] Combined with the data lineage information provided by the master data platform, the abnormal data is traced to its source, and the nodes and processing links of the abnormal data in the entire process of collection, conversion, transmission, mapping, and storage are obtained. If the data has a primary key and change records, the entire path of the abnormal value from the first entry into the system to the current consumption link is retrieved. If there is a data modification log, all historical operation events related to the abnormal data are extracted, including entry, synchronization, modification, and abnormality alarms.

[0074] Combine the business events closest to the abnormal data to find the external causes related to the abnormality;

[0075] Conduct preliminary classification based on abnormal manifestation types, including missing values, format errors, illegal encoding, primary and foreign key conflicts, duplicate records, and value drift;

[0076] Use data traceability results, field distribution, and change records to attribute similar anomalies to the same cause;

[0077] Organize the above traceability and attribution results into attribution reports and trackable work orders to promote abnormal governance and form a business closed loop.

[0078] Furthermore, based on the anomaly attribution report and classified problem tickets, the policy recommendation model recommends the best handling strategy, distributes tickets by tenant and permission, and generates a collaborative governance task flow, as follows:

[0079] For each abnormal work order i′ and possible processing strategy s∈S, define the comprehensive benefit function :

[0080]

[0081] Where A(s) is the accuracy score, A(s)∈[0,1]; T(s) is the estimated completion time; C(s) is the resource cost; I(s) is the business impact, I(s)≥0; w A 、w T 、w C 、w I is the weight of each target, and w A +w T +w C +wI =1;T max 、C max , I max is the normalization coefficient of the corresponding indicator;

[0082] Optimal strategy selection :

[0083] ;

[0084] Determine the most suitable role assignee based on abnormal characteristics and optimal strategy i :

[0085] ;

[0086] Among them, R is the set of all possible processing roles; the Match function calculates the role r and the work order F i' degree of matching;

[0087] For exceptions involving multi-tenant data, introduce tenant permission constraints :

[0088] ;

[0089] When actually assigning, permission constraints must be met, so the most suitable role is Assignee i' :

[0090] ;

[0091] According to the priority of the work order, the sorted work order queue is obtained and the collaborative governance task flow is constructed;

[0092] For complex work orders that require multi-role collaboration, define collaborative processing procedures:

[0093] ;

[0094] Among them: Role j′ For the execution role; Task j′ For specific tasks; DependsOn j′ A collection of pre-dependent tasks; State j′ is the task status; is the structure of the i′th collaborative process; m′ is the number of tasks in the collaborative process; j′ represents the j′th subtask in the collaborative process.

[0095] The present invention has the following beneficial effects:

[0096] 1. This invention implements collaborative management of multi-tenant master data quality for smart grids. Through multi-level cleaning, standardization, and structured decomposition, it improves the standardization capabilities of multi-source heterogeneous data. Furthermore, by combining rule-based verification with intelligent algorithm models, it achieves efficient anomaly detection and refined classification covering the entire lifecycle of master data. Combined with data lineage analysis, it can track changes in data throughout the entire process, accurately attribute the source of anomalies, and address problematic data in a closed-loop collaborative process driven by work orders.

[0097] 2. This invention not only ensures the strict isolation and security of data from different tenants, but also significantly improves the automation and intelligence level of master data quality management through intelligent distribution and multi-role collaboration. It can automatically generate the optimal task flow and personnel assignment based on the exception type, processing priority and tenant permissions, greatly reducing the intensity of manual participation and the possibility of misjudgment. Not only can routine exceptions be discovered efficiently, but potential hidden data quality risks can also be revealed in a timely manner, thus significantly improving the comprehensiveness and accuracy of data management.

[0098] 3. This invention can improve the smart grid master data compliance governance, business collaboration, and data-driven intelligent operation level, laying a solid foundation for the high-quality development of the digital grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0100] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0101] refer to Figure 1 In this embodiment, a multi-tenant master data quality collaborative management method for smart grid is provided, including the following steps:

[0102] S1: Obtain the original business data of each tenant's business system and perform preliminary cleaning to obtain the master data set after preliminary cleaning;

[0103] S2: Perform data quality testing on the primary data set after preliminary cleaning to identify abnormal data sets;

[0104] S3: Based on traceability analysis, the causes of abnormal data are summarized for abnormal data sets, and an abnormal attribution report and classified problem work orders are obtained;

[0105] S4: Based on the anomaly attribution report and classified problem tickets, the policy recommendation model is used to recommend the best handling strategy, and tickets are distributed according to tenants and permissions to generate a collaborative governance task flow;

[0106] S5: Repair the master data based on the optimal processing strategy and collaborative governance task flow to obtain the corrected master data.

[0107] In this embodiment, the original business data of each tenant's business system is obtained and preliminarily cleaned to obtain a preliminarily cleaned master data set, as follows:

[0108] Each tenant's business system synchronizes raw business data to the master data platform through a predefined integration method (such as API, ETL, file upload, or message middleware), and stores the accessed data in partitions based on tenant ID or tenant domain.

[0109] After receiving the original data from each tenant, the master data platform performs data standardization and structuring (unifying fields, formats, time zones, and encodings):

[0110] Different tenants' raw data may use different field names for the same information (e.g., "User Number," "Account ID," "UserID"). Map each tenant's fields to a unified, standardized field name, and convert date, numeric, and string fields to a unified data format. For example, all time fields are standardized to UTC or the main site's time zone, with a standardized format (e.g., yyyy-MM-dd HH:mm:ss). Numeric values ​​are standardized with standardized units and decimal places, and the value ranges for Boolean and enumeration fields are unified.

[0111] The text is uniformly converted into the platform-specified character set (such as UTF-8) to avoid garbled characters or storage errors caused by encoding differences;

[0112] Deconstructs raw unstructured or semi-structured data (such as nested JSON and list fields) into a structured form, flattening the data into a standard field structure that the database can receive and analyze.

[0113] And through the initial screening of dirty data (simple verification such as null values ​​and format errors), the master data set that has been preliminarily cleaned is obtained.

[0114] In this embodiment, the original unstructured or semi-structured data is structured and flattened into a standard field structure that can be received and analyzed by the database, as follows:

[0115] First, use built-in data parsers (such as JSON / XML parsers) to identify whether the data uploaded by the business system contains nested objects, arrays, or irregular key-value pairs. For example, a user's information may contain multiple contact information, device information, or geolocation attributes, which are often present in the form of nested structures or lists within the data record.

[0116] For each field in a nested structure, the platform uses the mapping expansion rule of parent field_child field to expand the nested content level by level. For example, "location": {"province": "Jiangsu", "city": "Nanjing"} in the original data will be mapped and expanded into two independent fields: location_province and location_city.

[0117] List fields (such as contact number lists) are adapted based on specific business scenarios. For lists shorter than a preset value and with a fixed structure, they are expanded item by item using the "segment name_sequence number_subfield" method. For one-to-many lists with variable length or complex content, they are stored in a detailed table, with each detail linked to the main table via the primary key.

[0118] The master data platform splits the processed results into a master table and detail tables. The master table stores one-to-one basic attributes (such as user ID, name, city, etc.), while the detail table specifically stores multiple pieces of information that were originally nested arrays and links them to the master table through foreign keys. For example, multiple contact information within a user record will be split into several detail table entries, each of which retains the user ID as a foreign key.

[0119] During the decomposition process, the master data platform unifies the types and standardizes the formats of field contents. This includes converting all text to UTF-8 encoding, converting dates and times to a standard format (such as yyyy-MM-dd HH:mm:ss), and converting Boolean fields to a unified value range.

[0120] In this embodiment, the initial screening of dirty data includes null value detection, format verification, validity and value range check, uniqueness and simple duplication detection, as follows:

[0121] The null value detection checks all key fields in the master data set, including primary keys and required fields (such as user name, device number, contact information, etc.). If a record is found to have missing key fields, it will be directly marked as dirty data and removed from the main process to prevent the missing core data from affecting system stability.

[0122] The format verification uses predefined templates to check whether the preset fields (such as mobile phone number, email address, date, code, ID number, geographic coordinates, etc.) conform to the specified format. For example, mobile phone numbers can only be 11 digits, and the date format must be "yyyy-MM-dd" or the platform's unified standard format.

[0123] The aforementioned legality and value range checks check whether the content of fields with clear value domains or value rules (such as enumeration types, Booleans, and numeric ranges) is within the permitted range. For example, the device type can only be "A," "B," or "C," geographic coordinates must be within the legal range, and age cannot be negative.

[0124] The uniqueness and simple duplicate detection checks whether there are duplications in the data of preset business fields that are required to be unique (such as primary keys, user IDs, device numbers, etc.). For detected duplicates, all of them are marked as suspicious dirty data and entered into the exception processing pool.

[0125] In this embodiment, the data quality test is performed on the primary data set after preliminary cleaning to identify abnormal data, as follows:

[0126] Use the classification model to classify and identify the primary data set after preliminary cleaning to identify regular abnormal data;

[0127] Use anomaly detection models to detect hidden abnormal data on the primary data set after preliminary cleaning;

[0128] Based on regular exception data and hidden exception data, the final exception data set (by tenant level, field / table / object dimension) is obtained.

[0129] In this embodiment, the classification model is used to classify and identify the primary data set after preliminary cleaning to identify conventional abnormal data, as follows:

[0130] Extract historically labeled normal and abnormal data from the main data warehouse to form a training set; label definition:

[0131] ;

[0132] Extract data features to construct feature vectors, including field length features, missing markers, regularity check, business rule check, dictionary matching, primary and foreign key integrity, duplication, and time features; among them, the i-th data forms the feature vector X i ; y i is the label of the i-th data;

[0133] Use LightGBM to build a gradient boosting tree ensemble two-classification model. The model goal is to learn X i with y i The loss function uses the Logistic loss function:

[0134] ;

[0135] Among them, LOSS represents the loss function, N is the amount of training set data; is the predicted abnormal probability;

[0136] The LightGBM model generates logit scores through an additive tree structure :

[0137] ;

[0138] Among them, T k (X i ) is the k-th tree for sample X i The output of; K is the total number of trees;

[0139] Use Sigmoid function to convert to probability :

[0140] ;

[0141] σ represents the Sigmoid function;

[0142] Set a threshold value θ (such as 0.5, 0.7) for risk control:

[0143] Filter {i|P abn,i ≥θ}, and is summarized as the regular anomaly dataset E clf .

[0144] In this embodiment, an anomaly detection model is used to detect hidden anomaly data on the primary cleaned data set, as follows:

[0145] For the eigenvector X i , perform normalization to obtain X' i ;Build an anomaly detection model based on the improved autoencoder, including encoder and decoder;

[0146] The encoder uses an L-layer perceptron structure and gradually reduces the dimension:

[0147] ;

[0148] Among them, σ L is the activation function of the L-th layer encoder (such as ReLU, LeakyReLU, etc.), W (L) is the weight matrix of the L-th layer perceptron, b (L) is the bias vector of the L-th layer perceptron; z i is the low-dimensional representation of the encoder output; h (L-1) is the output of the L-1 layer perceptron;

[0149] The decoder is an L-layer structure symmetrical to the encoder, gradually restoring the dimensions:

[0150]

[0151] in, To reconstruct data; is the activation function of the L-th layer decoder; is the output of the L-1 layer of the decoder; 、 are the weight and bias of the Lth layer of the decoder respectively;

[0152] Anomaly score calculation:

[0153] Overall reconstruction error :

[0154] ;

[0155] Weighted reconstruction error :

[0156] ;

[0157] Among them, m represents the total number of features; w j represents the weight coefficient of the jth feature; represents the true value of the jth feature of the i-th sample; represents the reconstructed value of the jth feature of the i-th sample;

[0158] Field-level anomaly scores :

[0159] ;

[0160] Sparsity penalty (For special field combinations):

[0161] ;

[0162] Final anomaly score :

[0163] ;

[0164] Where λ represents the balance coefficient;

[0165] For each data record, based on the comparison of its anomaly score with the threshold, the data with a score greater than the threshold is identified as hidden anomaly data.

[0166] In this embodiment, based on the source analysis of abnormal data sets, the causes of abnormal data are summarized, and abnormal attribution reports and classified problem work tickets are generated, as follows:

[0167] For each identified abnormal data set, collect detailed information about the abnormal data, including associated metadata (such as primary key, tenant ID, source system, operation time, etc.);

[0168] Combined with the data lineage information provided by the master data platform, abnormal data is traced to its source, and the nodes and processing links that the data passes through in the entire process of collection, conversion, transmission, mapping, and storage are obtained. If the data has a primary key and change records, the entire path of the abnormal value from the first entry into the system to the current consumption link is retrieved. If there is a data modification log, all historical operation events related to the data are extracted, including entry, synchronization, modification, and abnormality alarms.

[0169] Abnormal features and abnormal point location focus on fields or objects detected as abnormal, and analyze their specific changes during the data lifecycle, including changes in values, structures, and associated fields.

[0170] Combine recent business events related to the abnormal data (such as import, batch interface data synchronization, business process operation anomalies, etc.) to find external triggers related to the abnormality.

[0171] Conduct preliminary classification based on abnormal manifestation types, including missing values, format errors, illegal encoding, primary and foreign key conflicts, duplicate records, and value drift;

[0172] Use data traceability results, field distribution, and change records to attribute similar anomalies to the same cause;

[0173] For example: unclear business-side master data entry rules lead to format errors; external interface source data is missing in batch synchronization; outdated dictionary entries lead to legality verification failures; and omissions in cross-system mapping rule changes lead to primary and foreign key errors.

[0174] To manage traceability and reuse, gradually establish a standardized anomaly attribution label system; for example: type = "missing value", cause = "upstream source data is incomplete", process = "synchronous storage", object = "field X" / "table Y" / "tenant Z", impact = "medium"

[0175] Organize the above traceability and attribution results into attribution reports and trackable work orders to promote abnormal governance and form a business closed loop.

[0176] Specific process:

[0177] Attribution reports generate structured data attribution reports, mainly including: a global anomaly overview (anomaly rate, distribution, and trend); anomaly details and main causes by tenant / table / field; top N statistics and sample cases of common anomaly causes for each anomaly type; impact assessment of major anomaly causes; governance recommendations for common causes (such as rule optimization, dictionary maintenance, and interface adjustments); and dedicated governance recommendations for individual causes (such as tenant-specific issues and customized process remediation guidelines).

[0178] Automatically assign classified problem tickets based on attribution tags and business impact. The main contents include: ticket type (abnormal type + attribution tag), impact scope (related tenants, tables, fields, business objects, etc.),

[0179] Details of abnormal samples and key traceability links; recommended responsible departments / teams / individuals; work order priorities and handling suggestions.

[0180] In this embodiment, based on the anomaly attribution report and classified problem tickets, the policy recommendation model recommends the best handling strategy, distributes tickets by tenant and permission, and generates a collaborative governance task flow, as follows:

[0181] For each abnormal work order i′ and possible processing strategy s∈S, define the comprehensive benefit function :

[0182]

[0183] Where A(s) is the accuracy score, A(s)∈[0,1]; T(s) is the estimated completion time; C(s) is the resource cost; I(s) is the business impact, I(s)≥0; w A 、w T 、w C 、w I is the weight of each target, and w A +w T +w C +w I =1;T max 、C max , I max is the normalization coefficient of the corresponding indicator;

[0184] The policy library S includes but is not limited to: automatic repair (such as rule replacement and filling missing values); manual re-entry (manual review and correction); cross-tenant collaborative review (multi-party confirmation); and feedback from upstream source systems.

[0185] Synchronous update of data dictionary; rule optimization and metadata update; standardization governance (coding unification and format standardization);

[0186] Optimal strategy selection :

[0187] ;

[0188] Determine the most suitable role assignee based on abnormal characteristics and optimal strategy i :

[0189] ;

[0190] Among them, R is the set of all possible processing roles; the Match function calculates the role r and the work order F i' degree of matching;

[0191] For exceptions involving multi-tenant data, introduce tenant permission constraints :

[0192] ;

[0193] When actually assigning, permission constraints must be met, so the most suitable role is Assignee i' :

[0194] ;

[0195] According to the priority of the work order, the sorted work order queue is obtained and the collaborative governance task flow is constructed;

[0196] For complex work orders that require multi-role collaboration, define collaborative processing procedures:

[0197] ;

[0198] Among them: Role j′ For the execution role; Task j′ For specific tasks; DependsOn j′ A collection of pre-dependent tasks; State j′ is the task status; is the structure of the i′th collaborative process; m′ is the number of tasks in the collaborative process; j′ represents the j′th subtask in the collaborative process.

[0199] A multi-tenant master data quality collaborative management system for smart grids includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps of the multi-tenant master data quality collaborative management method for smart grids as described above.

[0200] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0204] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A multi-tenant master data quality collaborative management method for smart grid, characterized by: The following steps are involved: S1: Obtain the original business data of each tenant's business system and perform preliminary cleaning to obtain the master data set after preliminary cleaning; S2: Perform data quality testing on the primary data set after preliminary cleaning to identify abnormal data sets; S3: Based on traceability analysis, the causes of abnormal data are summarized for abnormal data sets, and an abnormal attribution report and classified problem work orders are obtained; S4: Based on the anomaly attribution report and classified problem tickets, the policy recommendation model is used to recommend the best handling strategy, and tickets are distributed according to tenants and permissions to generate a collaborative governance task flow; S5: Perform master data repair based on the optimal processing strategy and collaborative governance task flow to obtain the corrected master data; Obtain the original business data of each tenant's business system and perform preliminary cleaning to obtain the master data set after preliminary cleaning, as follows: Each tenant's business system synchronizes the original business data to the master data platform through a set integration method, and accesses data partition storage based on tenant ID or tenant domain; After receiving the original data from each tenant, the master data platform performs data standardization and structuring: Map each tenant's fields to a unified standard field name, and convert date values ​​and string fields into a unified data format; The text is uniformly converted into the character set specified by the platform to avoid garbled characters or storage errors caused by encoding differences; Deconstruct the original unstructured or semi-structured data and flatten the data into a standard field structure that the database can receive and analyze; And through the initial screening of dirty data, we get the master data set after preliminary cleaning; For abnormal data sets, we analyze the source and summarize the causes of abnormal data, and then generate abnormal attribution reports and classification problem tickets. The details are as follows: For the identified abnormal data sets, collect the details of the abnormal data one by one, including the associated metadata; Combined with the data lineage information provided by the master data platform, the abnormal data is traced to its source, and the nodes and processing links of the abnormal data in the entire process of collection, conversion, transmission, mapping, and storage are obtained. If the data has a primary key and change records, the entire path of the abnormal value from the first entry into the system to the current consumption link is retrieved. If there is a data modification log, all historical operation events related to the abnormal data are extracted, including entry, synchronization, modification, and abnormality alarms. Combine the business events closest to the abnormal data to find the external causes related to the abnormality; Conduct preliminary classification based on abnormal manifestation types, including missing values, format errors, illegal encoding, primary and foreign key conflicts, duplicate records, and value drift; Use data traceability results, field distribution, and change records to attribute similar anomalies to the same cause; Organize the above traceability and attribution results into attribution reports and trackable work orders to promote abnormal governance and form a business closed loop.

2. The method for collaborative management of multi-tenant master data quality for smart grids according to claim 1, characterized in that: The original unstructured or semi-structured data is structured and flattened into a standard field structure that can be received and analyzed by the database, as follows: First, use the built-in data parser to identify whether the data uploaded by the business system contains nested objects, arrays, or irregular key-value pairs; For each field in a nested structure, the platform uses the mapping expansion rule of parent field_child field to expand the nested content level by level; List fields are adapted based on specific business scenarios. For lists shorter than a preset value and with a fixed structure, they are expanded item by item using the "segment name_sequence number_subfield" method. For one-to-many lists with variable length or complex content, they are stored in a detail table, with each detail linked to the main table using the primary key. The master data platform splits the processed results into a master table and a detail table. The master table stores one-to-one basic attributes, while the detail table stores multiple pieces of information that originally belonged to nested arrays and corresponds to the master table through foreign keys. During the disassembly process, the master data platform unifies the types and standardizes the formats of field contents, including converting all texts to UTF-8 encoding, converting dates and times to a standard format, and converting Boolean fields to a unified value range.

3. The method for collaborative management of multi-tenant master data quality for smart grids according to claim 1, characterized in that: The initial screening of dirty data includes null value detection, format verification, legality and value range check, uniqueness and simple duplication detection, as follows: The null value detection checks the key fields in all master data sets, including primary keys and required fields, for null values. Records with missing key fields are marked as dirty data and removed from the main process. The format check uses a predefined template to check whether the preset fields conform to the specified format; The legality and value range check, for fields with clear value ranges or value rules, checks whether the field content is within the allowed range; The uniqueness and simple duplicate detection checks whether there are duplicates in the data of the preset business fields that are required to be unique. For the duplicate items detected, all are marked as suspicious dirty data and entered into the exception processing pool.

4. The method for collaborative management of multi-tenant master data quality for smart grids according to claim 1, characterized in that: The master data set after preliminary cleaning is subjected to data quality testing to identify abnormal data, as follows: Use the classification model to classify and identify the primary data set after preliminary cleaning to identify regular abnormal data; Use anomaly detection models to detect hidden abnormal data on the primary data set after preliminary cleaning; According to the regular abnormal data and hidden abnormal data, the final abnormal data set is obtained.

5. The method for collaborative management of multi-tenant master data quality for smart grids according to claim 4, characterized in that: The classification model is used to classify and identify the primary data set after preliminary cleaning, and identify conventional abnormal data, as follows: Extract historically labeled normal and abnormal data from the main data warehouse to form a training set; label definition: ； Extract data features to construct feature vectors, including field length features, missing markers, regularity check, business rule check, dictionary matching, primary and foreign key integrity, duplication, and time features; among them, the i-th data forms the feature vector X i ; y i is the label of the i-th data; Use LightGBM to build a gradient boosting tree ensemble two-classification model. The model goal is to learn X i with y i The loss function uses the Logistic loss function: ； Among them, LOSS represents the loss function, N is the amount of training set data; is the predicted abnormal probability; The LightGBM model generates logit scores through an additive tree structure: ； Among them, T k (X i ) is the k-th tree for sample X i The output of; K is the total number of trees; Use Sigmoid function to convert to probability : ； σ represents the Sigmoid function; Set the threshold θ for risk control: Filter {i|P abn,i ≥θ}, and is summarized as the regular anomaly dataset E clf .

6. The method for collaborative management of multi-tenant master data quality for smart grids according to claim 5, characterized in that: The anomaly detection model is used to detect the primary data set after preliminary cleaning to find hidden abnormal data, as follows: For the eigenvector X i , perform normalization to obtain X' i ;Build an anomaly detection model based on the improved autoencoder, including encoder and decoder; The encoder uses an L-layer perceptron structure and gradually reduces the dimension: ； Among them, σ L is the activation function of the L-th layer encoder, W (L) is the weight matrix of the L-th layer perceptron, b (L) is the bias vector of the L-th layer perceptron; z i is the low-dimensional representation of the encoder output; h (L-1) is the output of the L-1 layer perceptron; The decoder is an L-layer structure symmetrical to the encoder, gradually restoring the dimensions: ； in, To reconstruct data; is the activation function of the L-th layer decoder; is the output of the L-1 layer of the decoder; 、 are the weight and bias of the Lth layer of the decoder respectively; Anomaly score calculation: Overall reconstruction error : ； Weighted reconstruction error : ； Among them, m represents the total number of features; w j represents the weight coefficient of the jth feature; represents the true value of the jth feature of the i-th sample; represents the reconstructed value of the jth feature of the i-th sample; Field-level anomaly scores : ； Sparsity penalty : ； Final anomaly score : ； Where λ represents the balance coefficient; For each data record, the final anomaly score is compared with the threshold, and those greater than the threshold are identified as hidden anomaly data.

7. The method for collaborative management of multi-tenant master data quality for smart grids according to claim 6, characterized in that: Based on the anomaly attribution report and classified problem tickets, the policy recommendation model is used to recommend the best processing strategy, and tickets are distributed according to tenants and permissions to generate a collaborative governance task flow, as follows: For each abnormal work order i′ and processing strategy s∈S, define the comprehensive benefit function : ； Where A(s) is the accuracy score, A(s)∈[0,1]; T(s) is the estimated completion time; C(s) is the resource cost; I(s) is the business impact, I(s)≥0; w A 、w T 、w C 、w I is the weight of each target, and w A +w T +w C +w I =1;T max 、C max , I max is the normalization coefficient of the corresponding indicator; Optimal strategy selection : ； Determine the most suitable role for handling based on abnormal characteristics and optimal strategies Assignee i’ : ； Among them, R is the set of all processing roles; the Match function calculates the role r and the work order F i degree of matching; For exceptions involving multi-tenant data, introduce tenant permission constraints : ； When actually assigning permissions, the most suitable role to handle the role must meet the permission constraints. Assignee i’ : ； According to the priority of the work order, the sorted work order queue is obtained and the collaborative governance task flow is constructed; For complex work orders that require multi-role collaboration, define collaborative processing procedures: ； Among them: Role j′ For the execution role; Task j′ For specific tasks; DependsOn j′ A collection of pre-dependent tasks; State j′ is the task status; is the structure of the i′th collaborative process; m′ is the number of tasks in the collaborative process; j′ represents the j′th subtask in the collaborative process.

Images