A large model-based data governance method, device and medium

Abstract

The application discloses a large model-based data governance method and device and medium, and relates to the technical field of electric digital data processing. The method comprises the following steps: based on a preset large model, performing data exploration on sample data to obtain sample metadata corresponding to the sample data, and storing the sample metadata in a preset data knowledge base; for a target table to be governed, the target table is compared with the data knowledge base to determine target table fields required for governing the target table and a governance operation function for establishing a mapping relationship between the target table fields and corresponding source table fields according to a comparison result; the expected field credibility of the source table fields is predicted according to the influence effect of the governance operation function on the source table fields, and a governance task corresponding to the target table is generated based on the governance operation function and the expected field credibility; and the governance task is sent to a processing node to realize data governance on the target table through the processing node.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, specifically to a data governance method, device, and medium based on a large model. Background Technology

[0002] With the accelerated advancement of digital transformation, data has become a core asset for enterprises. Data governance, as the core process for enterprises to effectively manage and control the quality of data assets, aims to improve the accuracy, consistency, and credibility of data, thereby supporting business decisions and value extraction. Traditional data governance technologies mainly revolve around the standardized management of structured data, heavily relying on expert experience and manual configuration. When faced with massive amounts of multi-source data, these technologies are inefficient and struggle to guarantee the objectivity and consistency of governance strategies. Summary of the Invention

[0003] To address the aforementioned issues, this application proposes a data governance method based on large models, including:

[0004] Based on a pre-defined large model, data exploration is performed on the sample data to obtain the sample metadata corresponding to the sample data, and the sample metadata is stored in a pre-defined data knowledge base; wherein, the sample metadata includes a source table, source table fields and their corresponding field metadata, and the field metadata includes at least field confidence;

[0005] For the target table to be governed, the target table is compared with the data knowledge base to determine the target table fields required for governing the target table and the governance operation function used to establish the mapping relationship between the target table fields and their corresponding source table fields based on the comparison results.

[0006] Based on the impact of the governance operation function on the source table fields, the expected field credibility of the source table fields is predicted, and based on the governance operation function and the expected field credibility, the governance task corresponding to the target table is generated.

[0007] The governance task is distributed to the processing node so that the data governance of the target table can be achieved through the processing node.

[0008] In one implementation of this application, the target table is compared with the data knowledge base to determine the target table fields required for governing the target table based on the comparison results, specifically including:

[0009] Obtain the source table from the data knowledge base, and compare the field descriptions of each field in the source table and the target table to calculate the semantic similarity between the field descriptions;

[0010] Source tables with semantic similarity greater than a preset threshold are selected as candidate source tables for accessing the target table;

[0011] The field names and field descriptions corresponding to each field in the candidate source table and the target table are sequentially compared and correlated to obtain the matching degree between each field;

[0012] The field with the highest matching degree in the candidate source table is selected as the target table field required for governing the target table.

[0013] In one implementation of this application, the expected field confidence of the source table field is predicted based on the impact effect of the governance operation function on the source table field, specifically including:

[0014] For each governance operation function, identify the function type corresponding to the governance operation function, and construct a business context based on the business types corresponding to the source table and the target table respectively;

[0015] Based on the function type and the business context, at least one impact rule corresponding to the governance operation function is selected from a pre-set historical impact rule library; wherein, the historical impact rule library includes historical impact records of different types of governance operation functions on field credibility under different business context combinations;

[0016] Based on the at least one influence rule, determine the historical average influence value corresponding to the governance operation function;

[0017] Based on the historical average impact, the field credibility of the source table field is adjusted to obtain the corresponding expected field credibility.

[0018] In one implementation of this application, the field confidence level corresponding to the source table field is corrected based on the historical average impact value to obtain the corresponding expected field confidence level, specifically including:

[0019] Determine whether the governance operation function is the first execution function targeting the field in the source table;

[0020] If so, the expected field confidence level corresponding to the source table field is calculated based on the sum of the historical average impact and the field confidence level.

[0021] If not, the historical average impact values ​​corresponding to the governance operation functions executed on the source table fields are accumulated sequentially. Based on the accumulated historical average impact values ​​and the credibility decay coefficient corresponding to the superimposed effect of the governance operation functions, the field credibility corresponding to the source table fields is corrected to obtain the corresponding expected field credibility.

[0022] In one implementation of this application, a governance task corresponding to the target table is generated based on the governance operation function and the expected field confidence level, specifically including:

[0023] Determine whether the expected field credibility of the source table field meets the preset credibility conditions;

[0024] If not, the governance parameters of the governance operation function are adjusted, and several governance tasks are generated based on the adjusted governance parameters; wherein, each governance task corresponds to a version of the target table, and the governance parameters include the execution order of the governance operation function, the implementation algorithm, and the key parameters of the function.

[0025] In one implementation of this application, determining whether the expected field credibility of the source table field meets a preset credibility condition specifically includes:

[0026] Determine whether the credibility of the expected field is not less than a preset credibility threshold;

[0027] If so, obtain the basic sample information corresponding to the source table field in the data knowledge base; wherein, the basic sample information includes the sample size and sample source;

[0028] Based on the basic information of the samples, determine the variance of the confidence distribution of the source table fields among the various sample sources;

[0029] If the sample size is less than a preset sample size threshold and the variance of the confidence distribution is greater than a preset variance, or if the confidence of the expected field is less than the confidence threshold, it is determined that the confidence of the expected field corresponding to the source table field does not meet the preset confidence condition.

[0030] In one implementation of this application, based on a preset large model, data exploration is performed on the sample data to obtain sample metadata corresponding to the sample data, specifically including:

[0031] Obtain sample data and the initial table structure corresponding to the sample data from the currently connected data source;

[0032] Based on a pre-defined large model, data exploration is performed on the sample data to determine the source table and source table fields corresponding to the sample data;

[0033] Based on the null value rate of each source table field in the sample data, the corresponding field credibility is determined; based on the field type and field annotation information of the initial table structure and industry standard documents, the field type and field description of the sample data are determined; and based on the field description, a code set with semantic similarity to the source table field is obtained.

[0034] Using the field credibility, field type, field description, and code set as field metadata, the sample metadata corresponding to the sample data is obtained based on the source table, the source table fields, and the field metadata.

[0035] In one implementation of this application, after the governance task is distributed to the processing node, the method further includes:

[0036] According to the preset monitoring cycle, obtain the node status corresponding to each processing node in the processing node group;

[0037] If the load difference between the processing nodes in the processing node group exceeds a preset threshold when all nodes are online, the processing nodes are arranged in descending order of load to obtain the corresponding node sequence.

[0038] For each processing node in the node sequence, a corresponding node performance profile is generated based on the load and task execution logs of the processing node.

[0039] Based on the node performance profile, determine the compatibility between the target processing node in the node sequence whose load exceeds a preset load threshold and other processing nodes in the processing node group;

[0040] Based on the adaptability, the governance tasks currently being executed in the target processing node are migrated to other processing nodes.

[0041] This application provides a data governance device based on a large model, the device comprising:

[0042] At least one processor;

[0043] And, a memory communicatively connected to the at least one processor;

[0044] The memory stores instructions that can be executed by the at least one processor, which, when executed, enable the at least one processor to perform a large-model-based data governance method as described in any of the preceding claims.

[0045] This application provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as: a data governance method based on a large model as described in any of the preceding claims.

[0046] The data governance method based on a large model proposed in this application can bring the following beneficial effects:

[0047] By automating the exploration of sample data using a large model, metadata containing information such as field credibility is generated and a knowledge base is built. This replaces the subjective and inefficient manual exploration process that heavily relies on expert experience, achieving semantic understanding of massive, multi-source data. Subsequently, through intelligent association and comparison between the target table and the knowledge base, matching source table fields and governance operation functions are automatically recommended. This transforms the traditional process, which relies on manual judgment and configuration, into a data-driven, precise matching process, ensuring the objectivity and consistency of the governance strategy from the source. By quantitatively predicting the impact of governance operations on field credibility and dynamically optimizing the governance scheme when conditions are not met, the final data quality can be pre-assessed and guaranteed before task execution, overcoming the drawback of unpredictable governance effects in traditional methods. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 A flowchart illustrating a data governance method based on a large model, provided for an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the structure of a data governance device based on a large model, provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 As shown in the embodiments of this application, a data governance method based on a large model is provided, including:

[0054] S101: Based on the preset large model, perform data exploration on the sample data to obtain the sample metadata corresponding to the sample data, and store the sample metadata in the preset data knowledge base; wherein, the sample metadata includes the source table, the source table fields and their corresponding field metadata, and the field metadata includes at least the field credibility.

[0055] Data governance refers to processing and integrating scattered, disorganized raw data from various business systems into a unified target table according to specific business specifications and quality requirements. It aims to solve problems such as data silos and data inconsistencies, providing reliable data support for enterprise decision-making by constructing standardized target tables. In the initial stage of data governance, the first step is to explore the sample data to fully understand its basic characteristics. First, user needs are identified. Based on these needs, relevant industry standards and data requiring governance are collected, and a corresponding data knowledge base is built. Then, the current data source is determined based on the database connection information provided by the user. Sample data and its corresponding initial table structure are obtained from the data source and input into a pre-set large model. This pre-set large model can be a large language model trained on a large amount of text data. After receiving the aforementioned parameters, the large model processes them while simultaneously loading industry standard documents from the data knowledge base. It then slices and reduces the dimensionality of these documents to generate a vector library. Using Retrieval-augmented Generation (RAG) technology, it compares and retrieves the vector library against the parameters passed to the large model, thus exploring the sample data and obtaining the corresponding sample metadata. During this exploration, the large model automatically identifies entities, relationships, and data patterns within the sample data, determining which source tables the sample data originates from and which specific source table fields each table contains. After obtaining the source tables and their fields, the large model further extracts and calculates field metadata. Field metadata includes at least field confidence, which refers to the reliability of the field within the dataset. Data exploration essentially aims to obtain detailed field information from the sample data. This detailed information is stored as sample metadata in the data knowledge base, where it, along with the industry standard documents, performs data governance on the data to be governed.

[0056] In one embodiment, sample data and the corresponding initial table structure are obtained from the currently connected data source. The initial table structure and sample data are then input into the large model. The pre-defined large model is invoked to perform in-depth analysis of the obtained sample data. First, by analyzing the sample data content, table name context, and relationships between fields, the source table and its fields are determined. After identifying the data exploration targets, precise field analysis of the source tables is required to obtain detailed field information, i.e., field metadata. Field metadata includes not only field confidence but also field name, field type, field description, field length, whether it is a primary key, whether it can be nullable, and the corresponding code set. Specifically, the large model determines field confidence based on the null value rate of each source table field in the sample data. Specifically, it calculates the proportion of null values ​​appearing in the source table field in the sample data relative to the total number of records. A higher null value rate indicates lower field confidence, and vice versa. For example, if a field in a source table has 50 null values ​​out of 1000 sample data entries, its null value rate is 5%. Based on a pre-defined mapping rule between null value rate and credibility, the credibility of that source table field can be calculated. Simultaneously, the large model determines the field type and description corresponding to the sample data based on the field types and annotations in the initial table structure. For instance, if a field in the initial table structure is labeled as `int` and annotated as "user age," the large model can determine that the field type is integer and the field description is used to record user age information. Furthermore, the large model also searches the data knowledge base for code sets semantically similar to the source table field based on the field description. For example, when the field description is "user gender," the large model will match code sets containing "gender." A code set refers to a dictionary of codes that are semantically similar or related to the source table field. For example, when the source table field is "gender," its corresponding code set could be a set of numbers, with different numbers corresponding to different genders, such as 1 for male and 2 for female. Finally, the information obtained above, such as field credibility, field type, field description, and code set, is integrated into field metadata and associated with the source table and source table fields to form the sample metadata corresponding to the sample data. This metadata is then stored in the data knowledge base to provide basic data support for subsequent data governance operations.

[0057] S102: For the target table to be governed, the target table is compared with the data knowledge base to determine the target table fields required for governance based on the comparison results, as well as the governance operation functions used to establish the mapping relationship between the target table fields and their corresponding source table fields.

[0058] In this embodiment, data governance refers to accessing data from a source table, performing operations such as comparison, deduplication, normalization, extraction, and tagging, and then outputting the processed data to a target table. Therefore, in the initial stage of data governance, a target table to be governed needs to be selected. The target table is the standardized data storage structure that the enterprise expects to ultimately form through data governance. Subsequently, the target table is compared with the sample metadata already constructed in the data knowledge base. Based on the comparison results, the target table fields whose table names are associated with and logically related to the target table fields need to be determined. Simultaneously, governance operation functions need to be determined to establish the mapping relationship between the target table fields and their corresponding source table fields, including but not limited to data normalization, data extraction, dictionary code conversion, and field format conversion. These governance operation functions can process the source table fields, converting them into data formats and content that meet the requirements of the target table fields.

[0059] In one embodiment, a source table is retrieved from a data knowledge base. The field descriptions of each field in the source table and the target table are compared and correlated to calculate the semantic similarity between the field descriptions. Source tables with a semantic similarity greater than a preset threshold are then selected as candidate source tables for accessing the target table. For example, if a field description in the target table is a user's contact number, and a field description in a source table in the data knowledge base is a customer's mobile phone number, the semantic similarity between the two is calculated using a large model. If the similarity exceeds a preset threshold, the source table containing that field is selected as a candidate source table. For the selected candidate source tables, the field names and descriptions corresponding to each field in the candidate source table and the target table are sequentially compared and correlated to obtain the matching degree between each field. The matching degree is calculated based on the character matching degree of the field name and the semantic similarity of the field description. For example, if the target table field name is "user name" and the candidate source table field name is "customer name," the character matching degree is high, and both field descriptions record user name information, indicating high semantic similarity. Therefore, the matching degree between the candidate source table field and the target table field is high. Based on the matching degree of each field, the field with the highest matching degree is selected from the candidate source table and used as the source of the target table fields required for governance. Therefore, different fields in the target table originate from different candidate source tables.

[0060] After determining the target table fields and their corresponding source table fields, the large model automatically generates or selects appropriate governance operation functions based on the target table fields' field types, descriptions, code sets, and the source table fields' metadata. For example, if the target table field requires the standard date format "YYYY-MM-DD," while the corresponding source table field uses "MM / DD / YYYY," the large model will determine that a date format conversion function needs to be called as the governance operation function to achieve data format uniformity. As another example, when the code sets of the target table fields differ from those of the source table fields—for instance, the gender field in the target table uses numbers while the source table uses male and female—the large model will generate a dictionary code conversion function to map the code values ​​of the source table fields to the code values ​​required by the target table fields. These governance operation functions will be used for subsequent specific processing of the source table field data to ensure that the processed data can be accurately populated into the corresponding fields of the target table.

[0061] S103: Based on the impact of the governance operation function on the source table fields, predict the expected field credibility of the source table fields, and generate the governance task corresponding to the target table based on the governance operation function and the expected field credibility.

[0062] In traditional data governance processes, once governance operation functions are determined, they are executed directly. However, different governance operation functions have varying impacts on data quality. For example, a format conversion function may correct data errors and improve credibility, while a numerical derivation function may introduce new uncertainties due to algorithmic errors. Without prior effect evaluation, the credibility of fields after governance may not meet expectations, or even decrease, ultimately affecting the data governance effect. To address this, this application's embodiments predict the expected field credibility of the source table based on the impact of a series of recommended governance operation functions on the field credibility of the source data before executing the governance task. Based on this prediction result, the feasibility of the current governance scheme is determined, or how to optimize and adjust it, thereby generating the governance task corresponding to the target table to ensure the quality and reliability of the final target table from the source.

[0063] In one embodiment, for each governance operation function, its corresponding function type is identified, such as data cleaning, format conversion, code mapping, numerical calculation, etc., and a corresponding business context is constructed by combining the business types of the source table and the target table. The business context refers to the business scenario information related to the governance operation function. For example, in data governance in the financial industry, when processing fields involving user account information, the business context may include account type, transaction rules, compliance requirements, etc. Based on the business context, the impact of the same type of governance operation function on data credibility in different business scenarios can be more accurately judged and evaluated.

[0064] Furthermore, the system queries a pre-built historical impact rule base. This base is built by accumulating historical governance task execution effect analysis data. It records the historical impact of different types of governance operation functions on field credibility under various business context combinations, typically quantified as a credibility change value, such as +5% or -3%. Based on the current function type and business context, the system matches and selects one or more of the most relevant impact rules from the historical rule impact base. When selecting impact rules, they can be sorted according to their matching degree, prioritizing the rules with the highest matching degree.

[0065] Furthermore, based on at least one selected influence rule, the historical average impact of the governance operation function on credibility in this specific context is calculated. The historical average impact is calculated based on the average credibility change value corresponding to the selected influence rule. Based on the calculated historical average impact, the original field credibility of the source table field is corrected. This correction typically uses a linear overlay method, where the historical average impact is used as an adjustment and added to the original field credibility to obtain the expected field credibility of the source table field after applying the current governance operation function.

[0066] In one embodiment, when correcting field confidence, the combined quality impact of multiple governance operation functions applied sequentially to the same source table field is not a simple linear sum of the independent effects of each function. If only the independent impact of each governance operation function is evaluated, operational interference or information loss introduced during continuous processing may be overlooked, leading to a deviation of the final predicted field confidence from the actual situation. Therefore, after obtaining the historical average impact of a single governance operation function for a source table field, it is first necessary to determine whether the governance operation function is the first function executed for that source table field. Here, "first" refers to the first function applied to that field in the entire processing chain planned to generate the current target table. If it is the first governance operation function, it indicates that the original state of the field has not been affected by this governance process. The historical average impact can be directly summed with the field confidence of that field, and the final sum is the expected field confidence after applying the governance operation function to the source table field. If the governance operation function is not the first function executed, it means that one or more preceding governance operation functions have been planned for this field. In this case, the governance operation needs to consider the combined effect of the function stacking. Specifically, the historical average impact values ​​corresponding to the governance operation functions for the same source table field are accumulated, and a confidence decay coefficient is introduced. The confidence decay coefficient is an empirical parameter (usually between 0 and 1) derived from the analysis of a large amount of historical data. It is used to quantify the discount or attenuation phenomenon that occurs due to error propagation, rule conflicts, and other factors under the continuous processing of governance operation functions. The accumulated historical impact value is then discounted based on this confidence decay coefficient, that is, the product between the confidence decay coefficient and the accumulated historical average impact value is calculated. The discounted result is then summed with the field confidence of the source table field to finally obtain the expected field confidence of the source table field.

[0067] In one embodiment, after obtaining the expected field credibility of all relevant fields, it is necessary to determine whether the expected field credibility of the source table fields meets the preset credibility conditions. First, it is determined whether the expected field credibility is not less than a preset credibility threshold. The credibility threshold refers to the minimum threshold value for data quality requirements. If the expected field credibility is less than this threshold, it is directly determined that the expected quality of the field does not meet the credibility conditions.

[0068] If the expected field credibility is not less than the credibility threshold, further judgment on the field's credibility is needed to ensure the accuracy of the evaluation results. At this point, it is necessary to obtain the basic sample information corresponding to the source table field in the data knowledge base. This basic information includes the sample size and sample source. The sample source refers to which data tables the data originates from, and the sample size refers to the total number of data records for the source table field. Based on this basic sample information, the variance of the credibility distribution of the source table field across different sample sources is determined. The variance measures the dispersion of the field's credibility across different sample sources. A smaller variance indicates more stable credibility across different sample sources; a larger variance indicates significant differences in data quality across different sample sources. Therefore, even if the expected field credibility meets the credibility threshold, if the sample size supporting this evaluation result is too small (less than the preset sample size threshold) and the calculated variance of the credibility distribution is greater than the preset variance, the expected field credibility still does not meet the preset credibility condition. This is because the sample size of the original sample supporting the evaluation results is insufficient, and the cross-source distribution variance is too large. When the governance operation function faces the source table fields in heterogeneous data sources, the stability of the governance effect is questionable.

[0069] After all source table fields meet the trustworthiness criteria, the governance operation functions, the mapping relationship between target table fields and source table fields, and the expected field trustworthiness are integrated to generate a target table governance task that includes the task execution order and function parameter configuration. This governance task is then stored in the task scheduling system for subsequent execution. If the criteria are not met, it indicates that the data quality of the target table will be expected to be substandard after governance according to the current recommended solution. In this case, the governance parameters of the governance operation functions need to be automatically adjusted and reorganized. These governance parameters include the execution order of the governance operation functions, the specific implementation algorithm selected, and key parameters within the functions. Based on different combinations of governance parameters, multiple adjusted governance tasks are automatically generated. Each governance task corresponds to a version of the target table and its expected quality level, allowing users to make decisions or the system to automatically select the optimal version based on the optimization goals. This effectively avoids target table data quality problems caused by improper selection of governance operation functions or parameter settings at the source stage, ensuring that the final generated governance task can improve data trustworthiness while meeting the business's data quality requirements.

[0070] S104: Distribute the governance task to the processing node so that the data governance of the target table can be achieved through the processing node.

[0071] The governance task is distributed to the corresponding node group. Based on the real-time load of each processing node in the node group, the processing node with the lowest load is dynamically selected to execute the governance task, thereby achieving the governance of the target table.

[0072] In one embodiment, if all nodes are heavily loaded, accepting a task will exceed the load threshold, resulting in a task delivery failure and a notification message. Therefore, the node engine dynamically monitors the node status of all processing nodes according to a preset monitoring cycle to ensure the continuous and healthy execution of tasks. Node status includes whether the node is online and its specific load. If a node is shut down, the tasks running on that node can be dynamically switched through a re-election process. If the load of the remaining nodes is close to the threshold, no switching will occur, and a background message will be sent to the user. If a node is heavily loaded, the engine will search for the node with the lowest load among the remaining nodes to see if there is any available load. If so, task migration will be performed for load balancing.

[0073] Specifically, assuming all nodes are online, if the load difference between processing nodes in a processing node group exceeds a preset threshold, the processing nodes are arranged in descending order of load, resulting in a corresponding node sequence. For each processing node in the node sequence, a corresponding node performance profile is generated based on its load and task execution logs. This profile includes multiple dimensions such as historical task throughput, average task response time, resource utilization fluctuation range, types of tasks it excels at handling, and error handling capabilities. Based on the node performance profile, and considering the computational complexity, data volume, and timeliness requirements of the current governance task, the processing node with the highest suitability for the task characteristics is selected from the node sequence. The governance task currently being executed on the target processing node is then migrated to other processing nodes.

[0074] The above are embodiments of the methods proposed in this application. Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0075] Figure 2 This is a schematic diagram of the structure of a data governance device based on a large model, provided as an embodiment of this application. Figure 2 As shown, it includes:

[0076] At least one processor; and,

[0077] At least one processor-communication-connected memory; wherein,

[0078] The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to:

[0079] Implement a large-model-based data governance approach as described in any of the preceding items.

[0080] This application provides a non-volatile computer storage medium storing computer-executable instructions, which are configured as follows: a data governance method based on a large model as described in any of the preceding claims.

[0081] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0082] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0083] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0088] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0089] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0090] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0091] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data governance method based on a large model, characterized in that, The method includes: Based on a pre-defined large model, data exploration is performed on the sample data to obtain the sample metadata corresponding to the sample data, and the sample metadata is stored in a pre-defined data knowledge base; wherein, the sample metadata includes a source table, source table fields and their corresponding field metadata, and the field metadata includes at least field confidence; For the target table to be governed, the target table is compared with the data knowledge base to determine the target table fields required for governing the target table and the governance operation function used to establish the mapping relationship between the target table fields and their corresponding source table fields based on the comparison results. Based on the impact of the governance operation function on the source table fields, the expected field credibility of the source table fields is predicted, and based on the governance operation function and the expected field credibility, the governance task corresponding to the target table is generated. The governance task is distributed to the processing node so that the data governance of the target table can be achieved through the processing node; Based on the impact of the governance operation function on the source table fields, the expected field confidence of the source table fields is predicted, specifically including: For each governance operation function, identify the function type corresponding to the governance operation function, and construct a business context based on the business types corresponding to the source table and the target table respectively; Based on the function type and the business context, at least one impact rule corresponding to the governance operation function is selected from a pre-set historical impact rule library; wherein, the historical impact rule library includes historical impact records of different types of governance operation functions on field credibility under different business context combinations; Based on the at least one influence rule, determine the historical average influence value corresponding to the governance operation function; Based on the historical average impact, the field confidence level corresponding to the source table field is corrected to obtain the corresponding expected field confidence level; Based on the governance operation function and the expected field confidence level, a governance task corresponding to the target table is generated, specifically including: Determine whether the expected field credibility of the source table field meets the preset credibility conditions; If not, the governance parameters of the governance operation function are adjusted, and several governance tasks are generated based on the adjusted governance parameters; wherein, each governance task corresponds to a version of the target table, and the governance parameters include the execution order of the governance operation function, the implementation algorithm, and the key parameters of the function.

2. The data governance method based on a large model according to claim 1, characterized in that, The target table is compared and linked with the data knowledge base to determine the target table fields required for governance based on the comparison results, specifically including: Obtain the source table from the data knowledge base, and compare the field descriptions of each field in the source table and the target table to calculate the semantic similarity between the field descriptions; Source tables with semantic similarity greater than a preset threshold are selected as candidate source tables for accessing the target table; The field names and field descriptions corresponding to each field in the candidate source table and the target table are sequentially compared and correlated to obtain the matching degree between each field; The field with the highest matching degree in the candidate source table is selected as the target table field required for governing the target table.

3. The data governance method based on a large model according to claim 1, characterized in that, Based on the historical average impact, the field confidence levels corresponding to the source table fields are adjusted to obtain the corresponding expected field confidence levels, specifically including: Determine whether the governance operation function is the first execution function targeting the field in the source table; If so, the expected field confidence level corresponding to the source table field is calculated based on the sum of the historical average impact and the field confidence level. If not, the historical average impact values ​​corresponding to the governance operation functions executed on the source table fields are accumulated sequentially. Based on the accumulated historical average impact values ​​and the credibility decay coefficient corresponding to the superimposed effect of the governance operation functions, the field credibility corresponding to the source table fields is corrected to obtain the corresponding expected field credibility.

4. The data governance method based on a large model according to claim 1, characterized in that, Determining whether the expected field credibility of the source table field meets the preset credibility conditions specifically includes: Determine whether the credibility of the expected field is not less than a preset credibility threshold; If so, obtain the basic sample information corresponding to the source table field in the data knowledge base; wherein, the basic sample information includes the sample size and sample source; Based on the basic information of the samples, determine the variance of the confidence distribution of the source table fields among the various sample sources; If the sample size is less than a preset sample size threshold and the variance of the confidence distribution is greater than a preset variance, or if the confidence of the expected field is less than the confidence threshold, it is determined that the confidence of the expected field corresponding to the source table field does not meet the preset confidence condition.

5. The data governance method based on a large model according to claim 1, characterized in that, Based on a pre-defined large model, data exploration is performed on the sample data to obtain the corresponding sample metadata, specifically including: Obtain sample data and the initial table structure corresponding to the sample data from the currently connected data source; Based on a pre-defined large model, data exploration is performed on the sample data to determine the source table and source table fields corresponding to the sample data; Based on the null value rate of each source table field in the sample data, the corresponding field credibility is determined; based on the field type and field annotation information of the initial table structure and industry standard documents, the field type and field description of the sample data are determined; and based on the field description, a code set with semantic similarity to the source table field is obtained. Using the field credibility, field type, field description, and code set as field metadata, the sample metadata corresponding to the sample data is obtained based on the source table, the source table fields, and the field metadata.

6. The data governance method based on a large model according to claim 1, characterized in that, After the governance task is distributed to the processing node, the method further includes: According to the preset monitoring cycle, obtain the node status corresponding to each processing node in the processing node group; If the load difference between the processing nodes in the processing node group exceeds a preset threshold when all nodes are online, the processing nodes are arranged in descending order of load to obtain the corresponding node sequence. For each processing node in the node sequence, a corresponding node performance profile is generated based on the load and task execution logs of the processing node. Based on the node performance profile, determine the compatibility between the target processing node in the node sequence whose load exceeds a preset load threshold and other processing nodes in the processing node group; Based on the adaptability, the governance tasks currently being executed in the target processing node are migrated to other processing nodes.

7. A data governance device based on a large model, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a large-model-based data governance method as described in any one of claims 1-6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured as follows: a data governance method based on a large model as described in any one of claims 1-6.

Images