Self-evolution emergence type data governance method and system

Abstract

The invention discloses a self-evolution emerging type data governance method and system, and relates to the technical field of data governance, the method comprises the following steps: inputting a multi-source heterogeneous initial data set into a large model, and executing the following processing by the large model: performing quality evaluation and defect identification on the initial data set to obtain a ternary data set; based on a preset quality score threshold and the data source knowledge graph, performing cyclic self-emerging data supplementation according to the ternary data set to obtain a complete data set; performing data governance on the complete data set by adopting a preset governance rule set to obtain a high-quality data set, performing self-evolution on governance rules, and updating parameters of a large model through closed-loop feedback so as to update the preset governance rule set; the updated preset governance rule set is used for participating in data governance of the next initial data set. According to the method, full-automatic data management can be realized in a self-adaptive, self-optimized and self-explored manner.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data governance, in particular to a self-evolution emergent data governance method and system. BACKGROUND

[0002] In the current field of data governance, the following problems generally exist:

[0003] (1) Insufficient adaptive ability of governance rules for multi-source heterogeneous data: Existing data governance systems rely on pre-set static rules when dealing with multi-source heterogeneous data such as relational databases, semi-structured forms, and unstructured texts. These rules can only be imported / modified manually based on experience to adapt to the format, structure, and semantic differences of data sources, resulting in limited effectiveness of data governance.

[0004] (2) Inefficiency of data collaborative governance: Cross-system and cross-department data governance requires manual alignment of standards. For example, the definition of "customer ID" in CRM and ERP systems within an enterprise conflicts. Existing tools lack automatic conflict resolution capabilities, leading to low efficiency of data governance.

[0005] (3) Conflict between static rules and dynamic incremental data: Traditional data governance rules cannot adapt to dynamic changes in data sources, including new data fields and business logic adjustments. This results in governance strategies lagging behind actual needs, affecting the quality of data governance.

[0006] (4) Lack of demand-driven data active exploration capability: Existing governance only repairs "existing data" and cannot predict high-value missing data required for large model training / inference. Data governance is limited to "after-the-fact remediation" and cannot support forward-looking decision-making. SUMMARY

[0007] The purpose of the present application is to provide a self-evolution emergent data governance method and system that can adaptively, self-optimally, and self-exploratively achieve fully automated data governance.

[0008] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0009] In a first aspect, the present application provides a self-evolution emergent data governance method, comprising:

[0010] obtaining a multi-source heterogeneous initial data set;

[0011] inputting the initial data set into a large model and performing the following processing by the large model:

[0012] performing quality assessment and defect identification on the initial data set to obtain a three-element data group; the three-element data group includes the initial data set, an initial quality score of the initial data set, and an initial defect set;

[0013] based on the preset quality score threshold and the data source knowledge graph, the data is supplemented according to the ternary data group in a cyclic self-emergence manner to obtain a complete data set;

[0014] The preset governance rule set is used for data governance on the complete data set to obtain a high-quality data set. Meanwhile, the initial data set and the high-quality data set are used for self-evolution of the governance rule and updating of parameters of the large model through closed-loop feedback to update the preset governance rule set. The updated preset governance rule set is used for data governance on the next initial data set.

[0015] In a second aspect, the application provides a self-evolution emergent data governance system, which comprises a data access module and a data governance large model module. The data governance large model module comprises a data completeness detection unit, a data supplement unit, and a rule evolution and data governance unit.

[0016] The data access module is configured to obtain a multi-source heterogeneous initial data set and input the initial data set into the data governance large model module.

[0017] The data completeness detection unit is configured to perform quality evaluation and defect identification on the initial data set to obtain a ternary data group. The ternary data group comprises the initial data set, an initial quality score of the initial data set, and an initial defect set.

[0018] The data supplement unit is configured to perform cyclic self-emergence data supplement based on a preset quality score threshold and a data source knowledge graph according to the ternary data group to obtain a complete data set.

[0019] The rule evolution and data governance unit is configured to perform data governance on the complete data set by using a preset governance rule set to obtain a high-quality data set. Meanwhile, the initial data set and the high-quality data set are used for self-evolution of the governance rule and updating of parameters of the large model through closed-loop feedback to update the preset governance rule set. The updated preset governance rule set is used for data governance on the next initial data set.

[0020] According to the specific embodiments provided in the application, the application discloses the following technical effects: the application performs quality evaluation and defect identification on the initial data set to obtain an initial quality score and an initial defect set of the initial data set, thereby realizing quantitative evaluation of the initial data set and providing an accurate target for subsequent rule management. Then, the data set is completed through cyclic self-emergence of data, thereby providing high-precision data guarantee for subsequent data management and improving the precision of data management. Finally, the high-quality data set is obtained by performing data management on the complete data set using the preset rule set, and the parameters of the large model are updated through closed-loop feedback to realize updating of the preset rule set, thereby realizing a complete automatic closed loop for rule optimization, breaking the dependence on manual intervention in the traditional method, shortening the management cycle, and improving the data management efficiency due to the realization of fully automatic data management. In addition, the automatic closed loop is realized, which can dynamically adapt to changes in data sources and business requirements, rather than relying on static rules, thereby improving the intelligent level of data management. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained from these drawings without creative labor.

[0022] Figure 1 The flowchart of the self-evolution emergent data management method in an embodiment of the application.

[0023] Figure 2 The flowchart of data completion in an embodiment of the application.

[0024] Figure 3 The structural diagram of a computer device provided in an embodiment of the application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0026] In order to solve the existing data governance problem, the application provides a self-evolution emergent data governance method and system, wherein self-evolution refers to the dynamic updating ability of artificial preset static rules, and more emphasis is placed on the dynamic adaptability, continuous optimization and autonomous learning of the data governance ecology; emergent embodies the demand-driven heuristic active exploration ability, combined with the deep reasoning of the large model, analyzes the historical data governance records and business demand trends, predicts the high-value missing data required for large model training and reasoning, and actively triggers data collection tasks to supplement the data short board. The application aims to solve the core problem of existing data governance, realize a new generation of governance paradigm of self-adaptation, self-optimization and self-exploration, and provide bottom support for enterprise digital transformation and intelligent decision-making.

[0027] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the application will be further described in detail below in combination with the drawings and specific embodiments.

[0028] In one exemplary embodiment, as shown in Figure 1 An self-evolution emergent data governance method is provided, which is executed by a computer device, specifically by a terminal or a server, or by a terminal and a server together, and includes the following steps 101-102.

[0029] Step 101, acquiring a multi-source heterogeneous initial data set. In one specific application, the construction process of the initial data set in step 101 includes the following steps (11)-(12).

[0030] (11) Acquire the original data set from multiple different data sources with different structures or formats; specifically, access the original data set D_{raw} from multiple data sources (ERP, MES, CRM, external website, etc.) inside and outside the enterprise through API interface, message queue and other technical means.

[0031] (12) Perform data formatting processing, encoding unification processing and data security desensitization processing on the original data set to obtain the initial data set, i.e. the standardized initial data set D_{0}.

[0032] Step 102, inputting the initial data set into a large model and performing the following processing by the large model:

[0033] Step 21, quality assessment and defect identification are performed on the initial data set to obtain a three-element data group, so as to realize initial detection and quantitative evaluation of data completeness; the three-element data group comprises the initial data set, an initial quality score of the initial data set and an initial defect set; wherein a data quality evaluation function for realizing quality assessment quantifies data quality by weighted comprehensive evaluation of a plurality of core quality dimensions, and a calculation formula thereof is:

[0034] Q(D) = w comp ·M comp (D) + w cons ·M cons (D) + w acc ·M acc (D).

[0035] Wherein Q(D) is an initial quality score, w comp , w cons , w acc are weight coefficients of completeness, consistency and accuracy respectively, and w comp +w cons +w acc =1; M comp (D) is a completeness measurement function of the initial data set D, M cons (D) is a consistency measurement function of the initial data set D, and M acc (D) is an accuracy measurement function of the initial data set D. In addition, this function is executed by a computer, inputs a data set D, and outputs a scalar between 0 and 1, which is used to quantify the completeness, consistency and accuracy of the data set, and the higher the score is, the better the quality is.

[0036] A defect identification function for realizing defect identification is responsible for identifying and locating specific defects in data, and its output is a defect set, and its function formula is:

[0037] Detect(D) = Δ miss ∪Δ outlier ∪Δ conflict .

[0038] Wherein Detect(D) is an initial defect set, Δ miss is a missing value defect set, Δ outlier is an outlier defect set, and Δ conflict is a logic or semantic conflict defect set. This function is used to identify all data quality problems (such as missing values, outliers, semantic conflicts) in the data set, and return the set of these problems.

[0039] Through the above calculation, the initial quality score of the initial data set and the initial defect set are obtained, the baseline measurement of data quality is completed, and a technical foundation of the data completeness detection and quality evaluation method based on large model deep reasoning is formed.

[0040] Step 22, based on the preset quality score threshold and the data source knowledge graph, the data is supplemented according to the triple data set to obtain a complete data set through cyclic self-eruption; wherein, through the establishment of the cyclic self-eruption data supplement mechanism, the complete data set reaching the quality target can be obtained.

[0041] Specifically, when the initial iteration number i = 0, a cyclic iteration data supplement process is started, which continues to run until the quality score of the data set reaches or exceeds the preset quality score threshold. In a specific application, as shown in the figure, Figure 2 Step 22 includes the following steps:

[0042] (1) Determine whether the initial quality score exceeds the preset quality score threshold.

[0043] (2) If yes, mark the initial data set as a complete data set.

[0044] (3) If not, based on the data source knowledge graph, match the candidate data source set according to the initial defect set, and generate a supplement task; specifically, based on the information in the initial defect set, use the reasoning ability of the large model to automatically search and match internal (such as SCADA system) and external potential data sources, and generate a supplement task. The specific implementation process is as follows:

[0045] Defect semantic analysis: the large model first performs deep semantic analysis on each defect item in the initial defect set, extracts the type feature F type , business context feature F context and data structure feature F structure of the defect, forms a defect feature vector f defect = [F type , F context , F structure ].

[0046] Data source reasoning matching: maintain and pre-store a data source knowledge graph G source , which contains metadata information, data mode and business domain label of each data source, and the data source feature vector can be obtained therefrom. The large model automatically identifies the most matched candidate data source set by calculating the semantic similarity S sim = cosine (f defect , f source ) between the defect feature vector f defect and the data source feature vector f source .

[0047] Reasoning link construction: for complex data completion needs, a large model constructs a multi-hop reasoning link to determine the conversion path from source data to target missing data through logical reasoning, and generates specific data acquisition and processing strategies, i.e., completion tasks.

[0048] (4) For the completion task, a multi-strategy fusion mechanism is adopted to determine the completion data D patch ; specifically, step (4) includes the following specific steps:

[0049] 1) Based on the historical effective data set D hist , information extraction is performed to obtain historical completion data; specifically, the time evolution law of data is identified from the historical effective data set D hist by a time series pattern recognition algorithm Ptemporal(D hist ); the probability distribution characteristics of the data are analyzed by a statistical distribution matching method Pstatistical(D hist ); the most similar historical data segment to the missing data is found based on a similarity measurement function Similarity(D missing , D hist ), and the completion value is generated by interpolation or extrapolation method to obtain the historical completion data D hist_patch .

[0050] 2) Based on the pre-set industry expert knowledge base KBexpert, information extraction is performed to obtain expert knowledge completion data; specifically, the pre-set industry expert knowledge base KBexpert (such as equipment maintenance rule base, process parameter standard base, etc.) is called, and the expert rule set related to the current defect (i.e., the initial defect set) is identified by a rule matching engine Rmatch(KBexpert, Δi); the reasoning result is derived based on the entity relationship and constraint conditions in the expert rule set by a knowledge graph reasoning mechanism KGreasoning, i.e., the reasoning result is obtained; the confidence score of the reasoning result is evaluated by a confidence evaluation function Cconfidence to ensure the completion quality, thereby obtaining the final expert knowledge completion data D expert_patch . Wherein, Δi is the initial defect set.

[0051] 3) A weighted fusion strategy is adopted to fuse the historical completion data and the expert knowledge completion data to obtain the final completion data; specifically, the results of different completion methods are integrated by adopting a weighted fusion strategy, and the fusion weight is dynamically adjusted through feedback learning of historical completion effect: D patch = α·D hist_patch + β·D expert_patch , α + β = 1, both are weight coefficients, and the weight coefficients are determined according to the historical accuracy of each strategy and the current data characteristics.

[0052] In practical applications, in addition to the model strategies based on pattern matching and expert knowledge base completion mentioned above, the following methods can also be used: simulate the distribution of missing data using a generator to complete the missing values; infer the reasonable range of missing values ​​based on probabilistic graphical models; and complete the data completion through distributed collaboration in scenarios where privacy is limited for multi-source data. Relevant technical personnel can also select and adjust these methods as needed.

[0053] (5) Perform a union operation on the supplemented data and the initial dataset to obtain a new dataset.

[0054] (6) Perform quality assessment and defect identification on the new dataset to obtain a new set of three data elements; in practical applications, it is generally necessary to update the iteration count in this step, such as letting i = i + 1, and the obtained new set of three data elements is prepared for the next iteration.

[0055] (7) Update the three-data set to the new three-data set, and then return to the step of determining whether the initial quality score exceeds the preset quality score threshold.

[0056] Thus, through the aforementioned closed-loop process, this application constitutes a cyclical, self-emergent data completion closed-loop mechanism.

[0057] Step 23: Perform data governance on the complete dataset using a preset governance rule set to obtain a high-quality dataset D_{governed}. Simultaneously, based on the initial dataset and the high-quality dataset, perform self-evolution of the governance rules and update the parameters of the large model through closed-loop feedback to update the preset governance rule set. The updated preset governance rule set is used to participate in the data governance of the next initial dataset.

[0058] Specifically, based on the complete dataset obtained above, an adaptive data governance process is implemented through dynamic rules to obtain a feedback sample set and a current rule set; then, based on the feedback sample set and the current rule set, the governance rules achieve self-evolution and closed-loop feedback. In the first application, the preset governance rule set is a preset industry basic rule set; in subsequent applications, the preset governance rule set is the preset governance rule set updated in the previous application. Furthermore, the preset governance rule set R... k ={r1,r2,...,r n}) represents a series of data governance rules (such as deduplication, cleaning, and transformation rules) adopted in the k-th iteration.

[0059] In a specific application, the rule-based self-evolution operator E is invoked based on the feedback sample set S. k For the preset governance rule set R k Optimize and generate a new generation rule set R k+1 =E(R)k S k ), corresponding to step 23, comprising the following steps:

[0060] (1) Using all rules in the preset governance rule set, performing a governance operation on the complete data set to obtain a feedback sample set; specifically, according to the rules in the preset governance rule set, automatically performing a series of governance operations such as deduplication, cleaning, conversion, and standardization on the complete data set. In the processing process, record the successful and failed governance cases to form positive and negative feedback samples, that is, the positive feedback samples correspond to the success of the governance operation, and the negative feedback samples correspond to the failure of the governance operation. The positive feedback sample set and the negative feedback sample set constitute the feedback sample set, which is used for model evolution.

[0061] (2) Feature extraction is performed on the samples in the feedback sample set to obtain a feature vector; specifically, the feedback sample set S k is deeply analyzed and decomposed into a positive feedback sample set S k (governance success cases) and a negative feedback sample set S k (governance failure cases). For each sample s i ∈ S k , the system extracts its feature vector f = [f data , f rule , f context , f result ], f data is a data feature, f rule is an applied rule feature, f context is a business context feature, and f result is a governance effect feature, which includes quality improvement degree, processing efficiency, and computing cost.

[0062] (3) Effect evaluation is performed based on the feature vector to obtain an effect evaluation score of each sample; the calculation formula of the effect evaluation score is as follows:

[0063]

[0064] wherein Score(s i ) is the effect evaluation score of sample s i , e is the natural base (Euler's number), Q improve is the quality improvement degree, T efficiency is the processing efficiency, C cost is the computing cost, and a, b, g are weight parameters.

[0065] (4) According to the effect evaluation scores of all samples, calculate the corresponding fitness of each rule in the preset governance rule set, and based on the corresponding fitness of each rule, update the rule weight of each rule using the gradient descent method.

[0066] The calculation formula of fitness is:

[0067]

[0068] wherein Fitness(r j ) is the fitness of rule r j , S k + (r j ) and S k - (r j ) are the positive feedback sample set and the negative feedback sample set applied to rule r j , λ is the penalty coefficient, and S k is the feedback sample set.

[0069] The calculation formula of the rule weight of each rule is:

[0070]

[0071] wherein w j (k+1) is the weight of rule r j in the k+1th iteration, w j (k) is the weight of rule r j in the kth iteration, and η is the learning rate.

[0072] (5) According to the preset governance rule set and the rule weight of each rule, a governance operation is performed on the complete data set to obtain a high-quality data set.

[0073] (6) Based on the feature vector and the rule weight of each rule, the rules in the preset governance rule set are subjected to evolutionary processing to obtain a new-to-use governance rule set. Specifically, a multi-strategy parallel evolutionary algorithm is used to generate a new-to-use governance rule:

[0074] Rule mutation: based on negative sample analysis, the common features of the failed rules are identified in combination with rule features, and a mutation rule is generated through a probability model Pmutate(r new | r old , S k -). The mutation probability is determined by the failure frequency: Pmut = 1-e-μ·Fail_Rate(r), μ is a mutation strength parameter, and Fail_Rate(r) is the failure frequency of rule r.

[0075] Rule crossover fusion: for a rule pair (r i , r j ) with high fitness, a new rule r new= Coverage(r i , j , θ), θ is the fusion weight determined by the complementarity of the two rules.

[0076] Rule autonomous generation: the large model directly generates new governance rules based on historical success patterns through a sequence generation model P(r new | Context, S k +), and the generation probability is calculated based on the context similarity and historical success rate.

[0077] (7) According to the initial data set and the high-quality data set, the parameters of the large model are updated to realize the update of the selection conditions of the governance rules.

[0078] Specifically, the initial data set is input into the large model for incremental learning, and the model parameter update adopts the Adam optimizer:

[0079]

[0080] wherein, and are the first and second moment estimates of the gradient, ∈ is a numerical stability parameter, and θ is the parameter of the large model. The learning convergence judgment is based on the performance improvement threshold: | Performance (k + 1) - Performance (k) | < δ.

[0081] (8) Based on the large model after parameter update and the new governance rule set to be used, data governance is performed on the initial data set, and the fitness of the new governance rules in the new governance rule set to be used is calculated.

[0082] (9) According to the fitness of the new governance rules, a new generation of governance rule set is determined by using a multi-objective optimization method; the new generation of governance rule set is used as the updated preset governance rule set to participate in the data governance of the next initial data set. Specifically, the evolved rules (i.e., new governance rules) are sorted according to the fitness, and the optimal rule subset is selected by using a multi-objective optimization method, and the optimization objective function corresponding to the multi-objective optimization method is:

[0083]

[0084] wherein, R k+1 is the new generation of governance rule set, Fitness (r) is the fitness of the rule r, Coverage (r) is the coverage of the rule r, φ is the complexity penalty coefficient, and | R k+1 | is the size of the new generation of governance rule set.

[0085] So far, the application realizes adaptive data governance rule generation based on large model deep reasoning. In addition to obtaining a high-quality data element set, i.e., a high-quality data set, which can be used by an application system, an evolved new governance rule set is obtained to participate in the next data governance.

[0086] Compared with the prior art, the application has the following advantages:

[0087] (1) The application realizes adaptive data governance rule generation based on large model deep reasoning, which involves dynamically generating and self-evolving data governance rules based on large model deep reasoning, can adapt to industry and enterprise characteristics, and continuously optimize; the application realizes a data completeness detection and quality evaluation method based on large model deep reasoning, which can dynamically generate evaluation indexes and identify various data defects by using a large model to detect and evaluate the completeness of multi-source heterogeneous data; the application constructs a cyclic self-emergence data completion closed-loop mechanism, which realizes a data self-improvement and continuous emergence completion mechanism through cyclic iteration, can actively discover defects, intelligently locate data sources, and efficiently complete them combined with historical data and expert knowledge base.

[0088] (2) The application automatically completes data, automatically executes governance, and automatically optimizes rules, forming a complete automatic closed loop, which eliminates the dependence on manual intervention of traditional methods, shortens the governance cycle, realizes full-automatic data governance capability, and improves data governance efficiency.

[0089] (3) The application provides accurate targets for governance through quality evaluation and defect identification, adopts a cyclic completion mechanism, and ensures the accuracy of data repair through multi-strategy fusion and iterative verification, which is much better than traditional simple interpolation methods. The synergistic use of the above two can achieve high-precision data governance guarantee and improve the precision of data governance.

[0090] (4) The application continuously optimizes the preset governance rule set according to the actual governance effect (i.e., feedback samples) through the rule self-evolution operator, thereby dynamically adapting to changes in data sources and business requirements, rather than relying on static rules, thus achieving dynamic adaptation capability and improving intelligent level.

[0091] (5) The determination of the quantitative indexes and feedback sample set obtained through quality evaluation and defect identification makes the entire governance process and effect transparent, measurable, and traceable, providing a data foundation for continuous optimization, i.e., achieving explainability and sustainability, and traceable governance effect.

[0092] Based on the same inventive concept, the embodiments of the present application also provide a system for implementing the above-mentioned method. The system provides a solution to the implementation similar to the implementation described in the above-mentioned method. Therefore, the specific limitations in one or more self-evolution emergent data governance system embodiments provided below can refer to the limitations of the method described above, and will not be repeated here.

[0093] In an exemplary embodiment, a self-evolution emergent data governance system is provided, which includes a data access module and a data governance large model module; the data governance large model module includes a data completeness detection unit, a data completion unit, and a rule evolution and data governance unit.

[0094] The data access module is configured to obtain a multi-source heterogeneous initial data set and then input the initial data set into the data governance large model module.

[0095] The data completeness detection unit is configured to perform quality evaluation and defect identification on the initial data set to obtain a three-element data group; the three-element data group includes the initial data set, an initial quality score of the initial data set, and an initial defect set.

[0096] The data completion unit is configured to perform cyclic self-emergent data completion based on a preset quality score threshold and a data source knowledge graph according to the three-element data group to obtain a complete data set.

[0097] The rule evolution and data governance unit is configured to perform data governance on the complete data set using a preset governance rule set to obtain a high-quality data set, and simultaneously perform self-evolution of the governance rules and update the parameters of the large model through closed-loop feedback based on the initial data set and the high-quality data set to update the preset governance rule set; wherein the updated preset governance rule set is used for data governance of the next initial data set.

[0098] In an exemplary embodiment, a computer device can be a server or a terminal, and its internal structure diagram can be as shown in Figure 3As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to implement the self-evolution emergent data governance method.

[0099] Those skilled in the art can understand that, Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0100] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the above method embodiments.

[0101] In one exemplary embodiment, a computer readable storage medium is provided, storing a computer program, which is executed by a processor to implement the steps in each of the above method embodiments.

[0102] In one exemplary embodiment, a computer program product is provided, including a computer program, which is executed by a processor to implement the steps in each of the above method embodiments.

[0103] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0105] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0106] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0107] The principles and implementation modes of the present application are described by applying specific examples in the present application. The above-mentioned embodiments are only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. A self-evolving emergent data governance method, characterized in that, The method includes: Obtain a multi-source heterogeneous initial dataset; The initial dataset is input into the large model, which then performs the following processing: The initial dataset is subjected to quality assessment and defect identification to obtain a three-dimensional data set; the three-dimensional data set includes the initial dataset, the initial quality score of the initial dataset, and the initial defect set; Based on the preset quality scoring threshold and the data source knowledge graph, the data is supplemented cyclically and spontaneously according to the three data sets to obtain a complete dataset. The complete dataset is processed using a pre-defined set of governance rules to obtain a high-quality dataset. Simultaneously, based on the initial dataset and the high-quality dataset, the governance rules are self-evolved and the parameters of the large model are updated through closed-loop feedback to update the pre-defined set of governance rules. The updated set of governance rules is then used to participate in the data governance of the next initial dataset.

2. The self-evolving emergent data governance method according to claim 1, characterized in that, The process of constructing the initial dataset includes: Obtain raw datasets from multiple different data sources with different structures or formats; The original dataset is subjected to data formatting, encoding standardization, and data security desensitization to obtain the initial dataset.

3. The self-evolving emergent data governance method according to claim 1, characterized in that, The data quality assessment function used to implement quality assessment is: Q(D)=w comp ·M comp (D)+w cons ·M cons (D)+w acc ·M acc (D); Where Q(D) is the initial quality score, w comp w cons w acc These are the weighting coefficients for completeness, consistency, and accuracy, respectively; M comp (D) is the completeness measure function for the initial dataset D, M cons (D) is the consistency measure function for the initial dataset D, M acc (D) is the accuracy measure function for the initial dataset D; The defect identification function used to implement defect identification is: Detect(D)=Δ miss ∪Δ outlier ∪Δ conflict ; Where Detect(D) is the initial defect set, Δ miss For the missing value defect set, Δ outlier For the outlier defect set, Δ conflict This is a set of logical or semantic conflict defects.

4. The self-evolving emergent data governance method according to claim 1, characterized in that, Based on a preset quality scoring threshold and a knowledge graph of the data source, and according to the three-dimensional data set, cyclical emergent data completion is performed to obtain a complete dataset, including: Determine whether the initial quality score exceeds a preset quality score threshold; If so, the initial dataset is marked as a complete dataset; If not, then based on the data source knowledge graph, match the candidate data source set according to the initial defect set, and generate a completion task; For the completion task, a multi-strategy fusion mechanism is adopted to determine the completion data; The completed data and the initial dataset are combined to obtain a new dataset; The new dataset is subjected to quality assessment and defect identification to obtain a new three-dimensional data set; The three-dimensional data set is updated to the new three-dimensional data set, and then the process returns to the step of determining whether the initial quality score exceeds the preset quality score threshold.

5. The self-evolving emergent data governance method according to claim 1, characterized in that, The complete dataset is processed using a pre-defined set of governance rules to obtain a high-quality dataset. Simultaneously, based on the initial dataset and the high-quality dataset, the governance rules undergo self-evolution, and the parameters of the large model are updated through closed-loop feedback to achieve updates to the pre-defined set of governance rules. This includes: The complete dataset is governed using all rules from a pre-defined governance rule set to obtain a feedback sample set. Feature extraction is performed on the samples in the feedback sample set to obtain feature vectors; The effect is evaluated based on the feature vector to obtain the effect evaluation score for each sample; Based on the performance evaluation scores of all samples, the fitness of each rule in the preset governance rule set is calculated, and the rule weight of each rule is updated using gradient descent based on the fitness of each rule. Based on the preset governance rule set and the rule weight of each rule, the complete dataset is governed to obtain a high-quality dataset; Based on the feature vector and the rule weight of each rule, the rules in the preset governance rule set are subjected to evolutionary processing to obtain a new governance rule set to be used; Based on the initial dataset and the high-quality dataset, the parameters of the large model are updated to update the selection criteria for governance rules. Based on the updated large model and the set of new governance rules to be used, data governance is performed on the initial dataset, and the fitness of the new governance rules in the set of new governance rules to be used is calculated. Based on the fitness of the new governance rules, a multi-objective optimization method is used to determine the next generation of governance rule set; the next generation of governance rule set serves as the updated preset governance rule set and participates in the data governance of the next initial dataset.

6. The self-evolving emergent data governance method according to claim 5, characterized in that, The feature vector includes data features, rule features, business context features, and governance effect features; the governance effect features include quality improvement, processing efficiency, and computational cost. The formula for calculating the effectiveness evaluation score is as follows: Among them, Score(s i ) is the sample s i The performance evaluation score, where e is the natural base, Q improve It refers to the degree of quality improvement, T efficiency It's about processing efficiency, C cost α represents the computational cost, and β and γ are the weighting parameters.

7. The self-evolving emergent data governance method according to claim 6, characterized in that, The feedback sample set includes a positive feedback sample set and a negative feedback sample set; the positive feedback sample corresponds to a successful governance operation, and the negative feedback sample corresponds to a failed governance operation. The formula for calculating fitness is: Among them, Fitness(r j ) is rule r j fitness, S k + (r j ) and S k - (r j ) represent the application rules r j The positive feedback sample set and the negative feedback sample set, where λ is the penalty coefficient, S k For the feedback sample set; The formula for calculating the rule weight of each rule is as follows: Among them, w j (k+1) For rule r in the (k+1)th iteration j The weight, w j (k) For rule r in the k-th iteration j The weights are η, where η is the learning rate.

8. The self-evolving emergent data governance method according to claim 5, characterized in that, The objective function corresponding to the multi-objective optimization method is: Among them, R k+1 For the next generation of governance rule set, Fitness(r) is the fitness of rule r, Coverage(r) is the coverage of rule r, φ is the complexity penalty coefficient, and |R| is the coverage of rule r. k+1 | represents the size of the next-generation governance rule set.

9. The self-evolving emergent data governance method according to claim 4, characterized in that, A multi-strategy fusion mechanism is adopted to determine the missing data, including: Information is extracted from historical valid datasets to obtain historical supplementary data; Information is extracted based on a pre-set industry expert knowledge base to obtain expert knowledge supplementary data; A weighted fusion strategy is adopted to fuse the historical completion data and the expert knowledge completion data to obtain the final completion data.

10. A self-evolving emergent data governance system, characterized in that, The system includes a data access module and a large data governance model module; the large data governance model module includes a data completeness detection unit, a data completion unit, and a rule evolution and data governance unit; The data access module is used to: acquire a multi-source heterogeneous initial dataset and then input it into the data governance large model module; The data completeness detection unit is used to: perform quality assessment and defect identification on the initial dataset to obtain a three-dimensional data set; the three-dimensional data set includes the initial dataset, the initial quality score of the initial dataset, and the initial defect set; The data completion unit is used to: perform cyclical self-emergent data completion based on a preset quality score threshold and a data source knowledge graph, according to the three-element data set, to obtain a complete dataset; The rule evolution and data governance unit is used to: perform data governance on the complete dataset using a preset governance rule set to obtain a high-quality dataset; and simultaneously, based on the initial dataset and the high-quality dataset, perform self-evolution of the governance rules and update the parameters of the large model through closed-loop feedback to update the preset governance rule set; wherein, the updated preset governance rule set is used to participate in the data governance of the next initial dataset.

Images