Data blood relationship-based field-level data governance method and system

Abstract

The invention provides a field-level data governance method and system based on a data blood relationship, and relates to the technical field of data processing, and the method comprises the steps: obtaining upstream data and downstream data of each field in to-be-processed data; taking the upstream data and the downstream data as sample data to construct a data blood relationship map; determining the upstream and downstream blood relationship among the fields; determining a basic credibility score, a timeliness weight score and a rule verification score of each field; constructing a credibility model; inputting the sample data into a credibility model for training; acquiring real-time sample data of the to-be-processed data; inputting real-time sample data into the trained credibility model, and outputting final credibility; performing data treatment on the to-be-processed data according to the final credibility; and marking a data treatment result as sample data, and inputting the sample data into the credibility model again for training. According to the method, refined management of the data can be realized, and the data quality and availability are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a field-level data governance method and system based on data lineage. Background Art

[0002] The field-level data governance method based on data lineage is a refined data governance strategy. By analyzing the data flow paths and dependencies (i.e., data lineage) between various systems and fields, it can achieve precise control over the quality, credibility, and timeliness of each field. This method ensures the accuracy and integrity of data throughout its life cycle and is particularly suitable for processing complex data flows and multi-source data environments to improve data reliability and availability.

[0003] In today's digital age, the amount of data in various industries is growing exponentially, the data sources are extensive and the data processing process is complex. As a key asset for corporate decision-making, business operations and innovative development, the quality of data directly affects the effectiveness of various data applications. In the data processing process, fields are the smallest component units of data. The accuracy and credibility of field-level data become key factors affecting data value mining and decision-making. Therefore, an innovative field-level data governance method is needed to address the deficiencies of existing data governance technologies at the micro level and ensure the accuracy, completeness, consistency and timeliness of data.

[0004] However, traditional data governance methods often focus on the overall architecture and macro-management of data, lack effective means and strategies for detailed governance at the field level, and are unable to accurately track data sources and data flow paths. When data flows and transforms across systems, it is impossible to ensure the consistency and reliability of data quality from the source to the target field, making it difficult for the system to accurately evaluate the credibility of data at the field level and conduct effective governance. Summary of the invention

[0005] In order to solve the technical problem that traditional data governance methods often focus on the overall architecture and macro-management of data, lack effective means and strategies for detailed governance at the field level, cannot accurately track data sources and data flow paths, and cannot ensure the consistency and reliability of data quality from the source to the target field when data flows and transforms across systems, making it difficult for the system to accurately evaluate the credibility of data at the field level and perform effective governance, the present invention provides a field-level data governance method and system based on data lineage.

[0006] The technical solution provided by the embodiment of the present invention is as follows: First aspect: An embodiment of the present invention provides a field-level data governance method based on data lineage, including: S1: Obtain the upstream data and downstream data of each field in the data to be processed; S2: Using the upstream data and the downstream data as sample data to construct a data lineage map; S3: Determine the upstream and downstream blood relationship between each of the fields according to the data blood relationship map; S4: Based on the upstream and downstream blood relationship, determine the basic credibility score, timeliness weight score and rule verification score of each field; S5: constructing a credibility model according to the basic credibility score, the timeliness weight score and the rule verification score; S6: inputting the sample data into the credibility model for training; S7: Acquire real-time sample data of the data to be processed; S8: inputting the real-time sample data into the trained credibility model and outputting the final credibility; S9: Performing data governance on the data to be processed according to the final credibility, and storing the data governance results in the governance target library; S10: Mark the data governance result as sample data and return to step S6.

[0007] Second aspect: An embodiment of the present invention provides a field-level data governance system based on data lineage, including: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the field-level data governance method based on data lineage as described in the first aspect is implemented.

[0008] The third aspect: An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the field-level data governance method based on data lineage as described in the first aspect is implemented.

[0009] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: In an embodiment of the present invention, by acquiring and analyzing the upstream and downstream data of the fields, a data lineage map is constructed, the data flow path is clarified, and a clear upstream and downstream dependency relationship is provided for each field, so that data governance can be accurately controlled from the source to the target field, which facilitates the identification of potential data quality issues and logical inconsistencies between data. By establishing a credibility model based on basic credibility scores, timeliness weight scores and rule verification scores, the credibility of the fields can be accurately evaluated, and refined data governance can be achieved, thereby improving data quality and availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A flowchart of a field-level data governance method based on data lineage provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a field-level data governance system based on data lineage provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0013] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0014] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0015] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0016] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0017] Reference Manual Attached Figure 1 , showing a flow chart of a field-level data governance method based on data lineage provided by an embodiment of the present invention.

[0018] S1: Obtain the upstream data and downstream data of each field in the data to be processed.

[0019] Among them, a field refers to a column or data item in a database, usually representing a specific type of data; upstream data refers to the data source that affects the current field, usually transmitted from other systems, databases or fields; downstream data refers to the output data of the current field, or data affected by the current field. In the present invention, upstream data and downstream data are historical data used to train models.

[0020] It should be noted that by obtaining the upstream and downstream data of each field, a dependency relationship between the data is established, which enables data governance to be precisely controlled from source to target, helps identify potential problems in data flow, and ensures that the credibility assessment of each field has sufficient contextual information, thereby improving the accuracy and effectiveness of data processing.

[0021] S2: Use upstream data and downstream data as sample data to build a data lineage map.

[0022] Among them, the data lineage map is a visualization tool that describes data flow and dependencies. It tracks the path of data from source to destination and its transformation process.

[0023] It should be noted that by using upstream and downstream data as sample data to construct a data lineage map, the data flow path and dependencies can be intuitively displayed. Through the lineage map, the interaction between data can be more clearly understood, helping to identify potential sources of errors, data quality issues or data consistency issues, thereby improving the accuracy of data governance and making the data management and optimization process more transparent and traceable.

[0024] S3: Determine the upstream and downstream blood relationship between each field based on the data blood relationship map.

[0025] Among them, the upstream and downstream blood relationship describes the dependency between data fields. The upstream field is the data source that affects the current field, and the downstream field is the data affected or derived by the current field.

[0026] It should be noted that by determining the upstream and downstream blood relationships between fields based on the data bloodline map, we can clearly understand the data flow path and interdependencies, discover potential errors or inconsistencies, and perform accurate field-level assessments in the data governance process to ensure the correctness and integrity of the data during processing.

[0027] In a possible implementation, the upstream and downstream blood relationship specifically refers to: the flow path and conversion relationship between upstream data and downstream data.

[0028] S4: Based on the upstream and downstream blood relationship, determine the basic credibility score, timeliness weight score and rule verification score of each field.

[0029] Among them, the basic credibility score is used to evaluate the credibility of the data field, usually based on factors such as the reliability of the data source and the integrity of the field. The timeliness weight score is used to measure the timeliness of the data field. The closer the data is to the generation time, the higher the timeliness score, and vice versa. The rule verification score is verified according to the logical relationship between the fields or the predefined rules. If it meets the rules, points are added, and if it does not meet the rules, points are subtracted.

[0030] It should be noted that by combining upstream and downstream blood relationships, the credibility, timeliness and rule consistency of each field are comprehensively evaluated to ensure data quality. The basic credibility score improves the control over the reliability of the data source, the timeliness weight score ensures the timeliness of the data, and the rule verification score detects the logical consistency between data, thereby avoiding error propagation. This process evaluates each field in a refined manner, enhances the accuracy and effectiveness of data governance, and helps improve the reliability of decision-making.

[0031] In the present invention, the relevance of each upstream data source to the business where the target field is located is scored to obtain a basic credibility score. C r , with a score range of 1 to 100. For example, for the marital status field in the civil affairs business, the data provided by the civil affairs system itself has the highest relevance and can be given 90 to 100 points, while the credibility score of the field value provided by other non-directly related business systems (such as the education business system) is C r The corresponding score is reduced to 50~70 points. The specific score distribution can be set based on the analysis of historical data and the experience of business experts, and can be continuously adjusted and optimized in practice.

[0032] The timeliness weight score is calculated based on the time difference between the data generation time and the current time. t (days), and the validity period of data is set according to business T(days). The closer the data is to the current time, the higher its credibility weight score is. The credibility weight score of older data decreases with the passage of time. The credibility weight score of expired data that exceeds its validity period is C e 10 points (expired data still retains a lower score), timeliness weight score The value range is 0~100 points.

[0033] If the target field has a fixed mathematical calculation relationship or data constraint relationship with several upstream fields, additional evaluation rules, calculation rules, and verification points can be established based on these logical relationships. C a , C a The value range is -50~50 points. For example, in a customer information table, the age field and the date of birth field have a fixed calculation relationship. If the difference between the age and the age calculated based on the date of birth is within a reasonable range (such as ±1 year), it is considered logically consistent and is given extra credibility points. Otherwise, deduct points , multiple rule check scores are accumulated by addition. By mining the intrinsic connections between these fields, the rationality and credibility of the data can be evaluated more comprehensively.

[0034] In a possible implementation, the timeliness weight score is calculated as follows: ; in, C t represents the timeliness weight score, max represents maximization, t Represents the difference between the time when the data was generated and the current time. T Indicates the validity period of the data. C e Indicates the weight score of expired data, t now Indicates the current moment, t gen Indicates the time when the data was generated.

[0035] For example, if the validity period of business data is 365 days, and the business was generated 100 days ago, the timeliness weight score is: ; S5: Build a credibility model based on the basic credibility score, timeliness weight score and rule verification score.

[0036] It should be noted that the basic credibility score, timeliness weight score and rule verification score are integrated to build a credibility model to automatically evaluate the quality of data. By integrating evaluations in multiple dimensions, the model can more accurately calculate the credibility of data fields and effectively identify potential errors or inconsistencies. The construction of this model optimizes the data governance process, improves the efficiency and accuracy of data management, reduces manual intervention, and ensures the high quality and reliability of data.

[0037] In a possible implementation, the hidden layer activation function of the credibility model is specifically: ; Among them, ReLU () represents the ReLU activation function, max represents maximization, when When , the output is 0, when When x .

[0038] The output layer activation function of the credibility model is specifically: ; Among them, Sigmoid () represents the Sigmoid activation function, e represents the base of natural logarithms, x Represents the input variables of the output layer.

[0039] In actual business scenarios, the relationship between data is often very complex and is affected by the interaction of multiple factors. It is difficult to use simple rules to fully and accurately describe and judge the credibility of data. Neural network algorithms can automatically discover the hidden complex patterns and rules between data features and credibility by learning from historical data, such as the trend of data reliability changes in different business scenarios, the comprehensive impact of nonlinear associations between multiple related fields on the credibility of the target field, etc. When building the model, the above three dimensions (basic credibility score, C r , Timeliness Weight Rating C t , rule verification points C a ) respectively set the corresponding weight coefficients W r , W t and W a , calculate the final credibility C f The upstream data of the same destination field is the same set of samples, and during the training process, there is only one positive sample in the same set of samples. C f The value is 1, and all other samples in the same group are negative samples. Cf The value is 0. Different weight systems are calculated in different businesses through the algorithm model, so as to calculate the credibility more accurately, making up for the limitations of traditional rule methods in dealing with complex data relationships.

[0040] The credibility model is a neural network model built based on TensorFlow. The input layer consists of three neurons, corresponding to the basic credibility score. C r , Timeliness Weight Rating C t , rule verification points C a The hidden layer adopts a multi-layer fully connected structure with three layers. The number of neurons in each layer decreases successively. The first layer has 64 neurons, the second layer has 32 neurons, and the third layer has 16 neurons. ReLU (Rectified Linear Unit) is used as the activation function to introduce nonlinear factors and enhance the expression ability of the model. The output layer has only one neuron, and the final credibility of the output prediction is C f , the activation function uses the Sigmoid function to map the output value to the range of 0~1.

[0041] S6: Input the sample data into the credibility model for training.

[0042] It should be noted that by inputting sample data into the credibility model for training, the model can learn how to evaluate the credibility of the field based on actual data. The training process enables the model to automatically identify patterns and regularities in the data, so that it can perform credibility assessments more accurately when faced with new data. This process improves the degree of automation in data processing, reduces manual intervention, improves the efficiency and consistency of data governance, and enhances the model's adaptability in various scenarios.

[0043] In a possible implementation manner, S6 specifically includes: The sample data is input into the credibility model for training until the loss function value is less than the preset loss function value, and then the training is stopped.

[0044] It should be noted that those skilled in the art can set the size of the preset loss function value according to actual needs, and the present invention is not limited here.

[0045] Specifically, by continuously training the credibility model until the loss function value drops to the preset standard, it is ensured that the model can achieve optimal performance when evaluating data credibility, enabling the model to accurately capture data characteristics and rules and improve its ability to process new data. At the same time, the optimization process effectively avoids overfitting, ensures the generalization ability of the model in different scenarios, and improves the accuracy and reliability of data governance.

[0046] In a possible implementation, the loss function value is calculated as follows: ; in, MSE represents the loss function value, y i Indicates i The true value of the sample data, Indicates i The predicted value of sample data, , m Indicates the total number of sample data.

[0047] During the model training process, a large number of data samples are collected to ensure that various business scenarios and data changes are covered. The collected upstream and downstream field data are cleaned, and standardized conversions are performed according to the differences in data formats of different business systems. For example, the date format is unified as "YYYY-MM-DD", and decimal places are retained for numeric fields according to a unified precision. The corresponding basic credibility score, timeliness weight score and rule verification score are calculated for the sample data according to the calculation formulas of the above three dimensions. Business experts manually annotate the highest credible data for the same group of samples based on the actual situation of the business data. The upstream data of the same purpose field are the same group of samples. In the labeling process, only one piece of the highest credible data in the same group of samples is the positive sample, and other sample data are retained as negative samples.

[0048] The preprocessed and labeled data is divided into a training set and a test set in a ratio of 8:2. The training set is used for model parameter learning, and the test set is used to evaluate model performance. When training the model, the initial learning rate is set to 0.001, and the stochastic gradient descent optimization algorithm is used to update the model weights. The learning rate determines the step size of the model weight adjustment at each iteration. If it is too large, the model may not converge, and if it is too small, the training speed will be too slow. The actual accurate value of the target field is used in training. C f As the supervisory signal (label), the loss function between the predicted value and the true value is calculated. According to the loss function, the model weight is reversely derived, and the connection weights between the hidden layer and the output layer are adjusted so that the loss function value continues to decrease and the model prediction value gradually approaches the true value. The training process sets an early stopping mechanism to monitor the accuracy index on the test set. If the index does not improve for several consecutive epochs (training rounds), the training is terminated in advance to prevent overfitting. For example, if the test set accuracy does not increase by the threshold of 0.1% for 5 consecutive epochs, the training is stopped.

[0049] In a possible implementation manner, after S6, the method further includes: Determine the accuracy of the trained credibility model: ; Among them, Accuracy represents the accuracy rate, TP Indicates a true example, TN represents a true negative example, FP represents a false positive example, FN represents a false negative example.

[0050] Determine whether the accuracy reaches the preset accuracy. If so, output the trained credibility model. Otherwise, continue to train the credibility model.

[0051] It should be noted that those skilled in the art can set the preset accuracy rate according to actual needs, and the present invention is not limited thereto.

[0052] It should be noted that by evaluating the accuracy of the trained model, we ensure that the model can meet the expected performance standards. If the accuracy does not meet the preset requirements, the system will automatically retrain the model to optimize its performance. This process effectively avoids inefficient or inaccurate models from being put into use, ensuring the reliability of data processing and credibility assessment, while improving the adaptability and accuracy of the model, further improving the effectiveness of data governance.

[0053] S7: Obtain real-time sample data of the data to be processed.

[0054] S8: Input the real-time sample data into the trained credibility model and output the final credibility.

[0055] It should be noted that by inputting real-time sample data into the trained credibility model, the credibility of new data can be quickly evaluated. Through real-time processing, timely quality assessment of data can be ensured after it is generated, which improves the response speed and accuracy of data governance, avoids delayed data processing, and ensures the real-time and efficiency of data.

[0056] In a possible implementation, the final credibility is calculated as follows: ; in, C f Indicates the final credibility, C r represents the basic credibility score, C t represents the timeliness weight score, C a Indicates the rule verification score, W r Represents the weight coefficient corresponding to the basic credibility score, W tIndicates the weight coefficient corresponding to the timeliness weight score, W a Indicates the weight coefficient corresponding to the rule verification score.

[0057] In the present invention, after training and tuning, the dimensional weight coefficients of the model in each business are determined, and then multiple source data of the actual business scenario are brought into the calculation formula of the three dimensions to calculate the corresponding basic credibility score, timeliness weight score and rule verification score respectively, and then the weight coefficient of the corresponding business calculated by the model is input to calculate the final credibility, and the C f The low-confidence data less than the threshold 0.4 is the final confidence of other data greater than the threshold C f Sort and take the original data with the maximum value as the final data governance result.

[0058] S9: Perform data governance on the data to be processed based on the final credibility, and store the data governance results in the governance target library.

[0059] Among them, the governance target library refers to the database or storage system used to store the results of data governance. It contains all processed and verified data and usually has higher availability and quality for subsequent analysis, reporting or other operations.

[0060] It should be noted that data governance is performed on the data to be processed based on the final credibility to ensure the quality and consistency of the data. By storing the governance results in the target library, a traceable historical record is provided, which enhances the transparency and auditability of data management. At the same time, it provides a reliable foundation for subsequent analysis and decision-making, and improves the efficiency and accuracy of data governance.

[0061] In the present invention, the calculated final credibility score, scores of each credibility assessment factor, related metadata and other dimensional information are stored in the data governance target library. Fields with high final credibility scores are preferentially selected as data governance results. The governance results and calculation process data are stored at the same time for subsequent query, analysis and audit.

[0062] In a possible implementation manner, after S9, the method further includes: Manually review and correct data governance results.

[0063] It should be noted that by manually reviewing and correcting data governance results, ensuring that data that may be missed or erroneous in the automated processing process is corrected in a timely manner, it is possible to improve data quality, reduce errors in the model or automation process, ensure the accuracy and consistency of data governance results, and enhance the reliability and adaptability of data governance.

[0064] In the present invention, a visual manual correction interface is provided to allow data administrators or domain experts to manually review and correct data governance results. When logical errors are found in the automatically calculated credibility scores or data values, or deviations are found during actual use, manual modifications can be made on the interface, and the reasons and basis for the modifications are recorded, which is convenient for later adjustment of auxiliary verification rules and retraining. The manually corrected data is processed according to the data set format and fed back as new sample data to the neural network algorithm model for retraining, so that the algorithm can learn the experience and knowledge of manual correction, continuously optimize the credibility calculation model, and improve the intelligence and accuracy of the data governance method.

[0065] S10: Mark the data governance results as sample data and return to step S6.

[0066] It should be noted that by marking the data governance results as sample data and returning them to model training, a closed-loop feedback mechanism is formed. By continuously optimizing the training model, the accuracy and adaptability of the model can be improved, ensuring that the governance effect can be continuously improved when processing new data, and enhancing the system's self-learning and self-optimization capabilities.

[0067] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: In an embodiment of the present invention, by acquiring and analyzing the upstream and downstream data of the fields, a data lineage map is constructed, the data flow path is clarified, and a clear upstream and downstream dependency relationship is provided for each field, so that data governance can be accurately controlled from the source to the target field, which facilitates the identification of potential data quality issues and logical inconsistencies between data. By establishing a credibility model based on basic credibility scores, timeliness weight scores and rule verification scores, the credibility of the fields can be accurately evaluated, and refined data governance can be achieved, thereby improving data quality and availability.

[0068] Reference Manual Attached Figure 2 , showing a structural diagram of a field-level data governance system based on data lineage provided by the present invention.

[0069] The present invention also provides a field-level data governance system 20 based on data lineage, which is applied to the above-mentioned field-level data governance method based on data lineage, and includes: Processor 201.

[0070] The memory 202 stores computer-readable instructions, and when the computer-readable instructions are executed by the processor 201, a field-level data governance method based on data lineage as in the method embodiment is implemented.

[0071] The field-level data governance system 20 based on data lineage provided by the present invention can execute the above-mentioned field-level data governance method based on data lineage and achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0072] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: In an embodiment of the present invention, by acquiring and analyzing the upstream and downstream data of the fields, a data lineage map is constructed, the data flow path is clarified, and a clear upstream and downstream dependency relationship is provided for each field, so that data governance can be accurately controlled from the source to the target field, which facilitates the identification of potential data quality issues and logical inconsistencies between data. By establishing a credibility model based on basic credibility scores, timeliness weight scores and rule verification scores, the credibility of the fields can be accurately evaluated, and refined data governance can be achieved, thereby improving data quality and availability.

[0073] It should be understood that the processor in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0074] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0075] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0076] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0077] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0078] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0079] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0081] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0084] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0085] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, a field-level data governance method based on data lineage relationship as in the method embodiment is implemented.

[0086] A computer-readable storage medium provided by the present invention can implement the steps and effects of the field-level data governance method based on data lineage relationship of the above-mentioned method embodiment. To avoid repetition, the present invention will not go into details.

[0087] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: In an embodiment of the present invention, by acquiring and analyzing the upstream and downstream data of the fields, a data lineage map is constructed, the data flow path is clarified, and a clear upstream and downstream dependency relationship is provided for each field, so that data governance can be accurately controlled from the source to the target field, which facilitates the identification of potential data quality issues and logical inconsistencies between data. By establishing a credibility model based on basic credibility scores, timeliness weight scores and rule verification scores, the credibility of the fields can be accurately evaluated, and refined data governance can be achieved, thereby improving data quality and availability.

[0088] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0089] There are a few points to note: (1) The drawings of the embodiments of the present invention only involve structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0090] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0091] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0092] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A field-level data governance method based on data lineage, characterized in that: include: S1: Obtain the upstream data and downstream data of each field in the data to be processed; S2: Using the upstream data and the downstream data as sample data to construct a data lineage map; S3: Determine the upstream and downstream blood relationship between each of the fields according to the data blood relationship map; S4: Based on the upstream and downstream blood relationship, determine the basic credibility score, timeliness weight score and rule verification score of each field; S5: constructing a credibility model according to the basic credibility score, the timeliness weight score and the rule verification score; S6: inputting the sample data into the credibility model for training; S7: Acquire real-time sample data of the data to be processed; S8: inputting the real-time sample data into the trained credibility model and outputting the final credibility; S9: Performing data governance on the data to be processed according to the final credibility, and storing the data governance results in the governance target library; S10: Mark the data governance result as sample data and return to step S6.

2. The field-level data governance method based on data lineage according to claim 1 is characterized in that: The upstream and downstream blood relationship specifically refers to: the flow path and conversion relationship between the upstream data and the downstream data.

3. The field-level data governance method based on data lineage relationship according to claim 1 is characterized in that: The timeliness weight score is calculated as follows: ； in, C t represents the timeliness weight score, max represents maximization, t Represents the difference between the time when the data was generated and the current time. T Indicates the validity period of the data. C e Indicates the weight score of expired data, t now Indicates the current moment, t gen Indicates the time when the data was generated.

4. The field-level data governance method based on data lineage according to claim 1 is characterized in that: The hidden layer activation function of the credibility model is specifically: ； Among them, ReLU () represents the ReLU activation function, max represents maximization, when When , the output is 0, when When x ; The output layer activation function of the credibility model is specifically: ； Among them, Sigmoid () represents the Sigmoid activation function, e represents the base of natural logarithms, x Represents the input variables of the output layer.

5. The field-level data governance method based on data lineage relationship according to claim 1 is characterized in that: The S6 is specifically: The sample data is input into the credibility model for training, and the training is stopped when the loss function value is less than the preset loss function value.

6. The field-level data governance method based on data lineage relationship according to claim 5 is characterized in that: The loss function value is calculated as follows: ； in, MSE represents the loss function value, y i Indicates i The true value of the sample data, Indicates i The predicted value of sample data, , m Indicates the total number of sample data.

7. The field-level data governance method based on data lineage relationship according to claim 1 is characterized in that: After S6, the method further includes: Determine the accuracy of the trained credibility model: ； Among them, Accuracy represents the accuracy rate, TP Indicates a true example, TN represents a true negative example, FP represents a false positive example, FN represents a false negative example; Determine whether the accuracy reaches a preset accuracy; if so, output the trained credibility model; otherwise, continue to train the credibility model.

8. The field-level data governance method based on data lineage relationship according to claim 1 is characterized in that: The final credibility is calculated as follows: ； in, C f Indicates the final credibility, C r represents the basic credibility score, C t represents the timeliness weight score, C a Indicates the rule verification score, W r Represents the weight coefficient corresponding to the basic credibility score, W t Indicates the weight coefficient corresponding to the timeliness weight score, W a Indicates the weight coefficient corresponding to the rule verification score.

9. The field-level data governance method based on data lineage relationship according to claim 1 is characterized in that: After S9, the method further includes: The data governance results are manually reviewed and corrected.

10. A field-level data governance system based on data lineage, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the field-level data governance method based on data lineage as described in any one of claims 1 to 9 is implemented.

Images