Data governance method, system, medium and equipment based on multi-source data

Abstract

A data governance method, system, medium and device based on multi-source data, relating to the field of data governance technology. The method comprises: when a target file to be governed is obtained, matching a user-defined target template corresponding to the target file; parsing the target template to obtain parsed template data; matching the parsed template data with the data columns in the target file to obtain a matching result; filtering the matching result based on a preset filtering rule to obtain a filtering result; and data governing the filtering result to obtain a governance result. Implementing the technical solution provided by the present application can improve the versatility of the template-based data governance solution.

Description

Technical Field

[0001] The present application relates to the field of data governance technology, and specifically to a data governance method, system, medium and device based on multi-source data. Background Art

[0002] With the continuous advancement of enterprise digital transformation, various types of business data are growing explosively. How to effectively integrate and manage a large number of messy data sources has become a problem that many companies need to solve urgently. In response to the various types of massive data generated by different departments and business scenarios, companies generally adopt customized data governance solutions to achieve data structuring, standardization and quality improvement.

[0003] At present, customized data governance solutions based on Excel templates are often used in enterprise data governance. Such solutions generally require the pre-design of fixed Excel templates for different types of data, including but not limited to fixed header formats, etc., or require the writing of fixed code logic to parse and obtain attribute values ​​of different types of data. This model requires independent definition of templates or codes for each data type, which makes the flexibility and scalability of the entire solution weak. The smallest change in the template or code may affect the parsing effect. Therefore, how to improve the versatility of template-based data governance solutions is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a data governance method, system, medium and device based on multi-source data, which can improve the versatility of template-based data governance solutions.

[0005] In a first aspect, the present application provides a data governance method based on multi-source data, the method comprising:

[0006] When a target file to be managed is obtained, a user-defined target template corresponding to the target file is matched;

[0007] Parsing the target template to obtain parsed template data;

[0008] Performing data matching on the parsed template data and the data columns in the target file to obtain a matching result;

[0009] Filter the matching results based on preset filtering rules to obtain filtering results;

[0010] Data governance is performed on the screening results to obtain governance results.

[0011] By adopting the above technical solution and introducing a user-defined template matching mechanism, it is possible to adapt to data of different sources and types, thereby improving the versatility of the data governance solution. After obtaining the target file, the system can match the corresponding target template and parse the target template. The system then uses the parsed template data to match the column attributes and types of the target data to achieve rapid template-based mapping. After matching, rule screening and subsequent customized data governance operations can also be performed. The entire process implements a custom template-driven, configurable multi-source heterogeneous data governance solution. Without changing the template configuration and code logic, there is no need to independently design a parsing mechanism for each data type. At the same time, it supports rule and custom processing, thereby improving the versatility of data governance.

[0012] Optionally, when the target file to be managed is obtained, before matching the user-defined target template corresponding to the target file, it also includes: obtaining the type of target data, and determining an initial table file based on the target data type; obtaining the type of each worksheet in the initial table file, and the type of each data column in each worksheet; and generating a target template based on the type of each worksheet and the type of each data column in each worksheet.

[0013] By adopting the above technical solution, before obtaining the target file to be managed, an initial general table file template that matches the target data can be determined according to the type of the target data. Then the system parses the structure of the template, obtains the user's type definition of each worksheet, and the data type of each column in the worksheet. Based on this type information, the system can generate a template customized for the target data, that is, the type information of each worksheet and column of the target data is directly mapped to the structure of the template, thereby realizing user customization of the template and rapid generation of the template.

[0014] Optionally, parsing the target template to obtain parsed template data includes: parsing the header information of each worksheet in the target template; determining whether there is a special mark in the header information; if there is a special mark in the header information, matching the data column with the special mark according to the type corresponding to the special mark; if there is no special mark in the header information, matching the data column type of each worksheet according to a preset priority rule to obtain the parsed template data.

[0015] By adopting the above technical solution, the header content of the template is parsed to determine whether there is a special mark in the header. If there is a mark, the corresponding column type can be matched directly according to the custom type information in the mark. If there is no mark, the system can preset multiple column type matching rules with different priorities and match the column types in order. Through this technical means of custom parsing with marks and multi-level rule parsing, the column type parsing can be made more accurate and intelligent, which can not only automatically parse user-defined marks, but also reasonably use various auxiliary rules to match data column types.

[0016] Optionally, the preset priority rules are: the priority of the script rule is greater than the priority of the initial value specified by the table, the priority of the initial value specified by the table is greater than the priority of the special marking rule, and the priority of the special marking rule is greater than the priority of the target template configuration.

[0017] By adopting the above technical solution, during parsing, the system matches the column type level by level according to the preset priority rules, that is, first try to parse the target template configuration, and then parse the special tag rule. If the special tag rule has a result, the parsing result of the special tag rule will replace the parsing result of the target template configuration, and then parse the table to specify the initial value. If there is a result, it will also replace the type result obtained in the previous step, and so on. Matching is performed. This multi-level rule cascading mechanism can improve the accuracy of type matching.

[0018] Optionally, the data matching the parsed template data with the data columns in the target file to obtain a matching result includes: data matching the parsed template data with the data in the target file to obtain a matching degree and a data column that fails to match; when the matching degree is lower than a preset value, displaying a prompt message to prompt the user to re-match the data column that fails to match in the target file; when the matching degree is not lower than a preset value, taking the successfully matched data column as the matching result.

[0019] By adopting the above technical solution, the matching degree between the target column and the template column is calculated. For columns with low matching degree, the system can pop up a prompt so that the user can re-manually match according to actual needs. For columns with high matching degree, the system can directly confirm it as a successful match, thereby improving the matching quality and making the matching results more in line with user needs.

[0020] Optionally, the filtering of the matching results based on preset filtering rules, before obtaining the filtering results, also includes: obtaining the user's filtering requirements; generating filtering rules based on the filtering requirements and an abstract syntax tree; filtering the matching results based on the preset filtering rules to obtain filtering results, including: filtering out target data that meets the preset filtering rules in the matching results, and using the target data as the filtering results.

[0021] By adopting the above technical solution, the user's customized filtering requirements for the result set are obtained, and then the user requirements are parsed into structured filtering rules using technologies such as abstract syntax trees, and the target data that meets the preset filtering rules are filtered out as filtering results. The addition of user-defined requirements improves the customization of filtering, and the generation of rules combined with abstract syntax tree technology also makes filtering more intelligent. The final output filtering results are also more in line with user expectations for quality and business scenarios.

[0022] Optionally, performing data governance on the screening results to obtain governance results includes: obtaining user-defined data cleaning rules; and performing governance on the data in the screening results according to the data cleaning rules to obtain governance results.

[0023] By adopting the above technical solutions, we can obtain the cleaning rules written by users according to business needs, make customized requirements for data quality, conversion format, etc., and clean the result data according to these rules, such as identifying erroneous data, filling in missing data, standardizing formats, etc. The introduction of the mechanism of custom rules can significantly improve the data accuracy, completeness and consistency of the result set, making the final output data of higher quality and in line with the user's expectations for the results.

[0024] In a second aspect of the present application, a data governance system based on multi-source data is provided, the system comprising:

[0025] A template matching module is used to match a user-defined target template corresponding to a target file when a target file to be managed is obtained;

[0026] A template parsing module, used to parse the target template and obtain parsed template data;

[0027] A data matching module, used for matching the parsed template data with the data columns in the target file to obtain a matching result;

[0028] A data screening module, used to screen the matching results based on preset screening rules to obtain screening results;

[0029] The data governance module is used to perform data governance on the screening results to obtain governance results.

[0030] In a third aspect of the present application, a computer storage medium is provided, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the above method steps.

[0031] In a fourth aspect of the present application, an electronic device is provided, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0032] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0033] This application introduces a user-defined template matching mechanism that can adapt to data of different sources and types, thereby improving the versatility of data governance solutions. After obtaining the target file, the system can match the corresponding target template and parse the target template. The system then uses the parsed template data to match the column attributes and types of the target data to achieve rapid template-based mapping. After matching, rule screening and filtering can also be performed, as well as subsequent customized data governance operations. The entire process implements a custom template-driven, configurable multi-source heterogeneous data governance solution. Without changing the template configuration and code logic, there is no need to independently design a parsing mechanism for each data type. At the same time, it supports rule and custom processing, thereby improving the versatility of data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of a data governance method based on multi-source data provided in an embodiment of the present application;

[0035] Figure 2 It is a schematic diagram of a data governance system module based on multi-source data provided in an embodiment of the present application;

[0036] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0037] Description of reference numerals: 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0038] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0039] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0040] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0041] The following will provide a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0042] Please refer to Figure 1 , a flowchart of a data governance method based on multi-source data is proposed. The method can be implemented by a computer program, a single-chip microcomputer, or run on a data system based on multi-source data. The computer program can be integrated in an intelligent terminal or run as an independent tool application. Specifically, the method includes steps 10 to 50, and the steps are as follows:

[0043] As an optional embodiment, when a target file to be managed is obtained, before the step of matching a user-defined target template corresponding to the target file, a target template customization process is further included, specifically including the following steps:

[0044] Step 101: Obtain the type of target data, and determine an initial table file based on the type of target data.

[0045] In the scenario of multi-source heterogeneous data governance, in order to achieve efficient template matching, it is necessary to pre-parse the structural information of different types of data and generate the corresponding initial table files, which contain data types, column attributes, etc. The purpose of this is that when the target data that actually needs to be governed is obtained, the corresponding initial table files can be quickly located according to their data types, thereby determining the matching template.

[0046] Specifically, different data types are predefined, such as user data, product data, etc., and a sample data file is determined for each type. Then the structure of the sample file is parsed, and the meta-information therein is extracted, such as the table header, column name, data type, etc., to generate an initial table file that describes this type of data structure, and the correspondence between the data type and the table file is recorded. After the target data is acquired, its data type is determined by parsing its content, comparing features, etc., and then the initial table file that matches the target data type is directly located based on the type-table file correspondence established earlier. Then the corresponding template is loaded to achieve fast and efficient template matching. This table file mapping method based on data types realizes the rapid association of target data with matching templates, improves the efficiency of dynamic template matching, and provides effective support for subsequent parsing and governance processes.

[0047] Step 102: Obtain the type of each worksheet in the initial spreadsheet file and the type of each data column in each worksheet.

[0048] Specifically, a type definition service is provided, which includes common business object types, such as users, products, etc. Users can select appropriate types according to their own data conditions and label worksheets and columns. For example, define a user table, label the table type as "user", and label the columns as attribute types such as "name" and "age". The system parses this user-defined type annotation, generates the Sheet type definition of the worksheet, and the semantic type definition of the column. The generated template contains standardized structured type information. When the target data is template matched, accurate attribute mapping can be performed based on this type information, the source data columns can be matched with the template columns, and converted to standard types to guide subsequent parsing and governance work.

[0049] Step 103: Generate a target template based on the type of each worksheet and the type of each data column in each worksheet.

[0050] Specifically, after obtaining the business type annotation of the worksheet, such as "user table", and the type of each column, such as "name", these type information form the structured metadata of the template. Based on these type definitions, a standard target template is generated, which contains worksheets and columns with clear types. In addition, the target template can also contain configuration information, such as filtering rules, conversion scripts, etc., to guide subsequent governance workflows. The generated target template contains both type-based structural information and extensible configurations such as business rules. When the template is applied to a new data source, attributes can be quickly mapped through type association, and parsed and converted according to the configuration guidance to achieve adaptation to variable data.

[0051] Step 10: When the target file to be managed is obtained, match the user-defined target template corresponding to the target file.

[0052] In the multi-source heterogeneous data governance scenario, the key is to achieve dynamic matching of target data with appropriate templates. The purpose of this is to implement a universal template library to achieve reuse and rapid application of templates based on different target data.

[0053] Specifically, users can design multiple templates in advance according to different data types and features, which can be stored in the template library, and an index mapping from data features to templates can be established. When the target file to be managed is obtained, its basic structure and feature information, such as file type, header name, keywords, etc., are parsed. Then, based on these features, the best matching template is found in the template index mapping. For example, if a template related to "user data" is found, the system will load the definition of the template, including parsing rules, conversion logic, etc. After that, the template can be applied to the target file, and the target data can be converted through parsing and mapping. Templates can also be manually specified. Matching accurate templates can greatly reduce the manual workload of subsequent processing. In addition, the template mechanism provides scalability. When new data appears, as long as the template library is improved, it can be matched and applied, realizing the reuse of general templates.

[0054] Step 20: Parse the target template to obtain the parsed template data.

[0055] Specifically, after template matching, the internal structure of the template needs to be parsed and converted into structured data to guide subsequent mapping operations. The purpose is to obtain structured information such as data types and semantics contained in the template through parsing, and establish a mapping system from source data to target data model. The specific implementation can be: load the matched template file, scan its worksheet name, column name, type information of annotations, etc. Then parse these contents according to the configuration rules to determine the data type, semantic type, and meaning of different cells in the column. The parsing result generates the data structure of the template, which contains the mapping relationship between each column and the data attribute. In this way, the target attribute corresponding to the template column can be found for the source data, and the mapping relationship can be established, which provides a basis for subsequent data conversion and loading. This way of parsing the template to obtain the mapping system realizes the definition of custom data structure based on the template and converts it into structured data available to the system, which is the basis for rapid cleaning of source data.

[0056] Based on the above embodiment, as an optional embodiment, the step of parsing the target template to obtain the parsed template data may further include the following steps:

[0057] Step 201: Parse the header information of each worksheet in the target template.

[0058] Step 202: Determine whether there is a special mark in the header information.

[0059] Specifically, when performing template parsing, it is necessary to focus on parsing the header information and determine whether there are special tags. The purpose of this is to extract additional semantic information in the custom template through the special tags in the header to improve the parsing effect. The specific implementation can be: the system scans the content of the header cell in the template, matches the header text with regular expressions or specific vocabulary, and determines whether it contains special tags. Mark the tagged header, obtain additional information corresponding to the tag, and form a structured definition of the header. In this way, during subsequent mapping, the tagged column can automatically match the type corresponding to the tag, and realize custom mapping by tag, which is not limited to string matching. Columns without tags can also use the default mapping configuration.

[0060] Step 203: If there is a special mark in the header information, the data column with the special mark is matched according to the type corresponding to the special mark.

[0061] After the template is parsed, for columns with special tags in the table header, customized matching is required based on the tag type. The purpose of this is to use the additional semantic information provided by the tag to guide the precise mapping of the column. The specific implementation can be: If the parsing detects that there is a tag in the column header, such as "#type" indicating a category column, then the type information indicated by the tag is extracted, which is "type" here. Look up "type" in the system-predefined tag type dictionary and obtain its mapped target data type as "string". Then it can be directly determined that the type of the column is string type, and the corresponding data column can also be marked as string type. When mapping the subsequent source data column to the target model attribute, the source data column can be accurately mapped to the corresponding string type attribute based on the semantic type information.

[0062] Step 204: If there is no special mark in the header information, the data column types of each worksheet are matched according to a preset priority rule to obtain parsed template data.

[0063] Specifically, after parsing the template header, for columns without special tags, it is necessary to perform level-by-level matching according to the priority preset rules to determine the column type. The purpose of this is to make full use of various meta-information sources, match column types in multiple levels according to reliability, and improve the mapping quality of untagged columns.

[0064] The preset priority rule in the embodiment of the present application can be: the priority of the script rule is greater than the priority of the initial value specified by the table, the priority of the initial value specified by the table is greater than the priority of the special tag rule, and the priority of the special tag rule is greater than the priority of the target template configuration. The table in the embodiment of the present application is an Excel table, and the Excel header supports one or two rows. When it is one row, it is necessary to ensure that the column names in the current sheet are not repeated, and the schema value is empty. When it is two rows, it is necessary to ensure that the combined column names of the first and second rows in the current sheet are not repeated. It should be noted that when the sheet name and the content of the first row in the multi-row header mode contain the content of the {CLSNAME} structure, the (CLSNAME) structure or the [CLSNAME] structure, then CLSNAME will be determined as the current hierarchical type SheetCls or SchemaCls. Among them, SheetCls refers to the type name corresponding to each worksheet (Sheet) in the Excel template. SchemaCls refers to the content of the first row of the Excel template header. If it contains a special tag (for example, {Name} represents the name column), it can be parsed as the attribute name of the target data type. By setting the object type assignment priority, you can specify the type of the same field when mapping the Excel template to the target data type. That is to say, when multiple sources have different assignments for the field type, they will be determined according to the above priority.

[0065] Exemplarily, if there is no special mark in the header information, when matching the data column type of each worksheet, the system first parses the target template configuration. If the target template configuration can determine the column type, the parsing result of the target template configuration is obtained, and the special tag rules are continued to be parsed, where SchemaCls has a higher priority than SheetCls. The special tag rules are parsed, that is, whether there are special marks for the columns in the template. If the special tag rule is parsed and there is a result, the parsing result of the special tag rule replaces the parsing result of the target template configuration, and then the initial value specified by the table is parsed. The initial value specified by the table may refer to the initial value filled in the Excel type column cell. If the initial value specified by the table is parsed and there is a result, the table-specified initial value parsing result replaces the parsing result of the above-mentioned special mark. Finally, the script rules are parsed. If the script rule is parsed and there is a result, the script parsing result replaces the parsing result of the initial value specified by the table. And so on, the parsing results with higher priority will gradually replace the parsing results with lower priority. In this way, the matching results of various information sources can be integrated and matched according to reliability, taking into account customized script parsing and alternative multi-level default matching rules, thus achieving precise type definition of untagged columns. This multi-dimensional matching mechanism greatly improves the comprehensiveness and accuracy of column type parsing.

[0066] Step 30: Match the parsed template data with the data columns in the target file to obtain a matching result.

[0067] Specifically, first parse to obtain structural information such as the data type and semantic type of each column in the template. Then traverse each column of the target Excel file, extract the name, data characteristics, etc. of the column, match them with the template column information, and find the most similar template column. For example, the target column "age" matches the column with the type "Integer" and the semantics "Age" in the template. Repeat the matching process and finally establish a mapping relationship between each target column and the corresponding column in the template. If some columns fail to match, the user can be prompted to perform manual matching. After the matching is completed, the target data attributes corresponding to the source data column can be obtained according to the mapping relationship.

[0068] Based on the above embodiment, as an optional embodiment, the step of performing data matching between the parsed template data and the data column in the target file to obtain the matching result may further include the following steps:

[0069] Step 301: Match the parsed template data with the data in the target file to obtain the matching degree and the data column of the failed match.

[0070] Specifically, we can first use the string similarity algorithm to match based on the column name keywords to obtain a certain degree of matching. Then, for columns with special tags, we can accurately match them according to the tag type to achieve a higher degree of matching. Finally, based on semantic technology, we can determine whether the business meanings of the columns are similar, which can also improve the matching degree. The system will combine all matching results and calculate the summary matching degree of each column. If the matching degree exceeds the preset value, it is determined as a successful matching column, otherwise it is determined as a failed column, and all data columns that failed to match are obtained.

[0071] Step 302: When the matching degree is lower than a preset value, a prompt message is displayed to prompt the user to re-match the data columns in the target file that failed to match.

[0072] Step 303: When the matching degree is not lower than the preset value, the successfully matched data column is taken as the matching result.

[0073] Specifically, a matching degree threshold can be set in advance, for example, a match degree below 60% is considered a low match. If the matching degree between a target column and all template columns is relatively low, the column is judged to have failed to match. The system will pop up a prompt, displaying the name of the column that failed to match, and at the same time provide the current best matching column and matching degree for the user's reference. The user can reselect a template column that is more consistent with the column for matching based on the prompt and actual needs. This can avoid some errors in automatic matching, and can also match some special columns that are difficult for the system to match. When the matching degree between a target column and a template column is higher than this threshold, the system can directly determine that the match is successful, and directly write the corresponding relationship between this target column and its best matching template column into the matching result set. After the matching rules of all columns are executed, the final matching result set contains all columns with high matching degrees.

[0074] Step 40: Filter the matching results based on the preset filtering rules to obtain the filtering results.

[0075] Specifically, in order to refine high-quality target data from the preliminary column matching results, a screening mechanism needs to be established. Specifically, the system will first open an interface to obtain the user's customized screening requirements for the result set, which covers the user's personalized requirements for the final result. Then the system uses technologies such as abstract syntax trees to parse these requirements and convert them into structured screening rule expressions for program judgment and execution. In addition, the system also presets a series of general screening rule templates, which contain various basic data quality verification logics, which is an effective supplement to user rules. The system merges the two into a unified screening rule set, and scans and filters the matching result set based on this benchmark. Only the result data that meets both the user's customized rules and the general rules will be retained, thereby generating a high-quality screening result set that has been verified by multiple rules as the output of the module. This screening mechanism that combines customized requirements with general templates can effectively improve the quality and value of the result set, and better meet the user's expected requirements for the final output.

[0076] Step 50: Perform data governance on the screening results to obtain governance results.

[0077] Specifically, after obtaining a highly targeted screening result set, in order to further improve the data quality of the result set, it is necessary to introduce a data governance process for cleaning and conversion. Specifically, the system provides an interface for users to customize data cleaning rules, which stipulates the standardized format of data, verification logic and other requirements. These rules combine the user's expectations for the quality of the data result set. The system loads the scripts of these cleaning rules and parses them into executable code modules. Then the system traverses each data item in the screening result set according to the code logic, and performs verification and conversion operations on it. For example, identify and correct formatted data, fill in missing values, and standardize column values ​​to standard forms. The result data after being governed by custom rules has a unified format and higher accuracy. Finally, high-quality data governance results are output to complete the entire multi-source heterogeneous data processing process. This mechanism of introducing custom data governance rules enables the result data to meet domain requirements and ensure data accuracy, thereby effectively integrating data from different systems and realizing automated and intelligent data governance.

[0078] See also Figure 2 , is a schematic diagram of a data governance system module based on multi-source data provided in an embodiment of the present application. The data governance system based on multi-source data may include: a data configuration module, a mining rule generation module, a mining path generation module and a data mining module, wherein:

[0079] A template matching module is used to match a user-defined target template corresponding to a target file when a target file to be managed is obtained;

[0080] A template parsing module, used to parse the target template and obtain parsed template data;

[0081] A data matching module, used for matching the parsed template data with the data columns in the target file to obtain a matching result;

[0082] A data screening module, used to screen the matching results based on preset screening rules to obtain screening results;

[0083] The data governance module is used to perform data governance on the screening results to obtain governance results.

[0084] Optionally, a data governance system based on multi-source data also includes a template customization module, which is used to obtain the type of target data and determine an initial table file based on the target data type; obtain the type of each worksheet in the initial table file and the type of each data column in each worksheet; and generate a target template based on the type of each worksheet and the type of each data column in each worksheet.

[0085] Optionally, the template parsing module is also used to parse the header information of each worksheet in the target template; determine whether there is a special mark in the header information; if there is a special mark in the header information, match the data column with the special mark according to the type corresponding to the special mark; if there is no special mark in the header information, match the data column type of each worksheet according to a preset priority rule to obtain the parsed template data.

[0086] Optionally, the data matching module is also used to match the parsed template data with the data in the target file to obtain the matching degree and the data columns that failed to match; when the matching degree is lower than a preset value, a prompt message is displayed to prompt the user to re-match the data columns that failed to match in the target file; when the matching degree is not lower than a preset value, the data columns that successfully matched are used as the matching results.

[0087] Optionally, the data filtering module is also used to obtain the user's filtering requirements; generate filtering rules based on the filtering requirements and the abstract syntax tree; filter the matching results based on preset filtering rules to obtain filtering results, including: filtering out target data that meets the preset filtering rules in the matching results, and using the target data as the filtering results.

[0088] Optionally, the data governance module is further used to obtain user-defined data cleaning rules; govern the data in the screening results according to the data cleaning rules to obtain a governance result.

[0089] It should be noted that: when the system provided in the above embodiment realizes its functions, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0090] An embodiment of the present application also provides a computer storage medium, which can store multiple instructions. The instructions are suitable for being loaded by a processor and executing a data governance method based on multi-source data in the above embodiment. The specific execution process can be found in the specific description of the above embodiment, which will not be repeated here.

[0091] Please refer to Figure 3 The application also discloses an electronic device. Figure 3The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0092] The communication bus 302 is used to realize the connection and communication between these components.

[0093] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0094] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0095] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0096] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , as a computer storage medium, the memory 305 may include an operating system, a network communication module, a user interface module, and an application program of a data governance method based on multi-source data.

[0097] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the application program of a data governance method based on multi-source data stored in the memory 305, and when executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0098] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0102] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0103] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0104] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A data governance method based on multi-source data, characterized in that: The method comprises: When a target file to be managed is obtained, a user-defined target template corresponding to the target file is matched; Parsing the header information of each worksheet in the target template; Determine whether there is a special mark in the header information; If there is a special mark in the header information, the data column with the special mark is matched according to the type corresponding to the special mark; If there is no special mark in the table header information, the data column types of each worksheet are matched according to a preset priority rule to obtain the parsed template data; Performing data matching on the parsed template data and the data columns in the target file to obtain a matching result; Filter the matching results based on preset filtering rules to obtain filtering results; Data governance is performed on the screening results to obtain governance results.

2. The data governance method based on multi-source data according to claim 1 is characterized in that: When the target file to be managed is obtained, before matching the user-defined target template corresponding to the target file, the method further includes: Acquire the type of target data, and determine an initial table file based on the type of target data; Obtain the type of each worksheet in the initial table file, and the type of each data column in each worksheet; A target template is generated based on the type of each worksheet and the type of each data column in each worksheet.

3. The data governance method based on multi-source data according to claim 1 is characterized in that: The preset priority rules are: the priority of the script rule is greater than the priority of the initial value specified by the table, the priority of the initial value specified by the table is greater than the priority of the special marking rule, and the priority of the special marking rule is greater than the priority of the target template configuration.

4. The data governance method based on multi-source data according to claim 1 is characterized in that: The step of performing data matching on the parsed template data and the data columns in the target file to obtain a matching result includes: Performing data matching on the parsed template data and the data in the target file to obtain a matching degree and a matching failure data column; When the matching degree is lower than a preset value, a prompt message is displayed to prompt the user to re-match the data columns in the target file that failed to match; When the matching degree is not lower than the preset value, the successfully matched data column is taken as the matching result.

5. The data governance method based on multi-source data according to claim 1 is characterized in that: The method further comprises: filtering the matching results based on the preset filtering rules, before obtaining the filtering results: Obtain the user's screening requirements; Generate screening rules based on the screening requirements and the abstract syntax tree; The matching results are screened based on preset screening rules to obtain screening results, including: Target data that meets the preset screening rules is screened out from the matching results, and the target data is used as the screening result.

6. The data governance method based on multi-source data according to claim 1 is characterized in that: The performing data governance on the screening results to obtain governance results includes: Get user-defined data cleaning rules; The data in the screening result is governed according to the data cleaning rule to obtain a governance result.

7. A data governance system based on multi-source data, characterized in that: The system comprises: A template matching module is used to match a user-defined target template corresponding to a target file when a target file to be managed is obtained; A template parsing module, used to parse the header information of each worksheet in the target template; determine whether there is a special mark in the header information; if there is a special mark in the header information, match the data column with the special mark according to the type corresponding to the special mark; if there is no special mark in the header information, match the data column type of each worksheet according to a preset priority rule to obtain the parsed template data; A data matching module, used for matching the parsed template data with the data columns in the target file to obtain a matching result; A data screening module, used to screen the matching results based on preset screening rules to obtain screening results; The data governance module is used to perform data governance on the screening results to obtain governance results.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-6.

Images