A method and device for constructing a data standardization platform

Abstract

The present disclosure relates to a method and device for constructing a data standardization platform, the method comprising: configuring dictionary table data corresponding to different components in a data standard template to be configured, generating a basic code set; configuring data items and control types of data items used to define data standards based on the basic code set, generating a data standard template; collecting various values ​​of data standard data items based on the data standard template, generating a target data standard; the target data standard includes: basic standards corresponding to field metadata and code standards corresponding to dictionary table data with the same meaning in different systems; reviewing the target data standard; and if the review is passed, performing field metadata management, data quality management, and dictionary table data management according to the target data standard. The data standardization platform constructed by the present disclosure can better support the standardized processing of data and improve the data governance capability.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for constructing a data standardization platform. Background Art

[0002] Data governance analysis is to filter and integrate huge amounts of data, so that users can follow the real-time status of the data, facilitate users to analyze and judge the data business more accurately and quickly, and use data to drive the business, so as to reduce costs and increase efficiency for enterprises. The data analysis platform system developed based on big data technology can filter from multiple business systems to ODS intermediate libraries, ODS intermediate libraries to data warehouse models, and then bind the model components to realize graphical operations of data, so that users can observe and analyze data of different businesses more intuitively.

[0003] However, most enterprise systems are built directly based on business needs without an overall plan. In addition, different systems may be built by different manufacturers, so data inconsistency between different systems is difficult to avoid. The root cause is the lack of unified data standards to constrain. Many data problems that arise in the process of using data by enterprises are caused by the lack of standard constraints and overall planning and design. Therefore, in terms of data governance, a management platform that can unify and standardize data is needed. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a method and device for constructing a data standardization platform.

[0005] According to one aspect of the present disclosure, a method for constructing a data standardization platform is provided, comprising:

[0006] Configure the dictionary table data corresponding to different components in the data standard template to be configured, and generate a basic code set;

[0007] Based on the basic code set, data items used to define data standards and control types of the data items are configured to generate the data standard template;

[0008] Collecting values ​​of data standard data items based on the data standard template to generate target data standards; the target data standards include: basic standards corresponding to field metadata and code standards corresponding to dictionary table data with the same meaning in different systems;

[0009] Review the target data standards;

[0010] If the review is passed, field metadata management, data quality management and dictionary table data management are performed according to the target data standard.

[0011] According to another aspect of the present disclosure, a device for constructing a data standardization platform is provided, comprising:

[0012] A basic code configuration module is used to configure the dictionary table data corresponding to different components in the data standard template to be configured, and generate a basic code set;

[0013] A data standard template configuration module, configured to configure data items used to define data standards and control types of the data items based on the basic code set, and generate the data standard template;

[0014] A data standard generation module, used to collect various values ​​of data standard data items based on the data standard template to generate a target data standard; the target data standard includes: a basic standard corresponding to field metadata and a code standard corresponding to dictionary table data with the same meaning in different systems;

[0015] A data standard review module, used to review the target data standard;

[0016] The data management module is used to perform field metadata management, data quality management and dictionary table data management according to the target data standard when the audit is passed.

[0017] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0018] The present disclosure relates to a method and device for constructing a data standardization platform, the method comprising: configuring dictionary table data corresponding to different components in a data standard template to be configured, generating a basic code set; configuring data items and control types of data items used to define data standards based on the basic code set, generating a data standard template; collecting various values ​​of data standard data items based on the data standard template, generating a target data standard; the target data standard includes: basic standards corresponding to field metadata and code standards corresponding to dictionary table data with the same meaning in different systems; reviewing the target data standard; and if the review is passed, performing field metadata management, data quality management, and dictionary table data management according to the target data standard. The data standardization platform constructed by the present disclosure can better support the standardized processing of data and improve the data governance capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 This is a flow chart of a method for constructing a data standardization platform according to an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a basic code set according to an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of the basic standard template described in the embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of a code standard template according to an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of the process of reviewing the target data standard according to the embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the code mapping relationship described in the embodiment of the present disclosure;

[0027] Figure 7 This is a schematic diagram of a code conversion function generation process according to an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram of the cleaning rules described in the embodiment of the present disclosure;

[0029] Fig. 9 This is a structural block diagram of a device for constructing a data standardization platform according to an embodiment of the present disclosure;

[0030] Fig.10 It is a schematic diagram of the structure of the electronic device described in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0033] Many data problems that arise in enterprises during the use of data are caused by the lack of standard constraints and overall planning and design. For example: (1) Inconsistent data storage structure. When calling data from multiple systems, some data has different data storage structures in different systems, which makes it impossible to directly associate the data, affecting data sharing between different systems; (2) Inconsistent data definition. Different systems have different definitions of data naming, business meaning, value range, etc., such as the same name but different meanings, or the same meaning but different names; (3) Inconsistent data understanding. Different people have different understandings of data, which leads to a lot of communication time wasted when using data; (4) Inconsistent data sources. Data has multiple sources. When using data, it is not clear which system's data should be retrieved.

[0034] Therefore, in terms of data governance, a management platform that can unify and standardize data is needed. Based on this, the embodiment of the present disclosure provides a method and device for constructing a data standardization platform. For ease of understanding, the embodiment of the present disclosure is described below.

[0035] Figure 1 The flowchart of a method for constructing a data standardization platform provided in an embodiment of the present disclosure can be executed by a device for constructing a data standardization platform configured in a terminal, and the device can be implemented by software and / or hardware. Figure 1 As shown, the method for constructing a data standardization platform may include the following steps:

[0036] Step S101 , configuring the dictionary table data corresponding to different components in the data standard template to be configured, and generating a basic code set.

[0037] This embodiment can configure the dictionary table data required by different components in the data standard template in a visual manner, generate a basic code set, provide data sources for the components in the data standard template, and realize online visual maintenance of basic configuration data.

[0038] By configuring the basic code set, you can select the code set in the configuration of the data standard template. The configuration process can refer to the following. First, the basic code set includes system basic codes; specifically, a series of system basic codes can be built-in by default according to the data items built-in the data standard template, such as: code set subject classification, data type code, value domain type code, security level code, standard document code set, standard code level, standard document code set and other system basic codes. The system basic codes are all set to the enabled state. Secondly, combined with Figure 2 The basic code set also includes custom basic codes. Custom basic codes require configuration of code set name, description, and code member list; the code member list includes: code value, code name, parent code, and valid flag; code members can all be at the same level or in a tree structure.

[0039] Step S102: configuring data items and control types of data items used to define data standards based on the basic code set to generate a data standard template.

[0040] In one implementation, the user can freely and flexibly add and configure data items for defining data standards on the template management interface to achieve dynamic expansion of data items of data standards to form data standard templates that meet user needs. In this embodiment, the data standard templates generated by configuration may include: basic standard templates and code standard templates.

[0041] The basic standard template includes the system built-in data items and control types, which can be referred to Figure 3 For example, data items include: standard code, standard classification, standard Chinese name, standard English name, data type, data format, value domain type, value domain range, security level, remarks and formulation basis; the space types corresponding to the above data items are: input box, drop-down tree (single choice), input box, input box, drop-down tree (single choice), input box, drop-down box (single choice), input box, drop-down box (single choice), text box, drop-down box (single choice).

[0042] The code standard template contains the system's built-in data items and control types for reference Figure 4 For example, data items include: standard code, standard code level, standard English name, standard Chinese name, remarks, and basis for formulation; the space types corresponding to the above data items are: input box, drop-down box (single choice), input box, input box, text box, drop-down box (single choice).

[0043] For the above-mentioned basic standard template and code standard template, this embodiment can also add customized data items according to the user's new operation, such as the operation of the "add a row" button; according to the user's deletion operation and sorting operation, the data items are deleted and sorted accordingly.

[0044] Step S103, collect various values ​​of data standard data items based on the data standard template to generate a target data standard. The target data standard includes: a basic standard corresponding to field metadata and a code standard corresponding to dictionary table data with the same meaning in different systems.

[0045] The implementation process of this embodiment may include: generating a data standard collection interface based on a data standard template, the data standard collection interface including multiple data standard data items; the data standard collection interface can support users to enter data standard data items online, and provide the ability of dynamic interface rendering to meet the needs of users in different industries and different periods for data standard management.

[0046] Collect the values ​​of the data standard data items to obtain the original data standard. In one implementation, the values ​​of the data standard data items can be written in Excel according to a preset import template to import the original data standard.

[0047] The basic standards in the original data standard are standards formulated for field metadata, generally including: field name, field annotation, data type, data format, length, precision, value range, etc. The standard code is uniformly generated by the system to prevent code duplication, and the generated standard code can also be modified. The code standard in the target data standard is a standard formulated for dictionary table data with the same meaning in different systems.

[0048] The original data standard is checked for redundancy to obtain the target data standard. Specifically, this embodiment performs a redundancy check on the original data standard to check the redundancy between the current original data standard and other original data standards, thereby reducing the repeated creation of standards with the same meaning. The target data standard includes: basic standards and code standards.

[0049] Step S104, reviewing the target data standard.

[0050] In order to ensure the quality and credibility of the target data standards published to the public, the present embodiment may review the target data standards, and only the target data standards that pass the review will be published to the public and applied in actual production.

[0051] Reference Figure 5 The embodiment of reviewing the target data standard to be published can refer to the following content. For the target data standard, two operations of saving and publishing can be provided. Based on this, this embodiment includes:

[0052] (1) Monitor whether the received operation for the target data standard is a save operation or a publish operation.

[0053] (2) When a save operation for a target data standard is received, the target data standard is controlled to enter a draft state, and the target data standard is saved in a preset draft storage space.

[0054] (3) When a publishing operation for the target data standard is received, the target data standard is controlled to enter a waiting-for-publishing state. The publishing operation can be for the target data standard generated in step S103 or for the target data standard stored in the draft storage space.

[0055] (4) Conducting a release review on the target data standards to be released. In actual applications, the target data standards to be released can be manually reviewed by users, or automatically reviewed according to preset review conditions.

[0056] (5) If the release review fails, the target data standard is controlled to enter a rejected state, and the above step S103 can be executed, that is, the values ​​of the data standard data items are collected based on the data standard template to generate the target data standard.

[0057] (6) If the release review is passed, the control target data standard enters the released state.

[0058] For the published target data standard, it is still possible to return to the above step S103 to re-collect new values ​​of the data standard data items. Alternatively, it is also possible to choose to go offline according to actual needs. Refer to the following steps accordingly.

[0059] (7) After the target data standard enters the published state, when a logout operation for the target data standard is received, the target data standard is controlled to enter the waiting logout state.

[0060] (8) Conduct offline review of the target data standards to be taken offline.

[0061] (9) Change the status of the target data standard according to the offline review result. Specifically, if the offline review fails, the target data standard is controlled to return to the published state; if the offline review passes, the target data standard is controlled to enter the offline state. At this point, the processing flow for the target data standard ends.

[0062] In this embodiment, target data standards in the draft state, rejected state, and offline state can be deleted.

[0063] Step S105: if the review is passed, field metadata management, data quality management and dictionary table data management are performed according to the target data standard.

[0064] According to the above embodiments, when the publication review of the target data standard is passed, the target data standard can be used as the basis for data asset management, and data asset management can include field metadata management, data quality management and dictionary table data management.

[0065] Field metadata management includes: filling in the fields in the table to be created according to the basic standards. Specifically, you can create a table according to the target data standard. When creating a field, you can reference the standard English name, standard Chinese name, data type and data format of the basic standard, fill in the field name, field comment, type, length and precision of the table, and modify the content to finally create a data table.

[0066] Data quality management includes: selecting the code value or code name of the code standard as the field audit rule, and using the field audit rule to perform rule verification on the dictionary table data, and generating a quality inspection report. In this embodiment, when performing a quality inspection on data and creating a job, you can select the code value or code name of the code standard as the field audit rule, and use the field audit rule to perform rule verification on non-standard dictionary table data, and generate a quality inspection report, allowing users to more easily observe data quality issues. By associating field audit rules with code standards, this solution can, on the one hand, achieve field-level data quality verification, and on the other hand, directly build a more general data quality audit rule system to ensure the comprehensiveness and usability of the rules.

[0067] Dictionary table data management mainly includes the following contents: code mapping relationship, cleaning rules, standardized cleaning tasks, standardized cleaning task operation and standardized cleaning task overview. For ease of understanding, the following is a detailed description.

[0068] (I) Obtain code mapping information, the code mapping information including: mapping name, selecting code standard, selecting source business code table, adding description and mapping result; generating a code mapping relationship between the business system code and the code standard according to the code mapping information.

[0069] Reference Figure 6 Each code mapping relationship needs to configure the following code mapping information: mapping name, selecting code standard, selecting source business code table, adding description and configuring mapping result; a code mapping result includes: source code value, source code name, code standard code value and code standard code name.

[0070] This embodiment can add a source code value and a source code name; and can automatically match the connection mapping according to the similarity comparison of the code value or the code name to generate a code mapping relationship.

[0071] This embodiment can generate a code conversion function and a code conversion cleaning rule based on the code mapping relationship between the business system code and the code standard. A possible implementation process is as follows: Figure 7 As shown, it includes the following contents: Generate a txt file with the code mapping relationship, the file name is the mapping name, and each line of text in the file corresponds to the connection value of the code mapping relationship.

[0072] Determine whether a history file with the same mapping name already exists on HDFS (Hadoop Distributed File System). If not, upload the above generated txt file to HDFS; if yes, delete the history file on HDFS and upload the above generated txt file to HDFS.

[0073] Check whether there is a user-defined code mapping conversion function on HDFS. The customized code mapping conversion function can be called UDF (user-defined functions). If yes, check whether the code conversion function has been registered. If not, obtain the UDF submitted by the user, and upload the UDF to HDFS after registration. Use UDF to generate a code conversion function in the function management function and a code conversion cleaning rule in the cleaning rule function; then, check whether the code conversion function has been registered. Register the code conversion function if it has not been registered.

[0074] The above embodiments can automatically generate a variety of code conversion functions and code conversion cleaning rules by configuring the code mapping relationship between the business system code and the code standard, provide a reference relationship for the standardized processing of business codes in subsequent standardized cleaning tasks, and facilitate users to directly use the code cleaning function.

[0075] (II) Generate multiple cleaning rules using the code mapping relationship and preset standardized program information, where the standardized program information includes: program name, program classification, engine type, and standardized functions corresponding to each engine type.

[0076] Reference Figure 8 For example, a cleaning rule includes: program name, program category, engine type, description, and standardized functions and usage help corresponding to each engine type. Cleaning rules not only include system-generated code conversion cleaning rules, but also other functions and cleaning rules can be customized to clean different business data according to different rules. For example: ID card, mobile phone number, email address, longitude and latitude cleaning, content interception, space removal and null value processing. This further ensures the standardization and unification of business data and strengthens data sharing between systems.

[0077] This embodiment abstracts the standardized cleaning business logic into cleaning rules, which can help users to precipitate industry-level standardized cleaning business rules. When configuring the standardized data processing task, select the standardized processing program, fill in the program parameters, and the data standardized cleaning processing can be automatically completed, greatly improving the efficiency of user data standardized cleaning. When the system's own functions cannot meet the needs of complex businesses, you can use the user-writable UDF to generate code conversion functions and code conversion cleaning rules to process complex businesses based on this.

[0078] (III) Configure the standardized cleaning tasks corresponding to the wizard mode and script mode respectively.

[0079] In one embodiment, the process of configuring the standardized cleaning task corresponding to the wizard mode includes:

[0080] Get the first task configuration information, which includes: task name, directory, task description, computing engine, source data partition, source table, target data partition, target table, and field mapping connection between source table and target table. If there is no suitable target table, you can create a new table according to certain rules. Configure the field mapping connection between the source table and the target table. You can select the cleaning rule on the source table field and fill in the parameters required by the corresponding placeholder. The data in the source table will be cleaned according to the cleaning rule, and then the cleaned data will be inserted into the target table.

[0081] A standardized cleaning task corresponding to the wizard mode is generated according to the first task configuration information.

[0082] This embodiment can also configure manual scheduling or periodic scheduling. Periodic scheduling requires configuring a scheduling period. After the standardized cleaning task is released, the task scheduling service is called to save the current standardized cleaning task. The task scheduling service will trigger the cleaning task execution according to the scheduling period.

[0083] In another embodiment, the process of configuring the data standardization processing task corresponding to the script mode includes:

[0084] Get the second task configuration information, which includes: task name, directory, task description, computing engine, target data partition, SQL script and scheduling information; generate a standardized cleaning task corresponding to the script mode according to the second task configuration information. After the standardized cleaning task is published, call the task scheduling service to save the current standardized cleaning task. The task scheduling service will trigger the cleaning task execution according to the scheduling cycle.

[0085] The script mode is suitable for complex scenario data cleaning and processing. It can write scripts online, support online joint debugging, preview results, and provide a series of friendly interactive experiences such as syntax highlighting and keyword auto-completion. In actual applications, users can freely choose wizard mode or script mode according to business needs and complexity to meet the needs of different scenarios.

[0086] (IV) calling the standardized cleaning task, so that the standardized cleaning task cleans the source layer data according to the cleaning rules and then stores it into the standard layer.

[0087] In this embodiment, the distributed task scheduling platform can be connected to call the standardized cleaning task. After the standardized cleaning task is run, the cleaning rules will clean the chaotic data in the source layer to obtain normalized and standardized data, and store it in the standard layer. The task running process in this embodiment includes:

[0088] Query the task mode of the standardized cleaning task to be run, which includes wizard mode and script mode. In script mode, obtain the pre-written cleaning SQL (Structured Query Language, Structured Query Language database) script. In wizard mode, query the code mapping relationship; assemble the cleaning SQL script according to the code mapping relationship.

[0089] Then, the cleaning SQL script is used to generate a standardized cleaning task, and the standardized cleaning task is submitted to the data warehouse. The standardized cleaning task performs cleaning tasks on the data in the data warehouse; after the cleaning task is executed, the data cleaning results are obtained; and the results of the standardized cleaning task are changed.

[0090] In the process of calling the standardized cleaning task, this embodiment can control the standardized cleaning task to stop or re-run, view the running log and running code of the task instance, and preview the data of the instance running result. This can help users understand the running status of the standardized cleaning task and grasp the detailed information of the task running.

[0091] (V) Monitoring the running status of the standardized cleaning task in a preset format; wherein the preset format is a picture, a table, etc. Specifically, this embodiment can monitor the standardized cleaning task in multiple dimensions, and display the running status of the current standardized cleaning task in a combination of charts and other formats, such as: overall statistics, task running status, popular standardized procedures, task growth trend, task running time ranking, thereby helping users to fully understand the running status of the current standardized cleaning task.

[0092] In summary, the method for constructing a data standardization platform provided by the above-mentioned public embodiment, through the configuration and release of unified data standards, combined with institutional constraints, system control and other means, constructs a data standardization platform that can support data standardization processing. As for the platform, it can better realize the integrity, validity, consistency and standardization of platform data, promote the sharing and openness of data, build a unified data governance list, and provide an effective reference basis for data governance activities.

[0093] Fig. 9 This is a structural block diagram of a device for constructing a data standardization platform provided in an embodiment of the present disclosure, and the device is used to implement the method for constructing the above-mentioned data standardization platform. Fig. 9 , the device may include the following modules:

[0094] A basic code configuration module 201 is used to configure the dictionary table data corresponding to different components in the data standard template to be configured, and generate a basic code set;

[0095] A data standard template configuration module 202, configured to configure data items and control types of the data items for defining data standards based on the basic code set, and generate the data standard template;

[0096] The data standard generation module 203 is used to collect various values ​​of data standard data items based on the data standard template to generate a target data standard; the target data standard includes: a basic standard corresponding to field metadata and a code standard corresponding to dictionary table data with the same meaning in different systems;

[0097] A data standard review module 204 is used to review the target data standard;

[0098] The data management module 205 is used to perform field metadata management, data quality management and dictionary table data management according to the target data standard when the audit is passed.

[0099] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0100] Fig.10 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Fig.10 As shown, the electronic device 300 includes one or more processors 301 and a memory 302 .

[0101] The processor 301 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 300 to perform desired functions.

[0102] The memory 302 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 301 may run the program instructions to implement the method for constructing the data standardization platform of the embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0103] In one example, the electronic device 300 may further include: an input device 303 and an output device 304 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0104] In addition, the input device 303 may also include, for example, a keyboard, a mouse, and the like.

[0105] The output device 304 can output various information to the outside, including the determined distance information, direction information, etc. The output device 304 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0106] Of course, to simplify, Fig.10 Only some of the components related to the present disclosure in the electronic device 300 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 300 may also include any other appropriate components.

[0107] Furthermore, this embodiment also provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the method for constructing the above-mentioned data standardization platform.

[0108] The computer program product of a method, device, electronic device and medium for constructing a data standardization platform provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0109] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0110] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a data standardization platform, characterized in that: include: The dictionary table data corresponding to different components in the data standard template to be configured are configured to generate a basic code set; wherein, by configuring the basic code set, a code set is selected in the configuration of the data standard template; the basic code set includes a system basic code: a default built-in system basic code according to the built-in data items of the data standard template, the system basic code includes: code set subject classification, data type code, value domain type code, security level code, standard document code set, standard code level, standard document code set; the basic code set also includes a custom basic code; Based on the basic code set, the data items used to define the data standard and the control types of the data items are configured to generate the data standard template; wherein the configured data standard template includes: a basic standard template and a code standard template; Collecting values ​​of data standard data items based on the data standard template to generate target data standards; the target data standards include: basic standards corresponding to field metadata and code standards corresponding to dictionary table data with the same meaning in different systems; Review the target data standards; If the review is passed, field metadata management, data quality management, and dictionary table data management are performed according to the target data standards; The review of the target data standard includes: When a publishing operation for the target data standard is received, controlling the target data standard to enter a waiting-to-publish state; Conduct release review on the target data standards to be released; If the release review fails, the target data standard is controlled to enter a rejected state, and the step of collecting values ​​of each item of the data standard data item based on the data standard template is performed; If the release review is passed, the target data standard is controlled to enter the released state; The step of collecting values ​​of data standard data items based on the data standard template to generate a target data standard includes: Generate a data standard collection interface based on the data standard template, wherein the data standard collection interface includes a plurality of data standard data items; Collecting various values ​​of data items of the data standard to obtain the original data standard; A redundancy check is performed on the original data standard to obtain a target data standard.

2. The method according to claim 1, characterized in that The method further comprises: After the target data standard enters the published state, when a logout operation for the target data standard is received, controlling the target data standard to enter the waiting logout state; Conduct offline review on the target data standards to be offline; The status of the target data standard is changed according to the offline review results.

3. The method according to claim 1, characterized in that: The field metadata management includes: The fields in the table to be created are filled in according to the basic criteria.

4. The method according to claim 1, characterized in that The data quality management includes: The code value or code name of the code standard is selected as the field audit rule, and the field audit rule is used to perform rule verification on the dictionary table data to generate a quality inspection report.

5. The method according to claim 1, characterized in that The dictionary table data management includes: Obtain code mapping information, which includes: mapping name, code selection standard, source business code table selection, added description and mapping result; Generate a code mapping relationship between the business system code and the code standard according to the code mapping information; Generate multiple cleaning rules using the code mapping relationship and preset standardized program information, wherein the standardized program information includes: program name, program classification, engine type, and standardized functions corresponding to each engine type; Configure standardized cleaning tasks corresponding to wizard mode and script mode respectively; The standardized cleaning task is called so that the standardized cleaning task cleans the source layer data according to the cleaning rule and then stores the data into the standard layer.

6. The method according to claim 5, characterized in that The process of configuring the standardized cleaning tasks corresponding to the wizard mode includes: Acquire first task configuration information, the first task configuration information including: task name, directory, task description, computing engine, source-end data partition, source-end table, target-end data partition, target-end table, and field mapping connection between the source-end table and the target-end table; A standardized cleaning task corresponding to the wizard mode is generated according to the first task configuration information.

7. The method according to claim 5, characterized in that The process of configuring the data standardization and cleaning task corresponding to the script mode includes: Obtaining second task configuration information, the second task configuration information including: task name, directory, task description, computing engine, target end data partition, SQL script, and scheduling information; A standardized cleaning task corresponding to the script mode is generated according to the second task configuration information.

8. A device for constructing a data standardization platform, characterized in that: include: A basic code configuration module is used to configure the dictionary table data corresponding to different components in the data standard template to be configured, and generate a basic code set; wherein, by configuring the basic code set, a code set is selected in the configuration of the data standard template; the basic code set includes a system basic code: a default built-in system basic code according to the built-in data items of the data standard template, and the system basic code includes: code set subject classification, data type code, value domain type code, security level code, standard document code set, standard code level, standard document code set; the basic code set also includes a custom basic code; A data standard template configuration module, configured to configure the data items and the control types of the data items used to define the data standard based on the basic code set, and generate the data standard template; wherein the configured and generated data standard template includes: a basic standard template and a code standard template; A data standard generation module, used to collect various values ​​of data standard data items based on the data standard template to generate a target data standard; the target data standard includes: a basic standard corresponding to field metadata and a code standard corresponding to dictionary table data with the same meaning in different systems; A data standard review module, used to review the target data standard; A data management module, used to perform field metadata management, data quality management and dictionary table data management according to the target data standard if the audit is passed; The data standard review module is also used to: When a publishing operation for the target data standard is received, controlling the target data standard to enter a waiting-to-publish state; Conduct release review on the target data standards to be released; If the release review fails, the target data standard is controlled to enter a rejected state, and the step of collecting values ​​of each item of the data standard data item based on the data standard template is performed; If the release review is passed, the target data standard is controlled to enter the released state; The data standard generation module is also used for: Generate a data standard collection interface based on the data standard template, wherein the data standard collection interface includes a plurality of data standard data items; Collecting various values ​​of data items of the data standard to obtain the original data standard; A redundancy check is performed on the original data standard to obtain a target data standard.

Images