Data quality management method and device

Abstract

The invention relates to the field of artificial intelligence, and particularly provides a data quality management method and device, and the method comprises the following steps: S1, a data quality management rule library module converts a data quality evaluation index into a rule requirement of corresponding quality monitoring according to an actual business demand, and carries out the automatic discovery of a data quality problem based on a rule; s2, a data quality evaluation standard module constructs a data quality evaluation standard and automatically calculates a quality score; s3, a data quality monitoring prototype room module is processed and summarized according to application scene requirements on the basis of DPI data, and then the data quality monitoring prototype room module shares an application layer in a file mode to be displayed and presented; and S4, the data quality monitoring alarm module informs in an alarm form, and the alarm form is set according to different alarm levels. Compared with the prior art, the end-to-end whole-process management and control on the data quality can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of base station operation and maintenance, and specifically provides a data quality management method and device. Background Art

[0002] In the data, the centralized empowerment of the whole network business system has become an increasingly important trend in data value realization. However, the wide range of collected data, the variety of data, and the numerous responsible systems have made the data processing logic more and more complex. Ensuring data quality has become an important prerequisite and foundation for the effective use of data, which has increasingly highlighted the importance of data quality management.

[0003] However, current data quality management methods are limited by technical levels, management methods and data thresholds, and cannot provide end-to-end management and control of data quality. Summary of the invention

[0004] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a data quality management method with strong practicability.

[0005] A further technical task of the present invention is to provide a data quality management device that is rationally designed, safe and applicable.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A data quality management method comprises the following steps:

[0008] S1. The data quality management rule base module converts data quality evaluation indicators into corresponding quality monitoring rule requirements according to actual business needs, and automatically discovers data quality problems based on the rules;

[0009] S2, the data quality evaluation standard module builds data quality evaluation standards and automatically calculates quality scores;

[0010] S3, the data quality monitoring model room module is based on DPI data, which is processed and summarized according to the application scenario requirements, and then shared in the application layer in the form of files for display;

[0011] S4. The data quality monitoring alarm module notifies in the form of an alarm, and the alarm form is set according to different alarm levels.

[0012] Further, in step S1, the evaluation attributes are completeness, accuracy, compliance, uniqueness, timeliness, and consistency, which are used as the basis for setting the evaluation attribute weights in the data quality evaluation standard;

[0013] Evaluation attribute execution rules: Execution rules need to be formulated for each type of evaluation attribute, and the formulated execution rules are used to check data quality and output quality issues.

[0014] Furthermore, in step S2, the quality score has two dimensions, one is the data level, and the other is the quality attribute. The data level is divided into levels according to data source file->data collection->data calculation->data sharing. The higher the level, the higher the quality evaluation weight.

[0015] The quality attributes are derived from the data quality rule management library and are completeness, accuracy, compliance, uniqueness and timeliness.

[0016] Furthermore, the evaluation attribute quality score depends on the execution results of each attribute execution rule formulated in the data quality rule management library. The formula is the number of correct rules under each attribute / the total number of rules under each attribute. After the quality scores of the six types of attributes are calculated, the quality score of each row is weighted and multiplied by the weight of the level to which the row belongs, and finally the quality score of the monitoring data is output.

[0017] Further, in step S3, the integrity quality score, accuracy verification result, compliance verification result, uniqueness verification result, timeliness verification result and evaluation attribute quality score are calculated.

[0018] Further, in step S4, if the quality score is lower than N points, it is a prompt and general alarm, using interface warning and Email; if the quality score is higher than N points, it is an important alarm and a serious alarm, using interface warning and short message.

[0019] Further, the specific descriptions of various alarm forms are as follows:

[0020] (1) Interface warning: Display warning information on the data quality monitoring interface, and use different colors and prompt symbols to distinguish different warning levels;

[0021] (2) Short message: Alerts are sent via short message according to pre-defined alarm levels;

[0022] (3) Email method: Alerts are sent to relevant persons in charge by sending emails according to the pre-defined alarm level;

[0023] (4) Work Order: Generate a fault task work order and alert the relevant person in charge.

[0024] A data quality management device, comprising: at least one memory and at least one processor;

[0025] The at least one memory is used to store a machine-readable program;

[0026] The at least one processor is used to call the machine-readable program to execute a data quality management method.

[0027] Compared with the prior art, the data quality management method and device of the present invention have the following outstanding beneficial effects:

[0028] The present invention innovatively implements a data quality management method that breaks down entry barriers by fully managing data, improves the quality management level with professional and complete quality evaluation standards, and realizes end-to-end full-process management and control of data quality. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] A best embodiment is given below:

[0031] A data quality management method in this embodiment has the following steps:

[0032] S1. The data quality management rule base module converts data quality evaluation indicators into corresponding quality monitoring rule requirements based on actual business needs, automatically discovers data quality problems based on the rules, reduces the daily maintenance workload of data quality monitoring personnel, and improves the efficiency of problem discovery and response processing.

[0033] Evaluation attributes: completeness, accuracy, compliance, uniqueness, timeliness, and consistency, which serve as the basis for setting the weights of evaluation attributes in the data quality evaluation standards.

[0034] Evaluation attribute execution rules: Execution rules must be formulated for each type of evaluation attribute to check data quality and output quality issues.

[0035]

[0036]

[0037] S2. The data quality evaluation standard module constructs the data quality evaluation standard and automatically calculates the quality score.

[0038] There are two dimensions for quality scoring: data level and quality attribute. The data level is divided according to data source file->data collection->data calculation->data sharing. The higher the level, the higher the quality evaluation weight.

[0039] Quality attributes are derived from the data quality rule management library and are completeness, accuracy, compliance, uniqueness, and timeliness. Weight settings support editing and adjustment, and the initial settings are as follows:

[0040]

[0041]

[0042] The evaluation attribute quality score depends on the execution results of each attribute execution rule established in the data quality rule management library. The formula = number of correct rules under each attribute / total number of rules under each attribute. After the quality scores of the six types of attributes are calculated, the quality score of each row is weighted and multiplied by the weight of the level to which the row belongs. Finally, the quality score of the monitoring data is output.

[0043] S3. In the data quality monitoring model room module, 4G non-real-time city-level dimensional data is based on DPI data, processed and summarized according to application scenario requirements, and then shared in the application layer in the form of files for display and presentation.

[0044] Take this as an example to build a data quality monitoring model room, the content is as follows:

[0045] Completeness quality score (2 / 4 = 0.5)

[0046]

[0047] Accuracy check results (3 / 7=0.43)

[0048]

[0049]

[0050] Compliance verification results (2 / 6 = 0.33)

[0051]

[0052] Uniqueness check result (0 / 1=0)

[0053]

[0054] Timeliness verification results (1 / 3 = 0.33)

[0055]

[0056] Evaluation attribute quality score:

[0057] The data level of the quality check data of the data quality sample room is data sharing, and its own weight is w4. The individual scores of the evaluation attributes are as above, so the final data quality score calculated is w4*(w41*0.5+w12*0.43+w13*0.33+w14*0+w15*0.33).

[0058] S4. In the data quality monitoring and alarm module, the data quality monitoring results are notified in the form of alarms. The alarm form is set according to the different alarm levels. For example, if the quality score is lower than N points, it is a prompt and a general alarm, using interface warnings and emails; if the quality score is higher than N points, it is an important alarm and a serious alarm, using interface warnings and short messages.

[0059] The detailed descriptions of various alarm forms are as follows:

[0060] (1) Interface warning: Display warning information on the data quality monitoring interface, and use different colors and prompt symbols to distinguish different warning levels;

[0061] (2) Short message: Alerts are sent via short message according to pre-defined alarm levels;

[0062] (3) Email method: Alerts are sent to relevant persons in charge by sending emails according to the pre-defined alarm level;

[0063] (4) Work Order: Generate a fault task work order and alert the relevant person in charge.

[0064] Based on the above method, a data quality management device in this embodiment includes: at least one memory and at least one processor;

[0065] The at least one memory is used to store a machine-readable program;

[0066] The at least one processor is used to call the machine-readable program to execute a data quality management method.

[0067] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0068] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data quality management method, characterized in that: The steps are as follows: S1. The data quality management rule base module converts data quality evaluation indicators into corresponding quality monitoring rule requirements according to actual business needs, and automatically discovers data quality problems based on the rules; S2, the data quality evaluation standard module builds data quality evaluation standards and automatically calculates quality scores; S3, the data quality monitoring model room module is based on DPI data, which is processed and summarized according to the application scenario requirements, and then shared in the application layer in the form of files for display; S4. The data quality monitoring alarm module notifies in the form of an alarm, and the alarm form is set according to different alarm levels.

2. A data quality management method according to claim 1, characterized in that: In step S1, the evaluation attributes are completeness, accuracy, compliance, uniqueness, timeliness, and consistency, which are used as the basis for setting the evaluation attribute weights in the data quality evaluation standard; Evaluation attribute execution rules: Execution rules need to be formulated for each type of evaluation attribute, and the formulated execution rules are used to check data quality and output quality issues.

3. A data quality management method according to claim 2, characterized in that: In step S2, the quality score has two dimensions, one is the data level, and the other is the quality attribute. The data level is divided into levels according to data source file->data collection->data calculation->data sharing. The higher the level, the higher the quality evaluation weight. The quality attributes are derived from the data quality rule management library and are completeness, accuracy, compliance, uniqueness and timeliness.

4. A data quality management method according to claim 3, characterized in that: The evaluation attribute quality score depends on the execution results of each attribute execution rule established in the data quality rule management library. The formula is the number of correct rules under each attribute / the total number of rules under each attribute. After the quality scores of the six types of attributes are calculated, the quality score of each row is weighted and multiplied by the weight of the level to which the row belongs. Finally, the quality score of the monitoring data is output.

5. A data quality management method according to claim 4, characterized in that: In step S3, the integrity quality score, accuracy verification result, compliance verification result, uniqueness verification result, timeliness verification result and evaluation attribute quality score are calculated.

6. A data quality management method according to claim 5, characterized in that: In step S4, if the quality score is lower than N points, it will be a prompt and general warning, using interface warning and email; If the quality score is higher than N points, it is an important alarm or a serious alarm, which will be issued via interface warning and short message.

7. A data quality management method according to claim 6, characterized in that: The detailed descriptions of various alarm forms are as follows: (1) Interface warning: Display warning information on the data quality monitoring interface, and use different colors and prompt symbols to distinguish different warning levels; (2) Short message: Alarms can be sent via short message according to pre-defined alarm levels; (3) Email method: Alerts are sent to relevant persons in charge by sending emails according to the pre-defined alarm level; (4) Work Order: Generate a fault task work order and alert the relevant person in charge.

8. A data quality management device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 7.

Images