Data verification method, device, equipment and readable storage medium

Abstract

The embodiment of the present application proposes a data verification method, device, equipment and storage medium, the method comprising: in the process of verifying data flow, obtaining first data to be verified and corresponding first verification rules from a first data processing end; verifying the first data to be verified according to the first verification rule to obtain a first verification result; when the first verification result is not passed, and it is determined that the first data processing end is not the last data processing end in the data verification direction, obtaining second data to be verified and corresponding second verification rules of a second data processing end according to the data verification direction; verifying the second data to be verified according to the second verification rule to obtain a second verification result. Since data verification is performed in the opposite direction of data flow, quality problems in the data to be verified can be accurately and comprehensively determined.

Description

Technical Field

[0001] The present application relates to data verification technology, and more particularly to a data verification method, device, equipment and computer-readable storage medium. Background Art

[0002] In the related technology, the methods to improve data quality mainly include: a process based on data warehouse technology (Extract-Transform-Load, ETL), which mainly verifies data at nodes in the ETL process. However, since this method only considers the data quality in the ETL process, it may lead to inaccurate verification results. Summary of the invention

[0003] The embodiments of the present application are intended to provide a data verification method, apparatus, data verification device and computer storage medium.

[0004] In a first aspect, an embodiment of the present application provides a data verification method, the method comprising: in a verification process of data flow, obtaining first data to be verified and a corresponding first verification rule from a first data processing end; verifying the first data to be verified according to the first verification rule to obtain a first verification result; when the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, obtaining second data to be verified and a corresponding second verification rule of a second data processing end according to the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end; verifying the second data to be verified according to the second verification rule to obtain a second verification result.

[0005] In a second aspect, an embodiment of the present application provides a data verification device, comprising: a first acquisition module, used to acquire first data to be verified and a corresponding first verification rule from a first data processing end during a verification process of data flow; a first verification module, used to verify the first data to be verified according to the first verification rule to obtain a first verification result; a second acquisition module, used to acquire second data to be verified and a corresponding second verification rule of a second data processing end according to the data verification direction when the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end; and a second verification module, used to verify the second data to be verified according to the second verification rule to obtain a second verification result.

[0006] In a third aspect, the disclosed embodiment of the present application further provides a data verification device, comprising: a memory and a processor, wherein the memory is used to store executable instructions; and the processor is used to implement any of the above-mentioned data verification methods when executing the executable instructions stored in the memory.

[0007] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing executable instructions for implementing any of the above-mentioned data verification methods when executed by a processor.

[0008] In an embodiment of the present application, data verification is performed in the opposite direction of data flow. Only when the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, the verification of the data to be verified at the second data processing end is performed. Therefore, the quality problems in the data to be verified can be accurately and comprehensively determined; at the same time, since the data processing ends in the data flow direction can include the source data end, the database end, the data interface end and the front-end page end in sequence, and the data processing ends in the data verification direction include the front-end page end, the data interface end, the database end and the source data end in sequence, therefore, end-to-end data quality improvement from the front-end page end to the data source end can be achieved, and the data verified by the prior art is more comprehensive, avoiding inaccurate verification results due to omission of certain data. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of the data flow direction of an embodiment of the present application;

[0010] Figure 2 A flowchart of a data verification method according to an embodiment of the present application;

[0011] Figure 3 A flowchart of another data verification method according to an embodiment of the present application;

[0012] Figure 4 A flowchart of another data verification method according to an embodiment of the present application;

[0013] Figure 5 A flowchart of another data verification method according to an embodiment of the present application;

[0014] Figure 6 A flowchart of another data verification method according to an embodiment of the present application is shown;

[0015] Figure 7 A flowchart of another data verification method according to an embodiment of the present application;

[0016] Figure 8A flowchart of a method for verifying data of application A at time T according to an embodiment of the present application;

[0017] Fig. 9 A schematic diagram of the structure of a data verification device according to an embodiment of the present application;

[0018] Fig.10 A structural schematic diagram of a data verification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] The present application will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the embodiments provided herein are only used to explain the present application and are not intended to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions recorded in the embodiments of the present application can be implemented in any combination.

[0020] With the advent of the Internet and big data era, people communicate through the network, generating massive amounts of wireless, economic, fault and other data. Operators collect, process and analyze these data to form key performance indicators (KQI) and display the KPI through the front-end page. In this way, key indicators of the network and end-to-end services can be monitored to timely discover and present poor quality problems.

[0021] Due to the diversity of data collection methods and inconsistent data sources, as well as the existence of data with different time granularities, real-time and non-real-time data, different methods are used in the database to process the incoming data. Therefore, the front-end page has data quality problems such as missing numbers, indicator ranges that do not conform to business logic, data display delays, and inconsistent data with the same meaning on different pages. At the same time, these problems cannot be quickly located and solved in a timely manner. Therefore, in order to improve the timeliness, integrity, legality, accuracy and consistency of the data on the front-end page, and to quickly locate and solve the data quality problems generated by the front-end page in a timely manner, it is imperative to improve the end-to-end data quality of the front-end page.

[0022] Figure 1 Schematic diagram of data flow direction of the embodiment of the present application, such as Figure 1As shown, the data processing ends sequentially experienced in the process of data flow according to the data flow direction include: a source data end 101, a database end 102, a data interface end 103 and a front-end page end 104, wherein the source data end 101 acquires communication data and sends the acquired communication data to the database end 102, the database end 102 processes the communication data and sends the processed data to the data interface end 103, the data interface end 103 receives the data sent by the database end, and sends the received data to the front-end page end 104, and the front-end page end 104 displays the received data.

[0023] In the related technologies, there are two main methods to improve data quality: Method 1: Based on the ETL process, the corresponding rule verification is performed at each verification node and the verification result is generated. If the verification result meets the threshold range, an alarm detail list is generated, and the alarm detail list is summarized and presented in the database. Method 2: First, the relevant data is cleaned, and the problems found in the data cleaning are sent to the relevant data submitters for problem solving, so as to achieve sustainable management of data quality.

[0024] It can be seen that method 1 is mainly aimed at improving data quality in the ETL process. ETL is only one of the tools used by the database to process data. It only considers the data quality of the ETL process. Other tools used by the database to process data will include other processing processes, such as the aggregation process. Therefore, the data quality of other processing processes of the database is not considered. Method 2 only checks the data that cannot be cleaned by the database, without considering data conversion, data aggregation, etc., and does not check the data interface and front-end page data.

[0025] In response to the above technical problem, in some embodiments of the present application, a data verification method is proposed.

[0026] Figure 2 A flow chart of a data verification method according to an embodiment of the present application is shown in FIG. Figure 2 As shown, the process may include:

[0027] Step S201: During the verification process of data flow, first data to be verified and corresponding first verification rules are obtained from a first data processing end.

[0028] In a possible implementation, the first data terminal may be one of the following: a front-end page terminal, a data interface terminal, a database terminal, and a source data terminal; the first data to be verified may be data that needs to be verified among all the data on the first data terminal according to user needs. For example, if the user needs to verify the data quality of data C and data D in application A, then the first data to be verified is data C and data D determined in the data related to application A in the data on the front-end page terminal; the first verification rule may be a verification rule for the first verification data, and the verification rule includes but is not limited to requirements for timeliness, completeness, legality, accuracy, and consistency of the data. Here, timeliness can refer to the measurement standard of the timeliness and availability of data for specific applications within the expected period of time; integrity can refer to the standard of whether the data is incomplete or missing, for example, whether gender, place of origin, age, etc. are missing in the attributes of a person; legality can refer to the standard of whether the acquired data is valid, for example, whether the acquired data is inconsistent with common sense, such as age is greater than 150 years old; accuracy can refer to the standard of measuring the correctness of data content; consistency is the measurement of the degree of equivalence of data stored or used in the front-end page end, data interface end, database end, and source data end, as well as the measurement standard of the data equivalence processing flow.

[0029] In a possible implementation, the first verification rule may be a requirement for timeliness, completeness, legality, and accuracy. For example, for application A, the first verification rule may include whether a person's name can be displayed within 5 seconds, whether the person's attributes lack gender, place of origin, age, etc., whether the person's age is less than 130 years old, whether the person's age is an integer, and whether the person's name stored or used in the front-end page end, data interface end, database end, and source data end is completely consistent.

[0030] In one example, obtaining the first data to be verified and the corresponding first verification rule from the first data processing end may be to first determine the data acquisition method corresponding to the first data processing end, then obtain all the data on the first data end through the determined data acquisition method, and then filter all the data obtained on the first data end to obtain data that meets the user's needs or is of concern to the user, determine the data that meets the user's needs or is of concern to the user as the first verification data, and determine the verification standard corresponding to the first data to be verified according to the user's needs, and obtain the verification standard corresponding to the determined first data to be verified. Here, the user's needs may be certain data to be verified that the user in the application to be verified needs to verify, not all the data in the application to be verified.

[0031] Step S202: verify the first data to be verified according to the first verification rule to obtain a first verification result.

[0032] In one embodiment, the first verification result may be that all data in the first data to be verified do not conform to the first verification rule, or that part of the data in the first verification data do not conform to the first verification rule, but as long as there is data in the first verification data that does not conform to the first verification rule, the first verification result is considered to be failed; in another embodiment, the first verification result may be that all data in the first data to be verified conform to the first verification rule, or that a first preset number of data in the first verification data conform to the first verification rule, but as long as a first preset number of data in the first verification data conforms to the first verification rule, the first verification result is considered to be passed. Here, the first preset number proportion may be 98% or a proportion greater than 98% and less than or equal to 1.

[0033] In some possible implementations, the first data to be verified is verified according to the first verification rule to obtain a first verification result. All data in the first data to be verified can be verified to determine whether each data satisfies the first verification rule to obtain a verification result for each data.

[0034] Step S203: When the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, obtain the second data to be verified and the corresponding second verification rule of the second data processing end according to the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end.

[0035] In one embodiment, the data processing end in the data flow direction includes the source data end, the database end, the data interface end and the front-end page end in sequence, and the data processing end in the data verification direction includes the front-end page end, the data interface end, the database end and the source data end in sequence.

[0036] In one example, when the first data processing end is a front-end page end, the second data processing end may be a data interface end.

[0037] For the implementation method of obtaining the second data to be verified and the corresponding second verification rule of the second data processing end according to the data verification direction when the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, exemplarily, it can be that when there is data that does not satisfy the first verification rule in the first data to be verified and the first data processing end is not the source data end, the data acquisition method of the second data processing end is determined, all data on the second data processing end is acquired according to the determined data acquisition method, and the data to be verified of the second data end is filtered out according to user needs, and the verification rules corresponding to the second data to be verified are determined according to user needs, and the verification rules corresponding to the determined second data to be verified are acquired.

[0038] Step S204: verify the second data to be verified according to the second verification rule to obtain a second verification result.

[0039] Here, the second verification result may be that all the data in the second data to be verified do not conform to the second verification rule, or that part of the data in the second verification data do not conform to the second verification rule, but as long as there is data in the first verification data that does not conform to the second verification rule, the second verification result is considered to be failed; in another embodiment, the second verification result may be that all the data in the second data to be verified conform to the second verification rule, or that the data of the second preset number percentage in the second verification data conform to the second verification rule, but as long as the data of the second verification data that is greater than the second preset number percentage conforms to the second verification rule, the second verification result is considered to be passed. Here, the second preset number percentage is also equal to the first preset number percentage, and may also be greater than the first preset number percentage.

[0040] In some possible implementations, the second data to be verified is verified according to the second verification rule to obtain the second verification result. All data in the second data to be verified can be verified to determine whether each data satisfies the second verification rule to obtain the verification result of each data.

[0041] In practical applications, steps S201 to S204 can be implemented using a processor in a data verification device, and the processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), an FPGA, a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0042] In the embodiment of the present application, it can be seen that data verification is performed in the opposite direction of data flow. Only when the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, the verification of the data to be verified at the second data processing end is performed. Therefore, quality problems in the data to be verified can be accurately and comprehensively determined.

[0043] Figure 3 A flowchart of another data verification method according to an embodiment of the present application is shown in FIG. Figure 3 As shown, the process may include:

[0044] Step S301: During the verification process of data flow, the corresponding first verification rule is obtained from the first data processing end; the data processing end in the data flow direction includes the source data end, the database end, the data interface end and the front-end page end in sequence, and the data processing end in the data verification direction includes the front-end page end, the data interface end, the database end and the source data end in sequence.

[0045] Step S302: according to the identifier of the application to be verified, the data to be verified is obtained from the first data processing end as the first data to be verified.

[0046] Here, the identifier of the application to be verified may be the name of the application to be verified. According to the identifier of the application to be verified, the data to be verified is obtained from the first data processing end as the implementation method of the first data to be verified. For example, according to the name of the application to be verified, the data corresponding to the name of the application to be verified is obtained from the first data processing end, and the data corresponding to the obtained name is filtered according to user needs to filter out the data to be verified as the first data to be verified. The user needs here may be which specific data in the data corresponding to the application to be verified is to be verified, and the first data to be verified may be part of the data corresponding to the application to be verified.

[0047] Step S303: verify the first data to be verified according to the first verification rule to obtain a first verification result.

[0048] Step S304: When the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, obtain the second data to be verified and the corresponding second verification rule of the second data processing end according to the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end.

[0049] Step S305: verify the second data to be verified according to the second verification rule to obtain a second verification result.

[0050] In an embodiment of the present application, based on the identifier of the application to be verified, the data to be verified is obtained from the first data processing end as the first data to be verified, and the first data to be verified is verified, so that the data to be verified corresponding to the application to be verified on the first data processing end can be verified.

[0051] Figure 4 FIG. 1 is a flow chart of another data verification method according to an embodiment of the present application. Figure 4 As shown, the process may include:

[0052] Step S401: During the verification process of data flow, the corresponding first verification rule is obtained from the first data processing end; the data processing end in the data flow direction includes the source data end, the database end, the data interface end and the front-end page end in sequence, and the data processing end in the data verification direction includes the front-end page end, the data interface end, the database end and the source data end in sequence.

[0053] Step S402: when the first data processing end is a front-end page end, the element attributes corresponding to the application to be checked and the values ​​of the element attributes are obtained from the first data processing end through a web crawler.

[0054] Here, the implementation method of obtaining the element attributes corresponding to the application to be verified and the assignment of the element attributes from the first data processing end through a web crawler can be to use an application testing package such as a selenium package to simulate a browser to log in with a username and password, open the verification screen / report page on the first data processing end, and find the element attributes corresponding to the application to be verified and the specific assignment of the attribute elements among multiple IDs, keys, class, name and other element attributes.

[0055] Step S403: acquiring the first data to be checked from the first data processing end according to the element attribute corresponding to the application to be checked and the value assigned to the element attribute.

[0056] Here, according to the element attributes corresponding to the application to be verified and the value assigned to the element attributes, the implementation method of obtaining the first data to be verified from the first data processing end can be to obtain the relevant data of the page after JavaScript rendering according to the specific assignment of the attribute element; and to extract the first data to be verified from the relevant data of the page after JavaScript rendering according to user needs.

[0057] Step S404: verify the first data to be verified according to the first verification rule to obtain a first verification result.

[0058] Step S405: When the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, obtain the second data to be verified and the corresponding second verification rule of the second data processing end according to the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end.

[0059] Step S406: verify the second data to be verified according to the second verification rule to obtain a second verification result.

[0060] In an embodiment of the present application, when the first data processing end is a front-end page end, the element attributes corresponding to the application to be verified and the assignment of the element attributes can be obtained from the first data processing end through a web crawler, thereby obtaining the first data to be verified from the first data processing end, and verifying the first data to be verified, thereby realizing the verification of the data to be verified corresponding to the application to be verified on the first data processing end.

[0061] Figure 5 A flowchart of another data verification method according to an embodiment of the present application is shown in FIG. Figure 5 As shown, the process may include:

[0062] Step S501: During the verification process of data flow, the corresponding first verification rule is obtained from the first data processing end; the data processing end in the data flow direction includes the source data end, the database end, the data interface end and the front-end page end in sequence, and the data processing end in the data verification direction includes the front-end page end, the data interface end, the database end and the source data end in sequence.

[0063] Step S502: when the first data processing end is a data interface end, determine a plurality of data interfaces corresponding to the application to be checked.

[0064] Here, the application to be checked may be application A, and application A may correspond to multiple data interfaces, for example, data interface 1, data interface 2, data interface 3, and data interface 4.

[0065] Step S503: obtaining the location information and verification information of each data interface from the first data processing end through a web crawler.

[0066] Here, the application to be verified may be application A, the location information of the data interface may be the Uniform Resource Locator (URL) information in the data interface, and the verification information of the data interface may include the login (cookies) of the data interface, third-party authorization (authorization) and other information.

[0067] In a possible implementation, the location information and verification information of each data interface are obtained from the first data processing end through a web crawler, which may be multiple URLs, cookies, authorization and other information of each data interface of multiple data interfaces corresponding to the application to be verified. Here, the requests package is used to write crawlers and test server response data.

[0068] Step S504: acquiring first data to be verified from the first data processing end according to the location information and verification information of each of the data interfaces.

[0069] Here, step S504 may be implemented by using the get and post methods of the requests package to obtain the data set at each data interface location corresponding to the application to be checked; and extracting the first data to be checked from the data set on each data interface according to user needs.

[0070] Step S505: verify the first data to be verified according to the first verification rule to obtain a first verification result.

[0071] Step S506: When the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, obtain the second data to be verified and the corresponding second verification rule of the second data processing end according to the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end.

[0072] Step S507: verify the second data to be verified according to the second verification rule to obtain a second verification result.

[0073] In an embodiment of the present application, when the first data processing end is a data interface end, multiple data interfaces corresponding to the application to be verified are determined, and location information and verification information of each data interface are obtained from the first data processing end through a web crawler. Based on the location information and verification information of each data interface, the first data to be verified is obtained from the first data processing end, thereby obtaining the first data to be verified from the first data processing end, and verifying the first data to be verified, so as to realize verification of the data to be verified corresponding to the application to be verified on the first data processing end.

[0074] Figure 6 A flowchart of another data verification method according to an embodiment of the present application is shown in FIG. Figure 6 As shown, the process may include:

[0075] Step S601: During the verification process of data flow, the corresponding first verification rule is obtained from the first data processing end; the data processing end in the data flow direction includes the source data end, the database end, the data interface end and the front-end page end in sequence, and the data processing end in the data verification direction includes the front-end page end, the data interface end, the database end and the source data end in sequence.

[0076] Step S602: when the first data processing end is a database end, first data to be checked is obtained from each data processing node on the first data processing end through structured query language query.

[0077] Here, the data processing nodes on the first data processing end include: data cleaning nodes, data computing nodes, data aggregation nodes and other nodes.

[0078] In a possible implementation, step S602 may include: when the first data processing end is a database end, querying the data cleaning nodes, data computing nodes, data aggregation nodes and other nodes on the database end in sequence through the Structured Query Language (SQL), obtaining the data set corresponding to the application to be verified on each node, and determining the first data to be verified from the data set corresponding to the application to be verified on each node according to the needs.

[0079] Step S603: verify the first data to be verified according to the first verification rule to obtain a first verification result.

[0080] Step S604: When the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, obtain the second data to be verified and the corresponding second verification rule of the second data processing end according to the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end.

[0081] Step S605: verify the second data to be verified according to the second verification rule to obtain a second verification result.

[0082] In an embodiment of the present application, when the first data processing end is a database end, the first data to be verified is obtained from each data processing node on the first data processing end through a structured query language query, thereby obtaining the first data to be verified from the first data processing end, and verifying the first data to be verified, thereby realizing verification of the data to be verified corresponding to the application to be verified on the first data processing end.

[0083] Figure 7 This is a flow chart of another data verification method according to an embodiment of the present application. Figure 7 As shown, the process may include:

[0084] Step S701: during the verification process of data flow, first data to be verified and corresponding first verification rules are obtained from the first data processing end.

[0085] Step S702: verify the first data to be verified according to the first verification rule to obtain a first verification result.

[0086] Step S703: When the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, obtain the second data to be verified and the corresponding second verification rule of the second data processing end according to the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end.

[0087] Step S704: verify the second data to be verified according to the second verification rule to obtain a second verification result.

[0088] Step S705: When the first verification result is passed, determine whether the first data processing end is the first data processing end in the data verification direction.

[0089] Here, the first data processing end in the data verification direction may be a front-end page end. When the first verification result is passed, determining whether the first data processing end is the first data processing end in the data verification direction may be implemented by determining whether the first data processing end is a front-end page end when the first verification result is that the first data to be verified meets the first verification rule.

[0090] Step S706: When it is determined that the first data processing end is the first data processing end in the data verification direction, it is determined that there is no quality problem with the data in the entire data flow process.

[0091] In a possible implementation, step S706 includes: when it is determined that the first data processing end is the front-end page end, determining that the data in the middle of the entire data flow has no quality issues.

[0092] Step S707: When the first verification result is passed and it is determined that the first data processing end is not the first data processing end in the data verification direction, it is determined that the first data to be verified has no quality problems, and an alarm message is output; wherein the alarm message is used to indicate that the first data processing end has not provided data that complies with the first verification rules to the third data processing end, so that the third data processing end re-acquires data from the first data processing end to re-perform data flow; the third data processing end is the previous data processing end of the first data processing end in the data verification direction.

[0093] Here, the first data processing end can be any one of the data interface end, the database end and the source data end. When the first data processing end is the data interface end, the third data processing end is the front-end page end; when the first data processing end is the database end, the third data processing end is the data interface end; when the first data processing end is the source data end, the third data processing end is the database end.

[0094] In a possible implementation, when the first verification result is passed and it is determined that the first data processing end is not the first data processing end in the data verification direction, it is determined that the first data to be verified has no quality problems, and an alarm message is output. It can be that when the first data to be verified meets the first verification rule and the first data processing end is not the front-end page end, it is determined that the first data to be verified has no quality problems, and an alarm message is output that the first data end has not provided data that meets the first verification rule to the third data processing end, so that the third data processing end re-acquires data from the first data processing end to re-perform data flow.

[0095] Step S708: Acquire the first data to be checked and the corresponding first checking rule on the first data processing end; wherein the first data to be checked is the data to be checked that is re-transferred to the first data processing end.

[0096] Here, the first data processing end may be the front-end page end; the first data to be verified may be the data obtained by the third data processing end from the first data processing end again, and the data flow is re-transferred to the data to be verified on the first data processing end.

[0097] In some possible implementations, step S708 may include: obtaining the data to be verified and the corresponding verification rules after the flow update on the front-end page.

[0098] Step S709: verify the first data to be verified according to the first verification rule to obtain a first verification result.

[0099] Step S710: When the first verification result is failure and the first data processing end is determined to be the last data processing end in the data verification direction, the first verification result is determined to be that the first data to be verified has a quality problem. In one embodiment, step S710 includes: when the first data processing end is the source data end and the data to be verified on the source data end does not meet the verification rules of the source data end, the first verification result is determined to be that the data to be verified on the data source end has a quality problem.

[0100] Step S711: Feedback a first verification result indicating that the first data to be verified has quality problems to the source data party.

[0101] Here, the source data party refers to the data provider who provides the source data, not the data source.

[0102] In a possible implementation, step S711 includes: feeding back quality problems existing in the data to be checked on the data source end to the source data party.

[0103] In one embodiment, the source data may include signaling data and fault data reported by the ministry-province interface, XDR data of the front-end, China Mobile Network (CMNET) Deep Packet Inspection (DPI) data, and directly collected wireless data and classified data. After the database collects the source data, it processes the collected data, including data calculation, data aggregation, etc., and provides the processed data to the interface. The front-end page displays key scene large screens and related reports by obtaining interface data. Here, the ministry-province interface is the interface between the headquarters network management of China Mobile Communications and the provincial network management system. It is an important part of the network management of the traffic network. It is mainly used to realize the transmission of data requests and response messages between the headquarters network management system and the provincial network management system, and to realize the transmission of large data files from the provincial network management system to the headquarters network management system.

[0104] In this embodiment, based on the entire data flow process, reverse data verification is performed starting from the front-end page to achieve end-to-end data quality improvement, including: first, establishing a process of data extraction, data verification, data alarm, and data processing from the front-end page end, interface end, database end, and source data end to achieve end-to-end closed-loop management of data quality; secondly, summarizing the results of the data verification, focusing on the location and frequency of data quality, analyzing the root causes of data quality, and clarifying the key to improving data quality in each link of the data life cycle management.

[0105] In a possible implementation, the reverse data verification process of the front-end page mainly includes the following three steps, and the functions of each step are as follows:

[0106] Step 1: Data acquisition.

[0107] In one example, data acquisition may be extracting data from a front-end page, interface, database, and source data and filtering and processing the data according to user requirements to filter out data that needs to be checked.

[0108] The acquisition methods include but are not limited to web crawlers, SQL queries, Linux queries, etc. Among them, a web crawler is a program or script that automatically grabs World Wide Web information according to certain rules. A web crawler, also known as a web spider or a network robot, can obtain the data we need from the front-end page. Because of its low technical difficulty and obvious effect, it can obtain a large amount of data; web crawlers mainly use selenium packages and requests packages to crawl relevant data.

[0109] Step 2: Rule determination.

[0110] In one example, rule determination may be to determine whether the data to be checked meets the corresponding determination rules. In a possible implementation, rule determination may be to determine in sequence whether the front-end page end, the interface end, the database end, and the source data end meet the determination rules corresponding to each data processing end. Here, different data processing ends may correspond to different determination rules. The determination rules include but are not limited to the requirements of timeliness, completeness, legality, accuracy, and consistency of data.

[0111] Step 3: Alarm processing.

[0112] In one example, alarm processing can be that when the data to be verified does not meet the judgment rules, an alarm is issued for the data that does not meet the rules as appropriate, and the data that does not meet the judgment rules is processed, the data that meets the rules is obtained again, and the data is re-verified to achieve a closed loop.

[0113] In some embodiments of the present application, a data verification method for application A at time T is proposed, wherein application A is an application that needs to be verified. For application A, the verification rule corresponding to the front-end page end is verification rule 1, the verification rule corresponding to the interface end is verification rule 2, the verification rule corresponding to the database end is verification rule 3, and the verification rule corresponding to the source data end is verification rule 4.

[0114] Figure 8 FIG. 1 is a flow chart of a method for verifying data of application A at time T according to an embodiment of the present application. Figure 8 As shown, the process may include the following steps:

[0115] Step S801: Acquire the data to be verified on the previous page end 80.

[0116] In this embodiment, the data to be checked on the front page end 80 can be obtained through a web crawler, and the selenium package can be used to simulate the browser to log in with a username and password. After the simulated login, by entering a specific assignment in the element attribute (input in the code), the data related to application A in the page after JavaScript rendering can be obtained, that is, the front-end large screen / report related to application A is crawled. The data to be checked is extracted from the crawled data related to application A according to user needs. Here, the specific assignment is manually obtained through the developer tool of the Chrome browser. For example, the specific assignment of the element attributes and attribute elements corresponding to application A can be found in the multiple ID, keys, class, name and other element attributes under the developer tool.

[0117] Step S802: Determine whether the data to be checked on the previous page end 80 can pass the check of the check rule 1; if so, proceed to step S803; if not, proceed to step S804.

[0118] Step S803: End the verification process.

[0119] Step S804: Acquire the data to be verified on the data interface terminal 81.

[0120] In one example, step S804 includes: obtaining multiple data interfaces corresponding to application A through a web crawler, and determining the cookies, authorization and other verification information of multiple URLs corresponding to each data interface, and obtaining the data of each URL position of multiple input interfaces corresponding to application A by using the get and post methods of the requests package. After the relevant data corresponding to application A in the crawled data interface is extracted according to user needs, the data to be verified on the data interface terminal 81 is extracted.

[0121] Step S805: Determine whether the data to be checked on the previous page end 81 can pass the check of the check rule 2; if so, proceed to step S806; if not, proceed to step S808.

[0122] Step S806: Output the first alarm information.

[0123] Here, the first alarm information is used to indicate that the data interface end does not provide data that complies with verification rule 2 to the front-end page end or the front-end page end does not obtain data that complies with verification rule 2 from the data interface end.

[0124] Step S807: Process the data to be verified on the front-end page.

[0125] In one example, processing the data to be verified on the front-end page may include: the front-end page reacquires data on the data interface; the front-end page obtains updated data to be verified; and starting to re-verify the updated data to be verified on the front-end page from the front-end page.

[0126] Step S808: Acquire the data to be verified on the database end 82.

[0127] In one example, step S808 includes: using SQL to query the data related to application A in each data-processed node in the database, and extracting data corresponding to application A on the database end 82 according to user needs to extract the data to be verified on the database end 82.

[0128] Step S809: Determine whether the data to be checked on the database end 82 can pass the check of the check rule 3; if so, proceed to step S810; if not, proceed to step S812.

[0129] Step S810: Output the second alarm information.

[0130] Here, the second alarm information is used to indicate that the database end fails to provide data that complies with verification rule 3 to the data interface end or the data interface end fails to obtain data that complies with verification rule 3 from the database end.

[0131] Step S811: Process the data to be verified at the data interface.

[0132] In one example, processing the data to be verified at the data interface end may include: the data interface end reacquiring the data at the database end; the data interface end obtaining the updated data; and then the front-end page end obtaining the data on the front-end page end after the data interface end updates the data, and then, rechecking the updated data to be verified at the front-end page end starting from the front-end page end.

[0133] Step S812: Acquire the data to be verified on the source data terminal 83.

[0134] In one example, step S812 includes: using Linux shell script or Python associated ftp to obtain real-time streaming data corresponding to the source data, and querying the relevant data of application A from the real-time streaming data, and extracting the data to be verified from the relevant data of application A according to user needs.

[0135] Step S813: Determine whether the data to be checked on the source data terminal 83 can pass the check of the check rule 4; if so, proceed to step S814; if not, proceed to step S816.

[0136] Step S814: Output the third alarm information.

[0137] Here, the third warning information is used to indicate that the source data end does not provide data that complies with verification rule 4 to the database end or the database end does not obtain data that complies with verification rule 4 from the source data end.

[0138] Step S815: Process the data to be verified on the database side.

[0139] In one example, step S815 may include: the database end reacquires the data from the source data end; the database end obtains the updated data; the data interface end and the front-end page end obtain the updated data, the front-end page end obtains the data updated by the data interface end and the data on the front-end page end, and then, rechecks the updated data to be verified on the front-end page end starting from the front-end page end.

[0140] Step S816: Feedback quality problems in the data to be verified on the source data end to the source data party.

[0141] In one implementation, the verification results in the reverse data verification process of the front-end page can be summarized according to different applications, different time granularities, and different judgment rules. The number of occurrences, frequency, and location of the problems in the verification results are analyzed based on the summary results, and the root causes of data quality problems are gradually analyzed to identify the key to improving data quality.

[0142] Fig. 9 Schematic diagram of the structure of the data verification device according to the embodiment of the present application. Fig. 9 As shown, the device may include: a first acquisition module 901, used to acquire first data to be verified and a corresponding first verification rule from a first data processing end during the verification process of data flow; a first verification module 902, used to verify the first data to be verified according to the first verification rule to obtain a first verification result; a second acquisition module 903, used to acquire second data to be verified and a corresponding second verification rule of a second data processing end according to the data verification direction when the first verification result is not passed and it is determined that the first data processing end is not the last data processing end in the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end; a second verification module 904, used to verify the second data to be verified according to the second verification rule to obtain a second verification result.

[0143] In one embodiment, the device also includes: a determination device 905, which is used to determine whether the first data processing end is the first data processing end in the data verification direction when the first verification result is passed; when it is determined that the first data processing end is the first data processing end in the data verification direction, it is determined that there are no quality problems in the data during the entire data flow process.

[0144] In one embodiment, the device also includes: an alarm device 906, which is used to determine that the first data to be verified has no quality problems and output an alarm message when the first verification result is passed and it is determined that the first data processing end is not the first data processing end in the data verification direction; wherein the alarm information is used to characterize that the first data processing end has not provided data that complies with the first verification rules to the third data processing end, so that the third data processing end re-acquires data from the first data processing end to re-perform data flow; the third data processing end is the previous data processing end of the first data processing end in the data verification direction.

[0145] In one embodiment, the device also includes: a first processing device 907, used to obtain the first data to be verified and the corresponding first verification rule on the first data processing end; wherein the first data to be verified is the data to be verified that is re-transferred to the first data processing end; the first data to be verified is verified according to the first verification rule to obtain a first verification result.

[0146] In one embodiment, the device also includes: a third verification device 908, which, when the first verification result is failure and it is determined that the first data processing end is the last data processing end in the data verification direction, determines the first verification result as the first data to be verified having a quality problem; and feeds back the first verification result that the first data to be verified has a quality problem to the source data party.

[0147] In one embodiment, the data processing end in the data flow direction includes the source data end, the database end, the data interface end and the front-end page end in sequence, and the data processing end in the data verification direction includes the front-end page end, the data interface end, the database end and the source data end in sequence.

[0148] In one embodiment, the device further includes: a first acquisition module 909, which is used to acquire the data to be verified from the first data processing end as the first data to be verified according to the identification of the application to be verified; acquire the element attributes corresponding to the application to be verified and the values ​​of the element attributes from the first data processing end through a web crawler; acquire the first data to be verified from the first data processing end according to the element attributes corresponding to the application to be verified and the values ​​of the element attributes; a second acquisition module 910, which is used to acquire the data to be verified from the first data processing end as the first data to be verified according to the identification of the application to be verified; determine multiple data interfaces corresponding to the application to be verified; acquire each data interface from the first data processing end through a web crawler; The location information and verification information of a data interface; obtaining the first data to be verified from the first data processing end according to the location information and verification information of each data interface; a third acquisition module 911, used to obtain the data to be verified from the first data processing end as the first data to be verified according to the identification of the application to be verified; obtaining the first data to be verified from each data processing node on the first data processing end through structured query language query; a fourth acquisition module 912, used to obtain the data to be verified from the first data processing end as the first data to be verified according to the identification of the application to be verified; obtaining the first data to be verified from the first data processing end through the Linux scripting language or through Python connecting to ftp.

[0149] In actual applications, the first acquisition module 901, the first verification module 902, the second acquisition module 903, the second verification module 904, the determination device 905, the alarm device 906, the first processing device 907, the third verification device 908, the first acquisition module 909, the second acquisition module 910, the third acquisition module 911 and the fourth acquisition module 912 can be implemented using the processor in the data verification device, and the above-mentioned processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0150] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0151] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0152] Generally speaking, the computer program instructions corresponding to a data verification method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a U disk. When the computer program instructions corresponding to a data verification method in the storage medium are read or executed by a data verification device, any one of the data verification methods in the aforementioned embodiments is implemented.

[0153] Based on the same technical concept as the above embodiments, see Fig.10 , which shows a schematic diagram of the structure of a data verification device provided in an embodiment of the present application, which may include: a memory 1001 and a processor 1002; wherein,

[0154] The memory 1001 is used to store computer programs and data;

[0155] The processor 1002 is used to execute the computer program stored in the memory to implement any one of the data verification methods in the aforementioned embodiments.

[0156] In practical applications, the memory 1001 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 1002.

[0157] The processor 1002 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understandable that for different augmented reality cloud platforms, the electronic device used to implement the processor function may also be other, which is not limited in the embodiments of the present application.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

Claims

1. A data verification method, characterized in that: The method comprises: In the process of verifying the data flow, first to-be-verified data and corresponding first verification rules are obtained from the first data processing terminal; Verifying the first data to be verified according to the first verification rule to obtain a first verification result; When the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction, the second to-be-verified data and the corresponding second verification rule of the second data processing end are obtained according to the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end; the data processing ends in the data flow direction include the source data end, the database end, the data interface end and the front-end page end in sequence, and the data processing ends in the data verification direction include the front-end page end, the data interface end, the database end and the source data end in sequence; Verifying the second data to be verified according to the second verification rule to obtain a second verification result; If the first verification result is passed, determining whether the first data processing end is the first data processing end in the data verification direction; In the case where it is determined that the first data processing end is the first data processing end in the data verification direction, it is determined that there is no quality problem with the data in the entire data flow process; If the first verification result is passed and it is determined that the first data processing end is not the first data processing end in the data verification direction, it is determined that the first data to be verified has no quality problem and an alarm message is output; wherein, The alarm information is used to indicate that the first data processing end fails to provide data that complies with the first verification rule to the third data processing end, so that the third data processing end re-acquires data from the first data processing end to re-transfer data; the third data processing end is the previous data processing end of the first data processing end in the data verification direction.

2. The method according to claim 1, characterized in that After outputting the warning information, the method further includes: Acquire the first data to be checked and the corresponding first checking rule on the first data processing end; wherein the first data to be checked is the data to be checked that is re-transferred to the first data processing end; The first data to be verified is verified according to the first verification rule to obtain a first verification result.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: When the first verification result is failure and it is determined that the first data processing end is the last data processing end in the data verification direction, the first verification result is determined as the first to-be-verified data having a quality problem; A first verification result indicating that the first data to be verified has quality problems is fed back to the source data party.

4. The method according to claim 1, characterized in that: The obtaining of the first data to be verified from the first data processing end includes: obtaining the data to be verified from the first data processing end as the first data to be verified according to the identifier of the application to be verified; In the case where the first data processing end is a front-end page end, the obtaining the first data to be checked from the first data processing end includes: obtaining, through a web crawler, element attributes corresponding to the application to be checked and the values ​​assigned to the element attributes from the first data processing end; obtaining the first data to be checked from the first data processing end according to the element attributes corresponding to the application to be checked and the values ​​assigned to the element attributes; In the case where the first data processing end is a data interface end, the obtaining of the first data to be checked from the first data processing end includes: determining a plurality of data interfaces corresponding to the application to be checked; obtaining location information and verification information of each data interface from the first data processing end through a web crawler; and obtaining the first data to be checked from the first data processing end according to the location information and verification information of each data interface; In the case where the first data processing end is a database end, the obtaining the first data to be checked from the first data processing end includes: obtaining the first data to be checked from each data processing node on the first data processing end through a structured query language query; In the case that the first data processing end is the source data end, obtaining the first data to be checked from the first data processing end includes: obtaining the first data to be checked from the first data processing end through a Linux scripting language or through Python connecting to ftp.

5. A data processing device, characterized in that: The device comprises: A first acquisition module, used to acquire first to-be-verified data and corresponding first verification rules from a first data processing terminal during the verification process of data flow; A first checking module, configured to check the first to-be-checked data according to the first checking rule to obtain a first checking result; A second acquisition module is used to acquire second to-be-checked data and corresponding second verification rules of a second data processing end according to the data verification direction when the first verification result is failure and it is determined that the first data processing end is not the last data processing end in the data verification direction; the data verification direction is opposite to the direction of data flow; the second data processing end is the next data processing end of the first data processing end; the data processing ends in the data flow direction include a source data end, a database end, a data interface end and a front-end page end in sequence, and the data processing ends in the data verification direction include a front-end page end, a data interface end, a database end and a source data end in sequence; A second verification module, used for verifying the second data to be verified according to the second verification rule to obtain a second verification result; A determination device, used for determining whether the first data processing end is the first data processing end in the data verification direction when the first verification result is passed; and determining that there is no quality problem of the data in the entire data flow process when it is determined that the first data processing end is the first data processing end in the data verification direction; An alarm device is used to determine that the first data to be verified has no quality problems and output an alarm message when the first verification result is passed and it is determined that the first data processing end is not the first data processing end in the data verification direction; wherein the alarm message is used to indicate that the first data processing end has not provided data that complies with the first verification rules to the third data processing end, so that the third data processing end re-acquires data from the first data processing end to re-perform data flow; the third data processing end is the previous data processing end of the first data processing end in the data verification direction.

6. A data processing device, characterized in that: include: Memory and processor, The memory is used to store executable instructions; The processor is used to implement the data verification method described in any one of claims 1 to 4 when executing the executable instructions stored in the memory.

7. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the data verification method described in any one of claims 1 to 4 when executed by a processor.

Images