Method, apparatus, device and medium for data testing

Abstract

The application provides a data test method, device, equipment and medium, which are used for testing data in a visualization report and data in a data warehouse. The method comprises the following steps: obtaining first data under a target index in the visualization report through a preset first program; obtaining second data under the target index in the data warehouse through a preset second program; comparing the first data and the second data to determine a comparison result; and determining whether the data test passes according to the comparison result. The technical problem that data cannot be automatically tested in the related art can be at least solved.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a method, apparatus, equipment and medium for data testing. Background Technology

[0002] With the development of enterprise big data, mobile internet, and multimedia businesses, enterprises are experiencing increasingly larger data volumes, more diverse business scenarios utilizing data, and increasingly varied data structures. Consequently, the demand for data accuracy is constantly expanding, giving rise to data testing. Data testing specifically refers to the process of checking local data structures to ensure that data temporarily stored within a module is complete and correct during program execution.

[0003] Because data testing can ensure data quality, businesses are paying increasing attention to it. However, the inventors have discovered at least the following technical problem in the relevant technologies: the inability to automate data testing.

[0004] Specifically, in related technologies, most automated data monitoring is performed at the interface level, followed by manual data testing. For example, in solutions for testing data in visual reports and data in a data warehouse, data monitoring is typically performed at the interface level of the visual reports. Then, SQL queries are manually written on the enterprise-provided query platform, using engines such as Presto or Hive to calculate the data in the data warehouse. Finally, manual comparisons are made between the calculated data from the data warehouse and the monitored data in the visual reports, based on certain metrics, and the accuracy of the data is determined manually based on the comparison results. This process requires not only manual switching of servers but also, depending on the specific business requirements, the creation of new SQL queries based on the corresponding business logic, followed by manual data comparison and evaluation. In the process of manually writing SQL query statements on the query platform provided by the enterprise: if using Presto, the logic of the SQL query statement should not be too complex and the data volume should not be too large, otherwise errors will occur; if using Hive, although the logic can be very complex and it supports a large amount of data, the slow calculation speed needs to be considered, so the ability to write code manually is required to be high.

[0005] In summary, the analysis shows that the relevant technologies cannot automate data testing, relying heavily on manual operation, resulting in low testing efficiency, significant manpower and time costs, and the need for manual data maintenance. As enterprises accumulate more and more data in their data warehouses and increase the demand for visual data reporting, continuing to rely solely on the testing methods of these technologies will inevitably greatly increase the workload of relevant personnel in data testing. Summary of the Invention

[0006] One objective of this application is to provide a method, apparatus, device, and medium for data testing, at least to solve the technical problem in the related art that automated data testing cannot be achieved.

[0007] To achieve the above objectives, some embodiments of this application provide a data testing method for testing the relationship between data in a visualization report and data in a data warehouse. The method includes: obtaining first data under a target indicator in the visualization report through a preset first program; obtaining second data under the target indicator in the data warehouse through a preset second program; comparing the first data and the second data to determine the comparison result; and determining whether the data test passes based on the comparison result.

[0008] Optionally, obtaining the first data under the target indicator in the visualization report through a preset first procedure includes: determining preset parameters corresponding to the target indicator through a preset first procedure; concatenating the preset parameters to obtain a URL; and obtaining the first data under the target indicator in the visualization report based on the URL. This provides a specific implementation method for obtaining the first data under the target indicator in the visualization report through a preset first procedure, which facilitates flexible and varied implementation of the method for obtaining the first data under the target indicator in the embodiments of this application.

[0009] Optionally, obtaining the second data under the target indicator in the data warehouse through a preset second procedure includes: determining the priority of obtaining the preset parameter through the preset second procedure; and obtaining the second data corresponding to the preset parameter from the Hive base table of the data warehouse according to the priority. Here, a specific implementation method for obtaining the second data under the target indicator in the data warehouse through a preset second procedure is provided, which facilitates a flexible and varied implementation of the method for obtaining the second data under the target indicator in the embodiments of this application.

[0010] Optionally, comparing the first data and the second data to determine the comparison result includes: determining whether the first data and the second data are the same; if the first data and the second data are the same, a comparison result indicating that the characterization test has passed is obtained; if the first data and the second data are different, a comparison result indicating that the characterization test has failed is obtained. By automatically comparing the first data and the second data to determine the comparison result, it is unnecessary for manual comparison of certain indicators based on the calculated data in the data warehouse and the monitored visualization reports, and it is also unnecessary for manual judgment of the correctness of the corresponding data based on the comparison result. This can further reduce labor and time costs, while facilitating data maintenance and data updates.

[0011] Optionally, after obtaining a comparison result indicating that the characterization test failed if the first data and the second data are different, the method further includes: sending an alarm message according to a preset rule. Here, if the first data and the second data are different, automatically sending an alarm message according to a preset rule informs relevant personnel of the abnormal data test result without requiring manual notification, which can further improve the efficiency of data testing.

[0012] Optionally, the method further includes uploading the first preset program and the second preset program to the target cluster and / or the target scheduling platform. By uploading the first preset program and the second preset program to the target cluster and / or the target scheduling platform, when the data requirements for certain target indicators change, relevant personnel can directly modify and update the first preset program and the second preset program based on the target cluster and / or the target scheduling platform. The updated first preset program and the second preset program can then be distributed to other computer devices that need to perform data testing. Since manual modification is eliminated, this improves the efficiency of data testing and also facilitates data maintenance.

[0013] Optionally, after uploading the first preset program and the second preset program to the target cluster and / or the target scheduling platform, the method further includes: upon receiving configuration information, performing the data test operation according to the configuration information; wherein the configuration information includes any one or any combination of the following: task scheduling time information, task dependency information, task failure retry time information, and task failure notification rule information. Here, by allowing relevant personnel to directly modify and update the first preset program and the second preset program by uploading configuration information, a specific implementation method for modifying and updating the first preset program and the second preset program is provided.

[0014] Some embodiments of this application also provide a data testing apparatus, the apparatus comprising: a first data acquisition unit, configured to acquire first data under a target indicator in the visualization report through a preset first program; a second data acquisition unit, configured to acquire second data under the target indicator in the data warehouse through a preset second program; a data comparison unit, configured to compare the first data and the second data to determine the comparison result; and a result determination unit, configured to determine whether the data test passes based on the comparison result.

[0015] Some embodiments of this application also provide a computer device, the device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the method described above.

[0016] Some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods described above.

[0017] Compared to existing technologies, the data testing method provided in this application achieves automated data testing. Specifically, a first program is preset to automatically acquire the first data under the target indicator in the visualization report, and a second program is preset to automatically acquire the second data under the target indicator in the data warehouse. In particular, automatically acquiring the corresponding data through the preset second program can support larger data volumes and more complex calculation logic without manually writing and executing SQL query statements on the enterprise's query platform, thus significantly improving data acquisition efficiency. Since the first and second data can be automatically compared to determine the comparison result, and the data test pass or fail is determined based on the comparison result, there is no need for manual visual comparison of the data and for manual judgment of the correctness of the corresponding data based on the comparison result. Therefore, the efficiency of data testing is significantly improved, greatly reducing labor and time costs. It is worth mentioning that this application embodiment creatively extends the business of the visualization report to the data warehouse level, rather than just staying at the interface level, which has profound significance for realizing automated data testing. Attached Figure Description

[0018] Figure 1 An exemplary flowchart of a data testing method provided in Embodiment 1 of this application;

[0019] Figure 2 An exemplary flowchart of a data testing method provided in Embodiment 2 of this application;

[0020] Figure 3This is an exemplary flowchart of a data testing method provided in Embodiment 3 of this application;

[0021] Figure 4 This is an exemplary structural diagram of a computer device provided in Embodiment Six of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The following terms are used in this document.

[0024] Hive: A data warehouse tool used for data extraction, transformation, loading, storage, querying, and analysis of large-scale data stored in Hadoop. It supports complex and massive data computations, but the larger the data volume and the more complex the logic, the slower the speed.

[0025] Presto: A real-time interactive query engine based on big data, designed to improve the speed of querying massive amounts of data. It is faster than Hive queries when the amount of data being queried is not too large and the SQL is not too complex.

[0026] Spark: A distributed, primarily memory-based big data computing framework suitable for iterative computation. It supports massive amounts of data and complex logical calculations, and is faster than Hive.

[0027] PySpark: An API for Python developers to manipulate Spark, enabling Python developers to write Spark computation programs in Python and submit them to a Spark cluster.

[0028] URL: Uniform Resource Locator. It is a method used in World Wide Web services to specify the location of information.

[0029] Base table: The base table of the data warehouse is a subject-oriented data model generated by data developers after extracting the required data from the business database on a regular / real-time basis, cleaning, transforming, processing and integrating it.

[0030] A data warehouse is a centralized repository for storing structured and semi-structured data for reporting and analysis.

[0031] DWD: Data Warehouse Detail, is the foundational data of a data warehouse and the source data used for calculating business metrics.

[0032] DWB: The full English name is Data Warehouse Base, which is the basic data layer. It stores objective data and is generally used as an intermediate layer. It can be considered as a data layer for a large number of indicators.

[0033] Example 1

[0034] Embodiment 1 of this application provides a data testing method, which may include the following steps, such as... Figure 1 As shown:

[0035] Step S101: According to the preset first procedure, obtain the first data under the target indicator in the visualization report;

[0036] Step S102: Obtain the second data under the target indicator in the data warehouse through a preset second program;

[0037] Step S103: Compare the first data and the second data to determine the comparison result;

[0038] Step S104: Based on the comparison results, determine whether the data test has passed.

[0039] Specifically, regarding step S101, in some examples, the preset first program may be a Python program pre-written by relevant personnel.

[0040] In some examples, the interface data under the target metric can be obtained first from the visualization report, and then the interface data can be filtered to obtain the first data under the target metric. The interface data under the target metric is generally in JSON format. The pre-written Python program can filter out the JSON format interface data, and pandas json can be used to parse the interface data to obtain the first data under the target metric.

[0041] In some examples, there may be multiple target metrics, such as the number of new users, daily paying rate, number of paying users, number of active users, etc. Therefore, there may also be multiple corresponding first data, such as the first data corresponding to the number of active users, the first data corresponding to the payment amount, etc. The specific content of the target metrics can be flexibly selected according to actual business needs, and this application embodiment does not impose specific limitations on this.

[0042] Specifically, regarding step S102, in some examples, the preset second program may be a PySpark program pre-written by relevant personnel.

[0043] In some examples, the target metric data can be obtained from the Hive base table of the data warehouse using the PySpark program, and then filtered to obtain the second data under the target metric calculated by the PySpark program.

[0044] It is worth mentioning that there is no limitation on the order of steps S101 and S102 in the embodiments of this application. That is to say, step S102 can be executed first, followed by step S101, or steps S101 and S102 can be performed simultaneously. The embodiments of this application do not make specific limitations in this regard.

[0045] Regarding step S103, by comparing the first data and the second data, a comparison result of a preset dimension can be obtained, thereby determining whether the data test passes. The preset dimension can be a test of data consistency and / or data integrity. Here, data consistency refers to whether the first data and the second data are the same size when neither is empty, or whether the error between the sizes of the first data and the second data is within an acceptable range. Data integrity refers to whether the first data is empty when the second data is not empty, indicating that the first data is missing or incomplete.

[0046] In some examples, the Python program representing the first preset program and the PySpark program representing the second preset program can be compared. During the comparison, since the first preset program and the second preset program use different programming languages, the comparison can be based on the Python program, for example, using applications such as Pandas; or it can be based on the PySpark program, for example, using applications such as SparkSQL and RDD. The specific application chosen can be flexibly selected based on the data volume and / or data structure of the first and second data; this application does not impose specific limitations on this.

[0047] Specifically, regarding step S104, in some examples, if there are multiple target indicators, it is necessary to compare the first data with the second data corresponding to each target indicator separately to determine multiple comparison results. If all comparison results pass, the data test can be considered passed; if at least one of the comparison results fails, the data test can be considered failed.

[0048] It is not difficult to see that, compared with related technologies, the data testing method provided in this application embodiment achieves automated data testing. Specifically, through a preset first program, the first data under the target indicator in the visualization report is automatically obtained, and through a preset second program, the second data under the target indicator in the data warehouse is automatically obtained. In particular, the automatic acquisition of corresponding data through the preset second program can support larger data volumes and more complex calculation logic, without the need for manual writing and execution of SQL query statements on the enterprise's query platform, thus significantly improving the efficiency of data acquisition. Since the first data and the second data can be automatically compared to determine the comparison result, and the data test is determined based on the comparison result, there is no need for manual visual comparison of the data and for manual judgment of the correctness of the corresponding data based on the comparison result. Therefore, the efficiency of data testing is significantly improved, greatly reducing labor and time costs. It is worth mentioning that, in this application embodiment, the business of the visualization report is creatively extended to the data warehouse level, rather than just staying at the interface level, which has profound significance for realizing automated data testing.

[0049] Example 2

[0050] Embodiment 2 of this application is a further improvement based on Embodiment 1. Specifically, in Embodiment 2, the step of obtaining the first data under the target indicator in the visualization report through a preset first procedure may further include the following steps: Figure 2 As shown:

[0051] Step S1011: Determine the preset parameters corresponding to the target indicator through a preset first program;

[0052] Step S1012: Concatenate the parameters according to the preset parameters to obtain the URL;

[0053] Step S1013: Obtain the first data under the target indicator in the visualization report according to the URL.

[0054] Specifically, in some examples, for game-related businesses, the preset parameters may include, but are not limited to: server region, data acquisition time period, registration platform / behavior occurrence platform, country, and whether to remove secondary accounts, etc. The server region may include the Chinese server, Asian server, European server, etc., corresponding to servers in China, Asia, and Europe, respectively. The data acquisition time period may be the day before the current time, the fifteen days before the current time, etc. The registration platform / behavior occurrence platform refers to the platform used by the player when registering for the first time and the platform actually used after registration, respectively. The platform may include: iOS tablets, iOS phones, Android phones, PSP devices, etc. The country may include China, the United States, South Korea, etc., thereby allowing the acquisition of relevant data for the corresponding business in a specific country. Since many players have both main and secondary accounts, the option to remove secondary accounts can also be selected. Of course, the above preset parameters are only examples. In some other examples, the preset parameters may also include other content, such as account type, period dimension (by day / by week / by month), etc., which are not specifically limited in this embodiment. It is worth mentioning that since the preset parameters can include the server region, relevant data can be obtained based on the preset parameters without the need for manual switching of servers, thereby further reducing labor costs and improving data acquisition efficiency.

[0055] Specifically, in some examples, different target metrics can correspond to different preset parameters. For example, the preset parameters for target metric 1 may include preset parameter a, preset parameter b, and preset parameter c; the preset parameters for target metric 2 may include preset parameter a, preset parameter c, and preset parameter d.

[0056] Specifically, in some examples, the preset parameters may include mandatory parameters and optional parameters. The specific content of the mandatory and optional parameters can be determined mainly based on different business types and actual needs; this application embodiment does not impose specific limitations on this. For example, in some examples, the mandatory parameters may include: the time period for data acquisition, the server / region, and whether to remove secondary accounts; the optional parameters may include: the registration platform / behavior occurrence platform, and the country. Furthermore, in some examples, the system may pre-store default menus of mandatory and optional parameters corresponding to different target indicators, and relevant personnel can adjust these by adding, deleting, or modifying them based on the default menus.

[0057] Furthermore, in some examples, the URL obtained by concatenating the preset parameters includes protocol, host, port, and path information. The path information may include, for example, the data acquisition time period, the registration platform / behavior occurrence platform, and the country. For instance, the data acquisition time period may include a start time and an end time. The various parameters can be connected using the '&' symbol. Therefore, after obtaining the URL, requesting the URL will retrieve the first data point under the target metric in the visualization report.

[0058] It is not difficult to see that, compared with related technologies, the data testing method provided in this application provides a specific implementation method for obtaining the first data under the target indicator in the visualization report through a preset first procedure, which is conducive to the flexible and varied implementation of the method of obtaining the first data under the target indicator in this application embodiment.

[0059] Example 3

[0060] Embodiment 3 of this application is a further improvement based on Embodiment 1. Specifically, in Embodiment 3, the step of obtaining the second data under the target indicator in the data warehouse through a preset second procedure may further include the following steps: Figure 3 As shown:

[0061] Step S1021: Determine the priority of obtaining the preset parameters through a preset second procedure;

[0062] Step S1022: Based on the priority, retrieve the second data corresponding to the preset parameters from the Hive base table of the data warehouse.

[0063] Specifically, in some examples, the application layer data in Embodiment 3 of this application differs from that in Embodiment 2. The corresponding data cannot be directly obtained based on the preset parameters; instead, it needs to be calculated. Since the parameters in the preset parameters may have dependencies—for example, data B can only be determined based on data A when data A is obtained—the priority of obtaining the preset parameters needs to be determined first through the preset second procedure. Alternatively, in some other examples, by prioritizing the acquisition of the preset parameters, the second data corresponding to the preset parameters can be obtained with higher efficiency.

[0064] Furthermore, in some examples, data corresponding to the preset parameters can be retrieved from the Hive base tables of the DWD and DWB layers of the data warehouse according to the priority. This data is then grouped, aggregated, matched, and calculated to obtain second data corresponding to the preset parameters. The calculation may refer to the number of decimal places retained, etc. Specifically, the preset parameters can be used as filtering conditions based on target metrics, such as the data acquisition time period, registration platform / behavior occurrence platform, country, and whether secondary accounts are removed, to obtain the second data corresponding to the preset parameters.

[0065] Furthermore, in some examples, such as when the target metric is a measure of the daily new user count, after determining the priority of obtaining the preset parameters, data corresponding to the preset parameters is retrieved from the Hive base tables of the DWD and DWB layers of the data warehouse according to the priority. The preset parameters can be a specific day or several days, server / region, whether to remove secondary accounts, registration date, active date, etc., thereby counting the number of user IDs each day to obtain the daily new user count.

[0066] In some cases, the amount of data for the second data under the target indicator may be greater than the amount of data for the first data under the target indicator. For example, the amount of data for the first data under the target indicator may be the data from the previous day, while the amount of data for the second data under the target indicator may be the data from the previous 15 days. Therefore, the amount of data for the second data under the target indicator can be filtered first to select the data from the previous day, which is beneficial for making full use of computing resources.

[0067] It should be noted that the embodiments of this application may also be improvements made based on Embodiment 2.

[0068] It is not difficult to see that, compared with related technologies, the data testing method provided in this application embodiment offers a specific implementation method for obtaining the second data under the target indicator in the data warehouse through a preset second procedure, which is conducive to the flexible and varied implementation of the method of obtaining the second data under the target indicator in this application embodiment.

[0069] Example 4

[0070] Embodiment 4 of this application is a further improvement on Embodiment 1. Specifically, in Embodiment 4, comparing the first data and the second data to determine the comparison result may further include the following steps: determining whether the first data and the second data are the same; if the first data and the second data are the same, a comparison result indicating that the characterization test has passed is obtained; if the first data and the second data are different, a comparison result indicating that the characterization test has failed is obtained.

[0071] Furthermore, in some embodiments of this application, after obtaining a comparison result indicating that the characterization test has failed if the first data and the second data are different, the method may further include: sending an alarm message according to a preset rule. Here, if the first data and the second data are different, automatically sending an alarm message according to a preset rule informs relevant personnel of the abnormal data test result without requiring manual notification, which can further improve the efficiency of data testing.

[0072] Specifically, in some examples, if the first data and the second data are inconsistent, or if there is missing data, it indicates that the first data and the second data are different, and an alarm message can be sent according to a preset rule; otherwise, if the first data and the second data are the same, the operation of sending an alarm message according to the preset rule will not be executed. The preset rule can refer to a pre-set target address, such as the address of the communication software tool used by relevant staff, like a WeChat group or QQ group; the preset rule can also refer to email notification, in which case relevant staff need to write the relevant code for sending emails and package it into a JAR file or encapsulate it into an interface. The specific content, data format, and data style of the alarm message can be customized according to the actual preferences of relevant staff, and this application embodiment does not specifically limit this.

[0073] Furthermore, in some embodiments of this application, the method may further include: uploading the first preset program and the second preset program to the target cluster and / or the target scheduling platform.

[0074] Specifically, in some examples, the Python program representing the preset first program and the PySpark program representing the preset second program can be uploaded to the target cluster and / or the target scheduling platform. In this way, the target cluster and / or the target scheduling platform can distribute the preset first program and the preset second program to other computer devices that need to perform data testing. When the data requirements for certain target metrics change, relevant personnel can directly modify and update the preset first program and the preset second program based on the target cluster and / or the target scheduling platform, and then distribute the updated preset first program and preset second program to other computer devices that need to perform data testing.

[0075] Furthermore, in some embodiments of this application, after uploading the first preset program and the second preset program to the target cluster and / or target scheduling platform, the method may further include: upon receiving configuration information, performing the data test operation according to the configuration information; wherein, the configuration information includes any one or any combination of the following: task scheduling time information, task dependency information, task failure retry time information, and task failure notification rule information. It is worth noting that the names of the target cluster and / or target scheduling platform may differ between different enterprises; however, this does not affect the receipt of the configuration information or the execution of the data test operation according to the configuration information. Here, by uploading the first preset program and the second preset program to the target cluster and / or target scheduling platform, relevant personnel can issue configuration information based on the target cluster and / or target scheduling platform, thereby flexibly implementing some operations that originally required manual control. For example, there is no need to manually switch servers, execute pre-written SQL queries, or manually compare certain indicators with the data in the data warehouse and the data in the monitored visualization reports. There is also no need to manually judge whether the corresponding data is correct based on the comparison results or to manually issue alarm information. This can significantly improve the efficiency of data testing, further reduce labor and time costs, and facilitate data maintenance and updates.

[0076] It should be noted that the embodiments of this application may also be improvements made based on Embodiment 2 and / or Embodiment 3.

[0077] It is not difficult to see that, compared with related technologies, the data testing method provided in this application embodiment automatically compares the first data and the second data to determine the comparison result. This eliminates the need for manual comparison of certain indicators based on the calculated data in the data warehouse and the monitored data in the visualization report, and also eliminates the need for manual judgment of whether the corresponding data is correct based on the comparison result. This can further reduce labor and time costs, while also facilitating data maintenance and data updates.

[0078] Example 5

[0079] Embodiment 5 of this application provides a data testing apparatus. The apparatus includes a first data acquisition unit, a second data acquisition unit, a data comparison unit, and a result determination unit.

[0080] Specifically, the first data acquisition unit is used to acquire the first data under the target indicator in the visualization report through a preset first program; the second data acquisition unit is used to acquire the second data under the target indicator in the data warehouse through a preset second program; the data comparison unit is used to compare the first data and the second data to determine the comparison result; and the result determination unit is used to determine whether the data test passes based on the comparison result.

[0081] Optionally, in some examples, the first data acquisition unit may further include a first determining module, a splicing module, and a first data acquisition module. The first determining module is used to determine preset parameters corresponding to the target indicator through a preset first procedure; the splicing module is used to splice the parameters according to the preset parameters to obtain a URL; the first data acquisition module is used to acquire the first data under the target indicator in the visualization report according to the URL.

[0082] Optionally, in some examples, the second data acquisition unit may further include a second determining module and a second data acquisition module. The second data acquisition unit is configured to determine the priority of acquiring the preset parameters through a preset second procedure; the second data acquisition module is configured to acquire second data corresponding to the preset parameters from the Hive base table of the data warehouse according to the priority.

[0083] Optionally, in some examples, the data comparison unit may further include a judgment module and a comparison result determination module. The judgment module is used to determine whether the first data and the second data are the same; the comparison result determination module is used to obtain a comparison result indicating that the characterization test has passed when the first data and the second data are the same, and to obtain a comparison result indicating that the characterization test has failed when the first data and the second data are different.

[0084] Optionally, in some examples, the device further includes an alarm unit, which is used to send an alarm message according to a preset rule after the result determination module determines that the first data and the second data are different and obtains a comparison result indicating that the characterization test has failed.

[0085] Optionally, in some examples, the device may further include an uploading unit for uploading the first preset program and the second preset program to the target cluster and / or the target scheduling platform.

[0086] Optionally, in some examples, the device may further include a configuration information receiving unit, which is configured to, after the uploading unit uploads the first preset program and the second preset program to the target cluster and / or the target scheduling platform, if the target cluster and / or the target scheduling platform receives the configuration information, perform the data test operation according to the configuration information; wherein, the configuration information may include any one of the following or any combination thereof: task scheduling time information, task dependency information, task failure retry time information, and task failure notification rule information.

[0087] This application is an apparatus embodiment corresponding to any one or more of the method embodiments in Embodiments 1 to 4 above. The technical details provided in each of the above method embodiments are applicable here, and will not be repeated here to avoid repetition.

[0088] It is not difficult to see that, compared with related technologies, the data testing apparatus provided in this application embodiment achieves automated data testing. Specifically, through a preset first program, it automatically acquires the first data under the target indicator in the visualization report, and through a preset second program, it automatically acquires the second data under the target indicator in the data warehouse. Since the corresponding data is automatically acquired through the preset first and second programs, it can support larger data volumes and more complex calculation logic, without the need for manual writing and execution of SQL query statements on the enterprise's query platform, thus significantly improving the efficiency of data acquisition. Since the first and second data can be automatically compared, the comparison result can be determined, and the data test can be determined based on the comparison result, eliminating the need for manual visual comparison of the data and manual judgment of the correctness of the corresponding data based on the comparison result, thus significantly improving the efficiency of data testing and greatly reducing labor and time costs. It is worth mentioning that, in this application embodiment, the business of the visualization report is creatively extended to the data warehouse level, rather than just remaining at the interface level, which has profound significance for realizing automated data testing.

[0089] Example 6

[0090] Embodiment 6 of this application provides a computer device, the structure of which is as follows: Figure 4 As shown, the device includes a memory 11 for storing computer-readable instructions and a processor 12 for executing the computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor is triggered to execute the virtual content distribution method.

[0091] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0092] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0093] In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0094] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0095] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0096] In another aspect, embodiments of this application also provide a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable medium carries one or more computer-readable instructions, which may be executed by a processor to implement the steps of the methods and / or technical solutions of the various embodiments of this application.

[0097] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0098] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0099] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0100] Furthermore, this application also provides a computer program stored in a computer device, which causes the computer device to execute the method for executing the control code.

[0101] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0102] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A method of data testing, characterized by, A method for testing between data in a visual report and data in a data warehouse, the method comprising: obtaining first data under a target index in the visual report through a preset first program; the obtaining first data under the target index in the visual report through the preset first program comprising: determining a preset parameter corresponding to the target index through the preset first program; obtaining a URL by splicing according to the preset parameter; and obtaining first data under the target index in the visual report according to the URL; obtaining second data under the target index in the data warehouse through a preset second program; the obtaining second data under the target index in the data warehouse through the preset second program comprising: determining a priority of obtaining the preset parameter through the preset second program; and obtaining second data corresponding to the preset parameter from a Hive base table of the data warehouse according to the priority; comparing the first data and the second data to determine a comparison result; determining whether the data test passes according to the comparison result.

2. The method of claim 1, wherein, The comparing the first data and the second data to determine a comparison result comprises: determining whether the first data and the second data are the same; if the first data and the second data are the same, obtaining a comparison result representing that the test passes; if the first data and the second data are different, obtaining a comparison result representing that the test fails.

3. The method of claim 2, wherein, After the if the first data and the second data are different, obtaining a comparison result representing that the test fails, the method further comprises: sending an alarm information according to a preset rule.

4. The method of claim 1, wherein, The preset first program is a pre-written Python program; and the preset second program is a pre-written Pyspark program.

5. The method of claim 1, wherein, The comparing the first data and the second data to determine a comparison result comprises: if the number of target indexes is multiple, comparing the first data and the second data corresponding to the target indexes respectively to determine multiple comparison results; The determining whether the data test passes according to the comparison result comprises: if all the comparison results pass, determining that the data test passes; and if at least one of the comparison results fails, determining that the data test fails.

6. The method of claim 1, wherein, The method further comprises: uploading the preset first program and the preset second program to a target cluster and / or a target scheduling platform.

7. The method of claim 6, wherein, After the uploading the preset first program and the preset second program to the target cluster and / or the target scheduling platform, the method further comprises: after receiving configuration information, performing an operation of the data test according to the configuration information; wherein the configuration information comprises any one or any combination of the following: time information of task scheduling, task dependency information, task failure retry time information, and task failure notification rule information.

8. An apparatus for data testing, characterized by The apparatus comprises: The first data acquisition unit is configured to acquire first data under a target index in a visual report by a preset first program. The first data acquisition unit comprises a first determination module, a splicing module and a first data acquisition module. The first determination module is configured to determine a preset parameter corresponding to the target index by the preset first program. The splicing module is configured to splice according to the preset parameter to obtain a URL. The first data acquisition module is configured to acquire the first data under the target index in the visual report according to the URL. The second data acquisition unit is configured to acquire second data under the target index in a data warehouse by a preset second program. The second data acquisition unit comprises a second determination module and a second data acquisition module. The second data acquisition unit is configured to determine a priority of acquiring the preset parameter by the preset second program. The second data acquisition module is configured to acquire second data corresponding to the preset parameter from a Hive base table of the data warehouse according to the priority. The data comparison unit is configured to compare the first data and the second data to determine a comparison result. The result determination unit is configured to determine whether data testing is passed according to the comparison result.

9. A computer device, comprising: The device comprises: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the method of any one of claims 1 to 7.

10. A computer readable medium having stored thereon computer program instructions executable by a processor to implement the method of any one of claims 1 to 7.

Images