Method and device for testing iterative task of data warehouse

Abstract

The invention discloses a test method and device for an iterative task of a data warehouse, and the method comprises the steps: updating an associated task of a target task through responding to pre-release instruction information for the target task, and obtaining an updated task after the associated task is updated; obtaining historical metadata and update information of the update task; the historical metadata represents historical operation index information of the target historical task; performing index analysis based on the historical metadata and the update information to obtain an index analysis result of the update task; the index analysis result represents the execution state of the update task; under the condition that the index analysis result does not meet the preset index condition, index abnormal information in the updating task is determined based on the index analysis result; and generating a test report corresponding to the associated task based on the index exception information. When the pre-release is operated, the test task is generated, so that the task online test time is shortened, the release process automation is realized, and the problem of how to improve the task release efficiency of the data warehouse is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a testing method and device for data warehouse iterative tasks. Background Art

[0002] In the era of rapid development of big data technology and artificial intelligence technology, data is growing explosively, and more and more businesses need to use big data capabilities for data analysis. Data is processed and handled through tasks to build a data warehouse. The construction of a data warehouse is often a continuous construction process and involves a large number of tasks. As the number of data warehouse tasks increases, the links become more and more complex. In order to ensure the accurate execution of the data warehouse, each task released by the data warehouse needs to be tested.

[0003] In the prior art, the impact of newly added fields in a task on downstream tasks is usually tested by traversing in sequence, and the impact of newly added dependencies in a task on the output time of the leaf tasks of the task is tested. The corresponding warning method is matched according to the performance results to prompt the developer. There is a problem that the task release test of the data warehouse takes a long time and the release efficiency is low. Therefore, a test method for data warehouse iterative tasks that can improve the efficiency of data warehouse task release is urgently needed to solve the above problems. Summary of the invention

[0004] The present application provides a testing method for data warehouse iterative tasks. When running a pre-release, it automatically detects whether the release of the task causes the downstream task to exceed the latest task output time agreed with the demander based on the collected metadata, whether it will cause the downstream task to report an error, and generates a test task to feedback to the development, thereby shortening the test time for task launch, automating the data warehouse release process, improving the timeliness of task release, and solving the problem of how to improve the efficiency of data warehouse task release.

[0005] According to one aspect of the present application, a test method for a data warehouse iteration task is proposed, the method comprising:

[0006] In response to the pre-release instruction information for the target task, the associated task of the target task is updated to obtain the updated task after the associated task is updated; the target task is the task after the target historical task is modified, and the target historical task is used to manage the target data warehouse;

[0007] Acquire historical metadata and update information of the update task; the historical metadata represents historical operation indicator information of the target historical task;

[0008] Performing an index analysis based on the historical metadata and the update information to obtain an index analysis result of the update task; the index analysis result represents the execution status of the update task;

[0009] In the case where the indicator analysis result does not meet the preset indicator condition, determining indicator abnormality information in the update task based on the indicator analysis result;

[0010] A test report corresponding to the associated task is generated based on the indicator abnormality information.

[0011] In a possible implementation, obtaining historical metadata includes:

[0012] Obtaining an average running time of the target historical task within a preset time, and determining the average running time as the running time of the target historical task;

[0013] Obtain an average start running time of the target historical task within the preset time, and determine the average start running time as the start running time of the target historical task;

[0014] Obtaining an average task output time of the target historical task within the preset time, and determining the average task output time as the target historical task output time;

[0015] Obtaining the usage records of the target historical tasks and generating a downstream task list;

[0016] Obtaining the usage records of the leaf tasks of the target historical task, and generating a downstream leaf task list;

[0017] The historical metadata is obtained based on the task running time, the task start running time, the task output time, the downstream task list and the downstream leaf task list.

[0018] In a possible implementation, the update information includes at least one of first update information and second update information, the first update information indicates that the update task has a newly added field, and the second update information indicates that the update task has a newly added dependent task;

[0019] The performing of the indicator analysis based on the historical metadata and the update information to obtain the indicator analysis result of the update task includes:

[0020] In the case where the update information is the first update information, query analysis is performed based on the newly added field and the downstream task list to obtain a query analysis result, and the query analysis result is determined to be an indicator analysis result of the update task;

[0021] In the case where the update information is the second update information, the output time is calculated based on the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the downstream leaf task list to obtain the output time boundary value of the update task, and the output time boundary value is determined as the indicator analysis result of the update task.

[0022] In a possible implementation, the downstream task list contains historical task table data of the target historical task.

[0023] The query analysis is performed based on the newly added field and the downstream task list to obtain the query analysis result, including:

[0024] Acquire update table data of the update task; the update table data is generated by performing field addition processing on the historical task table data based on the newly added field;

[0025] Replace the historical task table data in the downstream task list based on the update table data to obtain a target downstream task list;

[0026] A query analysis is performed on the target downstream task list based on the explanation command to obtain the query analysis result.

[0027] In a possible implementation, the query analysis result includes a first query analysis result and a second query analysis result, wherein the first query analysis result indicates that a query on a newly added field of the update task is successful, and the second query analysis result indicates that a query on a newly added field of the update task is failed.

[0028] The method further comprises:

[0029] If the indicator analysis result is the first query analysis result, determining that the indicator analysis result meets the preset indicator condition;

[0030] If the indicator analysis result is the second query analysis result, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0031] In a possible implementation, the downstream leaf task list includes at least one leaf task.

[0032] The output time is calculated based on the newly added dependent task, the target historical task running time, the target historical task start running time, the target historical task output time and the downstream leaf task list to obtain the output time boundary value of the update task, including:

[0033] Obtain the running time of the update task, the running time of the newly added dependent task, and the historical leaf task output time of each leaf task;

[0034] The output time is calculated based on the running time of the update task, the running time of the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the historical leaf task output time of each leaf task to obtain the test leaf task output time of each leaf task;

[0035] In the case where the test leaf task output time is greater than the preset leaf task output time, the test leaf task output time greater than the preset leaf task output time is determined as the output time boundary value of the update task.

[0036] In a possible implementation, the method further includes:

[0037] Get the preset demand output time;

[0038] Compare the preset demand output time with the output time boundary value of the update task to obtain a comparison result;

[0039] If the comparison result indicates that the output time boundary value of the update task is less than or equal to the preset required output time, it is determined that the indicator analysis result meets the preset indicator condition;

[0040] If the comparison result indicates that the output time boundary value of the update task is greater than the preset required output time, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0041] In a possible implementation, determining the indicator abnormality information in the update task based on the indicator analysis result includes:

[0042] When the indicator analysis result indicates that the output time boundary value of the update task is greater than the preset required output time, an abnormality analysis is performed based on the running time of the update task and the running time of the newly added dependent task to obtain an abnormality analysis result;

[0043] In a case where the abnormal analysis result indicates that the running time of the update task is greater than a first preset running time, determining the running time of the update task as the indicator abnormality information in the update task;

[0044] When the abnormal analysis result indicates that the running time of the newly added dependent task is greater than the second preset running time, the running time of the newly added dependent task is determined as the indicator abnormality information in the update task.

[0045] In a possible implementation, in response to the pre-release instruction information for the target task, updating the associated task of the target task to obtain the updated task after the associated task is updated includes:

[0046] Obtaining pre-release instruction information for a target task input by the subject;

[0047] In a development environment, the associated tasks of the target task are updated to obtain updated tasks after the associated tasks are updated.

[0048] On the other hand, a testing device for a data warehouse iteration task is provided, the device comprising:

[0049] A pre-release module, for updating the associated tasks of the target task in response to the pre-release instruction information for the target task, and obtaining an updated task after the associated tasks are updated; the target task is a task after the target historical task is modified, and the target historical task is used to manage the target data warehouse;

[0050] A test information acquisition module, used to acquire historical metadata and update information of the update task; the historical metadata represents historical operating indicator information of the target historical task;

[0051] An indicator analysis module, configured to perform an indicator analysis based on the historical metadata and the update information to obtain an indicator analysis result of the update task; the indicator analysis result represents an execution status of the update task;

[0052] An indicator abnormality information determination module, used to determine the indicator abnormality information in the update task based on the indicator analysis result when the indicator analysis result does not meet the preset indicator condition;

[0053] A test report generation module is used to generate a test report corresponding to the associated task based on the indicator abnormality information.

[0054] In a possible implementation, the test information acquisition module includes a historical metadata acquisition unit, and the historical metadata acquisition unit is used to:

[0055] Obtaining an average running time of the target historical task within a preset time, and determining the average running time as the running time of the target historical task;

[0056] Obtain an average start running time of the target historical task within the preset time, and determine the average start running time as the start running time of the target historical task;

[0057] Obtaining an average task output time of the target historical task within the preset time, and determining the average task output time as the target historical task output time;

[0058] Obtaining the usage records of the target historical tasks and generating a downstream task list;

[0059] Obtaining the usage records of the leaf tasks of the target historical task, and generating a downstream leaf task list;

[0060] The historical metadata is obtained based on the task running time, the task start running time, the task output time, the downstream task list and the downstream leaf task list.

[0061] In a possible implementation, the update information includes at least one of first update information and second update information, the first update information indicates that the update task has a newly added field, and the second update information indicates that the update task has a newly added dependent task;

[0062] The indicator analysis module is used to:

[0063] In the case where the update information is the first update information, query analysis is performed based on the newly added field and the downstream task list to obtain a query analysis result, and the query analysis result is determined to be an indicator analysis result of the update task;

[0064] In the case where the update information is the second update information, the output time is calculated based on the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the downstream leaf task list to obtain the output time boundary value of the update task, and the output time boundary value is determined as the indicator analysis result of the update task.

[0065] In a possible implementation, the downstream task list contains historical task table data of the target historical task, and the indicator analysis module includes a first indicator analysis unit.

[0066] The query analysis unit is used to:

[0067] Acquire update table data of the update task; the update table data is generated by performing field addition processing on the historical task table data based on the newly added field;

[0068] Replace the historical task table data in the downstream task list based on the update table data to obtain a target downstream task list;

[0069] A query analysis is performed on the target downstream task list based on the explanation command to obtain the query analysis result.

[0070] In a possible implementation, the query analysis result includes a first query analysis result and a second query analysis result, wherein the first query analysis result indicates that a query on a newly added field of the update task is successful, and the second query analysis result indicates that a query on a newly added field of the update task is failed.

[0071] The first indicator analysis unit is further used for:

[0072] If the indicator analysis result is the first query analysis result, determining that the indicator analysis result meets the preset indicator condition;

[0073] If the indicator analysis result is the second query analysis result, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0074] In a possible implementation, the downstream leaf task list includes at least one leaf task, and the indicator analysis module further includes a second indicator analysis unit.

[0075] The second indicator analysis unit is used to:

[0076] Obtain the running time of the update task, the running time of the newly added dependent task, and the historical leaf task output time of each leaf task;

[0077] The output time is calculated based on the running time of the update task, the running time of the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the historical leaf task output time of each leaf task to obtain the test leaf task output time of each leaf task;

[0078] In the case where the test leaf task output time is greater than the preset leaf task output time, the test leaf task output time greater than the preset leaf task output time is determined as the output time boundary value of the update task.

[0079] In a possible implementation manner, the second indicator analysis unit is further used to:

[0080] Get the preset demand output time;

[0081] Compare the preset demand output time with the output time boundary value of the update task to obtain a comparison result;

[0082] If the comparison result indicates that the output time boundary value of the update task is less than or equal to the preset required output time, it is determined that the indicator analysis result meets the preset indicator condition;

[0083] If the comparison result indicates that the output time boundary value of the update task is greater than the preset required output time, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0084] In a possible implementation, the indicator abnormal information determination module is used to:

[0085] When the indicator analysis result indicates that the output time boundary value of the update task is greater than the preset required output time, an abnormality analysis is performed based on the running time of the update task and the running time of the newly added dependent task to obtain an abnormality analysis result;

[0086] In a case where the abnormal analysis result indicates that the running time of the update task is greater than a first preset running time, determining the running time of the update task as the indicator abnormality information in the update task;

[0087] When the abnormal analysis result indicates that the running time of the newly added dependent task is greater than the second preset running time, the running time of the newly added dependent task is determined as the indicator abnormality information in the update task.

[0088] In a possible implementation, the pre-release module is used to:

[0089] Obtaining pre-release instruction information for a target task input by the subject;

[0090] In a development environment, the associated tasks of the target task are updated to obtain updated tasks after the associated tasks are updated.

[0091] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a testing method for a data warehouse iteration task of any of the above aspects.

[0092] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement a testing method for a data warehouse iteration task as described in any of the above aspects.

[0093] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the test method of the data warehouse iteration task in any of the above aspects.

[0094] The embodiment of the present application updates the associated tasks of the target task in response to the pre-release instruction information for the target task, and obtains the updated task after the associated task is updated; the target task is a task after the target historical task is modified, and the target historical task is used to manage the target data warehouse; obtains the historical metadata and the update information of the update task; the historical metadata represents the historical operation index information of the target historical task; performs index analysis based on the historical metadata and the update information to obtain the index analysis result of the update task; the index analysis result represents the execution status of the update task; when the index analysis result does not meet the preset index condition, determines the index abnormal information in the update task based on the index analysis result; generates a test report corresponding to the associated task based on the index abnormal information. When running the pre-release, it automatically detects whether the task release causes the downstream task to exceed the latest output time of the task agreed with the demand side according to the collected metadata, whether it will cause the downstream task to report an error, generates a test task and feeds it back to the development, shortens the test time of the task launch, realizes the automation of the data warehouse release process, improves the timeliness of task release, and solves the problem of how to improve the efficiency of data warehouse task release. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0096] Figure 1 It is a flowchart of a method for testing a data warehouse iteration task provided by an embodiment of the present application;

[0097] Figure 2 It is a flowchart of a test method for data warehouse iteration tasks provided in an embodiment of the present application;

[0098] Figure 3 It is a structural block diagram of a testing device for data warehouse iteration tasks provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0100] With the advent of the big data era, data is growing explosively. More and more businesses need to use big data capabilities for data analysis. The tasks of data warehouses are increasing, the links are becoming more and more complex, and each release test takes a long time. We need to find a way to improve the efficiency of data warehouse releases, instead of checking each release task one by one downstream. On the one hand, we can avoid errors in the next day's tasks due to incomplete testing, and on the other hand, we can avoid the failure of leaf tasks to be produced on time and affect business use. More importantly, we can form a systematic processing solution to ensure efficient and correct release of data warehouse tasks.

[0101] The present application provides a method for testing data warehouse iteration tasks. When running pre-release, the method automatically detects whether the release of the task causes the downstream task to exceed the latest task output time agreed with the demander based on the collected metadata, whether it causes the downstream task to report an error, and generates a test task to feedback to the development, thus shortening the test time for the task to be launched. Figure 1 The testing method of the data warehouse iteration task includes steps S101 to S109.

[0102] In step S101, in response to pre-release instruction information for a target task, the associated task of the target task is updated to obtain an updated task after the associated task is updated; the target task is a task after modifying a target historical task, and the target historical task is used to manage a target data warehouse.

[0103] In a possible implementation, in response to the pre-release instruction information for the target task, updating the associated task of the target task to obtain the updated task after the associated task is updated includes:

[0104] Obtaining pre-release instruction information for a target task input by the subject;

[0105] In a development environment, the associated tasks of the target task are updated to obtain updated tasks after the associated tasks are updated.

[0106] In a possible implementation, the target data warehouse is used to support the business needs of users to provide users with data related to decision analysis. It is created for analytical reporting and decision support purposes.

[0107] In a possible implementation, the target historical task is an SQL (Structured Query Language) task of the target data warehouse at a historical time. Specifically, SQL is a database query and programming language used to access data and query, update and manage relational database systems.

[0108] In one possible implementation, SQL tasks are implemented by writing SQL code to achieve specific data processing requirements. After writing, you need to configure the task scheduling information, including task name, scheduling period, dependencies, etc., to ensure that the task can be executed as planned. After configuration, the task can be submitted and a release package can be generated. After testing, the task will be released to the production environment.

[0109] In a possible implementation, the target task is a SQL task that is a modification of the target historical task. Specifically, the target task is a SQL task that adds a field to the target historical task, the target task may also be a SQL task that adds a dependency to the target historical task, or the target task may also be a SQL task that adds a field and a dependency to the target historical task.

[0110] In a possible implementation, adding a new field to the target historical task includes adding a new field to the data table corresponding to the target historical task according to business requirements to expand the function of the data table. Specifically, the user data table corresponding to the target historical task contains the user's name and email address. If the target task needs to add information such as the user's phone number according to business requirements, it is necessary to add a new field to the data table corresponding to the target historical task to achieve the business requirements that the target task needs to complete.

[0111] In a possible implementation, the target historical task is added with a dependency, including adding a data dependency to the data corresponding to the target historical task according to business needs. Data dependency is a mutual constraint relationship between data, a semantic embodiment, and is mainly divided into functional dependency, multi-valued dependency and connection dependency.

[0112] In a possible implementation, in a development environment, a developer runs pre-release instruction information for a target task, updates an associated task of the target task, and obtains an updated task after the associated task is updated.

[0113] In a possible implementation, while the target task is being released, it is necessary to update the associated tasks of the target task to obtain the updated tasks.

[0114] In a possible implementation manner, the associated tasks of the target task include downstream tasks of the target task.

[0115] Under this implementation, in a development environment, in response to pre-release instruction information for a target task, the associated tasks of the target task are updated to obtain updated tasks after the associated tasks are updated. Pre-release is performed in a development environment to facilitate corresponding testing of the updated tasks, and formal release is performed after the test meets the release conditions, thereby improving the execution accuracy of the target task.

[0116] In step S103, historical metadata and update information of the update task are acquired; the historical metadata represents historical operation indicator information of the target historical task.

[0117] In a possible implementation, obtaining historical metadata includes:

[0118] Obtaining an average running time of the target historical task within a preset time, and determining the average running time as the running time of the target historical task;

[0119] Obtain an average start running time of the target historical task within the preset time, and determine the average start running time as the start running time of the target historical task;

[0120] Obtaining an average task output time of the target historical task within the preset time, and determining the average task output time as the target historical task output time;

[0121] Obtaining the usage records of the target historical tasks and generating a downstream task list;

[0122] Obtaining the usage records of the leaf tasks of the target historical task, and generating a downstream leaf task list;

[0123] The historical metadata is obtained based on the task running time, the task start running time, the task output time, the downstream task list and the downstream leaf task list.

[0124] In a possible implementation, the historical metadata of the target historical task is automatically collected by a program.

[0125] In a possible implementation, the preset time is approximately 31 days.

[0126] In a possible implementation, the execution time of the target historical task in the past 31 days is obtained, the execution time of 31 days is averaged to obtain the average running time (avg_run_time_31d) of the past 31 days, and the average running time is determined as the running time of the target historical task.

[0127] In a possible implementation, the start time of the target historical task in the past 31 days is obtained, the start time of the past 31 days is averaged to obtain the average start time (avg_start_time_31d), and the average start time is determined as the start time of the target historical task.

[0128] In a possible implementation, the output time of the target historical task in the past 31 days is obtained, the output time in the past 31 days is averaged to obtain the average task output time in the past 31 days (avg_finish_time_31d), and the average task output time is determined to be the target historical task output time.

[0129] In a possible implementation, the usage record of the target historical task is obtained, including all direct downstream lists of the target historical task and cross-system downstreams, such as the record of the target historical task being used in other systems, to generate the downstream task list.

[0130] In a possible implementation, a downstream list of leaf nodes of the target historical task is obtained to generate the downstream leaf task list.

[0131] In a possible implementation, the historical metadata includes the task running time, the task start time, the task output time, the downstream task list and the downstream leaf task list.

[0132] Under this implementation, metadata of tasks is collected, including the running time of tasks at each level, the start time, the output time, the downstream task list, the downstream leaf task list, etc. When developing and running the pre-release, it is automatically detected whether the release affects business usage based on the collected metadata, thereby realizing the automation of the data warehouse release process.

[0133] In step S105, an indicator analysis is performed based on the historical metadata and the update information to obtain an indicator analysis result of the update task; the indicator analysis result represents the execution status of the update task.

[0134] In a possible implementation, the update information includes at least one of first update information and second update information, the first update information indicates that the update task has a newly added field, and the second update information indicates that the update task has a newly added dependent task;

[0135] The performing of the indicator analysis based on the historical metadata and the update information to obtain the indicator analysis result of the update task includes:

[0136] In the case where the update information is the first update information, query analysis is performed based on the newly added field and the downstream task list to obtain a query analysis result, and the query analysis result is determined to be an indicator analysis result of the update task;

[0137] In the case where the update information is the second update information, the output time is calculated based on the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the downstream leaf task list to obtain the output time boundary value of the update task, and the output time boundary value is determined as the indicator analysis result of the update task.

[0138] In a possible implementation, when the update information is the first update information, the target task has a new field compared to the target historical task. That is, in the development environment, a new field is added, and the table in the downstream list is replaced according to the table with the added field, and it is necessary to ensure that the modification can be executed.

[0139] In a possible implementation, if the target task has a newly added field, during automatic detection, the downstream list of the target task is found according to the historical metadata, and the table structure of the target task is replaced with the table with the newly added field to see if the query is successful. If not, an error task is returned.

[0140] In one possible implementation, when the update information is the second update information, the target task has a new dependent task relative to the target historical task. In the case of the existence of the new dependent task, it is necessary to calculate the task output time of the downstream leaf task list to see whether it will be later than the latest output time agreed with the demander.

[0141] In one possible implementation, the task output time of the downstream leaf task list is calculated based on the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the downstream leaf task list.

[0142] In a possible implementation, the output time boundary value of the update task is the task output time of the updated leaf node (leaf node.cur_finish_time), and the calculation formula is as follows:

[0143] Leaf node.cur_finish_time=

[0144] max(newly added dependent task.avg_finish_time_31d, target historical task.avg_start_time_31d), + target task.cur_run_time - target historical task.avg_run_time_31d + leaf node

[0145] point.avg_finish_time_31d

[0146] in,

[0147] Added dependent task.avg_finish_time_31d, which is the average output time of the newly added dependent task in the past 31 days;

[0148] Target historical task.avg_start_time_31d is the average start time of the target historical task in the past 31 days, that is, the start time of the target historical task;

[0149] Target task.cur_run_time is the most recent trial run time of the target task;

[0150] Target historical task.avg_run_time_31d, which is the average task running time of the target historical task in the past 31 days, that is, the running time of the target historical task;

[0151] Leaf node.avg_finish_time_31d, the leaf node of the downstream leaf task list, the average output time in the past 31 days.

[0152] Under this implementation, the corresponding test indicators are determined according to the different update information of the task. Instead of checking each released task one by one downstream, this can avoid errors in the next day's tasks due to incomplete testing. On the other hand, it can also avoid the leaf tasks from being unable to be produced on time and affecting business use. A systematic automatic processing solution can be formed to ensure efficient and accurate release of data warehouse tasks.

[0153] In a possible implementation, the downstream task list contains historical task table data of the target historical task.

[0154] The query analysis is performed based on the newly added field and the downstream task list to obtain the query analysis result, including:

[0155] Acquire update table data of the update task; the update table data is generated by performing field addition processing on the historical task table data based on the newly added field;

[0156] Replace the historical task table data in the downstream task list based on the update table data to obtain a target downstream task list;

[0157] A query analysis is performed on the target downstream task list based on the explanation command to obtain the query analysis result.

[0158] In a possible implementation, the query analysis result includes a first query analysis result and a second query analysis result, wherein the first query analysis result indicates that a query on a newly added field of the update task is successful, and the second query analysis result indicates that a query on a newly added field of the update task is failed.

[0159] The method further comprises:

[0160] If the indicator analysis result is the first query analysis result, determining that the indicator analysis result meets the preset indicator condition;

[0161] If the indicator analysis result is the second query analysis result, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0162] In one possible implementation, the explain command, or EXPLAIN command, is a tool in MySQL for viewing the execution plan of SQL statements, which can help developers analyze and optimize the performance of SQL queries. The output of the EXPLAIN command can help identify performance bottlenecks in queries, such as whether valid indexes are used, whether there is a full table scan, etc. Specifically, when using the EXPLAIN command, just add the keyword EXPLAIN before the SELECT statement, and MySQL will return an execution plan showing how the query is processed by the optimizer. This execution plan includes a lot of useful information, such as the indexes used, the type of data reading operations, references between tables, the number of rows queried by the optimizer, etc.

[0163] In a possible implementation, if the target task A has newly added fields C1 and C2, during automatic detection, the direct downstream lists N1 and N2 of the target task A are found based on historical metadata, and the table structure of task A in N1 and N2 is replaced with the table with newly added C1 and C2 to see if EXPLAIN can succeed.

[0164] In a possible implementation, if EXPLAIN succeeds, it is determined that the query analysis result is the first query analysis result, and it is determined that the index analysis result meets the preset index condition.

[0165] In a possible implementation, if EXPLAIN is unsuccessful, it is determined that the query analysis result is the second query analysis result, and it is determined that the indicator analysis result does not meet the preset indicator condition, and it is necessary to return information of the error reporting task.

[0166] Under this implementation, a query analysis is performed based on the newly added fields and the downstream task list to obtain a query analysis result, and based on the query analysis result, it is determined whether the indicator analysis result meets the preset indicator condition, thereby avoiding the failure of downstream tasks caused by the newly added fields. It is convenient to locate and obtain the list of downstream failed tasks caused by the newly added fields, so as to locate and check the task nodes that cause the failure of downstream tasks, thereby improving the efficiency of pre-release testing.

[0167] In a possible implementation, the downstream leaf task list includes at least one leaf task.

[0168] The output time is calculated based on the newly added dependent task, the target historical task running time, the target historical task start running time, the target historical task output time and the downstream leaf task list to obtain the output time boundary value of the update task, including:

[0169] Obtain the running time of the update task, the running time of the newly added dependent task, and the historical leaf task output time of each leaf task;

[0170] The output time is calculated based on the running time of the update task, the running time of the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the historical leaf task output time of each leaf task to obtain the test leaf task output time of each leaf task;

[0171] In the case where the test leaf task output time is greater than the preset leaf task output time, the test leaf task output time greater than the preset leaf task output time is determined as the output time boundary value of the update task.

[0172] In a possible implementation, the method further includes:

[0173] Get the preset demand output time;

[0174] Compare the preset demand output time with the output time boundary value of the update task to obtain a comparison result;

[0175] If the comparison result indicates that the output time boundary value of the update task is less than or equal to the preset required output time, it is determined that the indicator analysis result meets the preset indicator condition;

[0176] If the comparison result indicates that the output time boundary value of the update task is greater than the preset required output time, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0177] In a possible implementation, if the target task B has newly added dependent tasks P1 and P2, according to the historical metadata, if the downstream list of leaf nodes of task B is L1 and L2, calculate the latest output time of these two leaf nodes (L1 / L2.cur_finish_time):

[0178] L1 / L2.cur_finish_time=

[0179] max(P1.avg_finish_time_31d, P2.avg_finish_time_31d, B.avg_start_time_31d)+B.cur_run_time-B.avg_run_time_31d+L1 / L2.avg_finish_time_31d,

[0180] in,

[0181] P1.avg_finish_time_31d, the average output time of P1 in the past 31 days;

[0182] P2.avg_finish_time_31d, the average output time of P2 in the past 31 days;

[0183] B.avg_start_time_31d, the average start time of the target historical task in the past 31 days;

[0184] B.cur_run_time is the trial run time of target task B;

[0185] B.avg_run_time_31d, which is the average task running time of the target historical task in the past 31 days, that is, the target historical task running time;

[0186] L1 / L2.avg_finish_time_31d is the average output time of L1 / L2 in the past 31 days.

[0187] In a possible implementation, L1 / L2.cur_finish_time is the latest production time of L1 / L2, that is, the one with a longer production time among leaves L1 and L2.

[0188] In a possible implementation, the preset demand output time is the time agreed with the demander.

[0189] In a possible implementation, if the output time boundary value of the update task is less than or equal to the preset required output time, it is determined that the indicator analysis result meets the preset indicator condition;

[0190] In a possible implementation, if the output time boundary value of the update task is greater than the preset required output time, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0191] Under this implementation, the output time is calculated based on the newly added dependent task, the running time of the target historical task, the start running time of the target historical task, the output time of the target historical task and the downstream leaf task list to obtain the output time boundary value of the update task, and the preset demand output time and the output time boundary value of the update task are compared to obtain a comparison result, and based on the comparison result, it is determined whether the indicator analysis result meets the preset indicator condition, thereby avoiding the output delay of the downstream leaf task caused by the newly added dependency, and facilitating the positioning of the task node with delayed output caused by the newly added dependency, so as to optimize the task node with delayed output and complete the business requirements agreed with the demander.

[0192] In step S107, when the indicator analysis result does not meet the preset indicator condition, indicator abnormality information in the update task is determined based on the indicator analysis result.

[0193] In a possible implementation manner, if the update information is the first update information, when the indicator analysis result does not meet the preset indicator condition, the error task information output by EXPLAIN is returned.

[0194] In a possible implementation, determining the indicator abnormality information in the update task based on the indicator analysis result includes:

[0195] When the indicator analysis result indicates that the output time boundary value of the update task is greater than the preset required output time, an abnormality analysis is performed based on the running time of the update task and the running time of the newly added dependent task to obtain an abnormality analysis result;

[0196] In a case where the abnormal analysis result indicates that the running time of the update task is greater than a first preset running time, determining the running time of the update task as the indicator abnormality information in the update task;

[0197] When the abnormal analysis result indicates that the running time of the newly added dependent task is greater than the second preset running time, the running time of the newly added dependent task is determined as the indicator abnormality information in the update task.

[0198] In a possible implementation, if the update information is the second update information, when the indicator analysis result does not meet the preset indicator condition, that is, when the output time boundary value of the update task is greater than the preset required output time, the first preset running time is obtained, and if the running time of the update task is greater than the first preset running time, it is determined that the output time of the update task is too late, and the running time of the update task is determined to be the indicator abnormality information in the update task, and the running time of the update task needs to be optimized. Specifically, the first preset running time is set based on the preset required output time.

[0199] In one possible implementation, if the update information is the second update information, when the indicator analysis result does not meet the preset indicator condition, that is, when the output time boundary value of the update task is greater than the preset required output time, the second preset running time is obtained, and if the running time of the newly added dependent task is greater than the second preset running time, it is determined that the output time of the newly added dependent task is too late, and the running time of the newly added dependent task is determined to be the indicator abnormality information in the update task, and the running time of the newly added dependent task needs to be optimized. Specifically, the second preset running time is set based on the preset required output time.

[0200] Under this implementation, the indicator abnormality information in the update task is determined based on the indicator analysis results, and the key tasks of the downstream failed task list caused by the newly added fields or the leaf node output delay caused by the return are automatically detected and returned, which is convenient for developers to troubleshoot based on the indicator abnormality information and improves the efficiency of determining key tasks.

[0201] In step S109, a test report corresponding to the associated task is generated based on the indicator abnormality information.

[0202] In a possible implementation, a test report corresponding to the associated task is generated based on the indicator abnormality information, so as to prompt the developer to locate the task node to be optimized according to the test report.

[0203] In an exemplary embodiment, Figure 2 As shown, the historical tasks need to be updated and released to obtain Task C: add fields C1 and C2, and add dependencies P1 and P2;

[0204] After the user pre-publishes, it is automatically detected based on the collected metadata information;

[0205] During automatic detection, find the direct downstream lists N1 and N2 of task C based on historically stored metadata, replace the table structure before task C is released in N1 and N2 with the table with C1 and C2 added to see if EXPLAIN succeeds. If not, return to the error task.

[0206] According to the historically stored metadata information, find the leaf node downstream lists L1 and L2 of task C. Calculate the latest output time of these two leaf nodes

[0207] L1 / L2.cur_finish_time=max(P1.avg_finish_time_31d,P2.avg_finish_time_31d,C.avg_st art_time_31d)+C.cur_run_time-C.avg_run_time_31d+L1 / L2.avg_finish_time_31d,

[0208] If the time is longer than the time agreed with the demander, check whether it is caused by the late output of P1 / P2 or the long running time after A iteration. Return the key tasks that caused the delay and optimize them.

[0209] In this implementation, when the pre-release is running in the development environment, it is automatically detected based on the collected metadata whether the release causes the downstream task to exceed the latest task output time agreed with the demander, and whether it will cause the downstream task to report an error. The detection results are fed back to the development to automate the data warehouse release process.

[0210] Figure 3 The structure diagram of a data warehouse iteration task testing device 300 provided in an embodiment of the present application is shown. The device has the function of implementing the data warehouse iteration task testing method in the above method embodiment. The function can be implemented by hardware or by hardware executing corresponding software. Figure 3 As shown, the device may include:

[0211] The pre-release module 301 is used to update the associated tasks of the target task in response to the pre-release instruction information for the target task, and obtain the updated task after the associated tasks are updated; the target task is the task after the target historical task is modified, and the target historical task is used to manage the target data warehouse;

[0212] The test information acquisition module 302 is used to acquire historical metadata and update information of the update task; the historical metadata represents the historical operation indicator information of the target historical task;

[0213] An indicator analysis module 303 is used to perform an indicator analysis based on the historical metadata and the update information to obtain an indicator analysis result of the update task; the indicator analysis result represents the execution status of the update task;

[0214] An indicator abnormality information determination module 304 is used to determine the indicator abnormality information in the update task based on the indicator analysis result when the indicator analysis result does not meet the preset indicator condition;

[0215] The test report generating module 305 is used to generate a test report corresponding to the associated task based on the indicator abnormality information.

[0216] In a possible implementation, the test information acquisition module 302 includes a historical metadata acquisition unit, and the historical metadata acquisition unit is used to:

[0217] Obtaining an average running time of the target historical task within a preset time, and determining the average running time as the running time of the target historical task;

[0218] Obtain an average start running time of the target historical task within the preset time, and determine the average start running time as the start running time of the target historical task;

[0219] Obtaining an average task output time of the target historical task within the preset time, and determining the average task output time as the target historical task output time;

[0220] Obtaining the usage records of the target historical tasks and generating a downstream task list;

[0221] Obtaining the usage records of the leaf tasks of the target historical task, and generating a downstream leaf task list;

[0222] The historical metadata is obtained based on the task running time, the task start running time, the task output time, the downstream task list and the downstream leaf task list.

[0223] In a possible implementation, the update information includes at least one of first update information and second update information, the first update information indicates that the update task has a newly added field, and the second update information indicates that the update task has a newly added dependent task;

[0224] The indicator analysis module 303 is used to:

[0225] In the case where the update information is the first update information, query analysis is performed based on the newly added field and the downstream task list to obtain a query analysis result, and the query analysis result is determined to be an indicator analysis result of the update task;

[0226] In the case where the update information is the second update information, the output time is calculated based on the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the downstream leaf task list to obtain the output time boundary value of the update task, and the output time boundary value is determined as the indicator analysis result of the update task.

[0227] In a possible implementation, the downstream task list contains historical task table data of the target historical task, and the indicator analysis module 303 includes a first indicator analysis unit.

[0228] The query analysis unit is used to:

[0229] Acquire update table data of the update task; the update table data is generated by performing field addition processing on the historical task table data based on the newly added field;

[0230] Replace the historical task table data in the downstream task list based on the update table data to obtain a target downstream task list;

[0231] A query analysis is performed on the target downstream task list based on the explanation command to obtain the query analysis result.

[0232] In a possible implementation, the query analysis result includes a first query analysis result and a second query analysis result, wherein the first query analysis result indicates that a query on a newly added field of the update task is successful, and the second query analysis result indicates that a query on a newly added field of the update task is failed.

[0233] The first indicator analysis unit is further used for:

[0234] If the indicator analysis result is the first query analysis result, determining that the indicator analysis result meets the preset indicator condition;

[0235] If the indicator analysis result is the second query analysis result, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0236] In a possible implementation, the downstream leaf task list includes at least one leaf task, and the indicator analysis module further includes a second indicator analysis unit.

[0237] The second indicator analysis unit is used to:

[0238] Obtain the running time of the update task, the running time of the newly added dependent task, and the historical leaf task output time of each leaf task;

[0239] The output time is calculated based on the running time of the update task, the running time of the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the historical leaf task output time of each leaf task to obtain the test leaf task output time of each leaf task;

[0240] In the case where the test leaf task output time is greater than the preset leaf task output time, the test leaf task output time greater than the preset leaf task output time is determined as the output time boundary value of the update task.

[0241] In a possible implementation manner, the second indicator analysis unit is further used to:

[0242] Get the preset demand output time;

[0243] Compare the preset demand output time with the output time boundary value of the update task to obtain a comparison result;

[0244] If the comparison result indicates that the output time boundary value of the update task is less than or equal to the preset required output time, it is determined that the indicator analysis result meets the preset indicator condition;

[0245] If the comparison result indicates that the output time boundary value of the update task is greater than the preset required output time, it is determined that the indicator analysis result does not meet the preset indicator condition.

[0246] In a possible implementation, the indicator abnormal information determination module 304 is used to:

[0247] When the indicator analysis result indicates that the output time boundary value of the update task is greater than the preset required output time, an abnormality analysis is performed based on the running time of the update task and the running time of the newly added dependent task to obtain an abnormality analysis result;

[0248] In a case where the abnormal analysis result indicates that the running time of the update task is greater than a first preset running time, determining the running time of the update task as the indicator abnormality information in the update task;

[0249] When the abnormal analysis result indicates that the running time of the newly added dependent task is greater than the second preset running time, the running time of the newly added dependent task is determined as the indicator abnormality information in the update task.

[0250] In a possible implementation, the pre-release module 301 is used to:

[0251] Obtaining pre-release instruction information for a target task input by the subject;

[0252] In a development environment, the associated tasks of the target task are updated to obtain updated tasks after the associated tasks are updated.

[0253] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0254] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a testing method for any data warehouse iteration task provided in the above method embodiments.

[0255] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0256] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to a testing method for implementing a data warehouse iteration task. The at least one instruction or the at least one program is loaded and executed by the processor to implement any one of the testing methods for data warehouse iteration tasks provided in the above method embodiments.

[0257] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0258] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0259] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0260] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0261] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A testing method for data warehouse iterative tasks, characterized in that: The method comprises: In response to the pre-release instruction information for the target task, the associated task of the target task is updated to obtain the updated task after the associated task is updated; the target task is the task after the target historical task is modified, and the target historical task is used to manage the target data warehouse; Acquire historical metadata and update information of the update task; the historical metadata represents historical operation indicator information of the target historical task; Performing an index analysis based on the historical metadata and the update information to obtain an index analysis result of the update task; the index analysis result represents the execution status of the update task; In the case where the indicator analysis result does not meet the preset indicator condition, determining indicator abnormality information in the update task based on the indicator analysis result; A test report corresponding to the associated task is generated based on the indicator abnormality information.

2. The data warehouse iterative task testing method according to claim 1, characterized in that: The obtaining of historical metadata includes: Obtaining an average running time of the target historical task within a preset time, and determining the average running time as the running time of the target historical task; Obtain an average start running time of the target historical task within the preset time, and determine the average start running time as the start running time of the target historical task; Obtaining an average task output time of the target historical task within the preset time, and determining the average task output time as the target historical task output time; Obtaining the usage records of the target historical tasks and generating a downstream task list; Obtaining the usage records of the leaf tasks of the target historical task, and generating a downstream leaf task list; The historical metadata is obtained based on the task running time, the task start running time, the task output time, the downstream task list and the downstream leaf task list.

3. The data warehouse iterative task testing method according to claim 2, characterized in that: The update information includes at least one of first update information and second update information, the first update information indicating that the update task has a newly added field, and the second update information indicating that the update task has a newly added dependent task; The performing of the indicator analysis based on the historical metadata and the update information to obtain the indicator analysis result of the update task includes: In the case where the update information is the first update information, query analysis is performed based on the newly added field and the downstream task list to obtain a query analysis result, and the query analysis result is determined to be an indicator analysis result of the update task; In the case where the update information is the second update information, the output time is calculated based on the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the downstream leaf task list to obtain the output time boundary value of the update task, and the output time boundary value is determined as the indicator analysis result of the update task.

4. The data warehouse iterative task testing method according to claim 3, characterized in that: The downstream task list contains historical task table data of the target historical task. The query analysis is performed based on the newly added field and the downstream task list to obtain the query analysis result, including: Acquire update table data of the update task; the update table data is generated by performing field addition processing on the historical task table data based on the newly added field; Replace the historical task table data in the downstream task list based on the update table data to obtain a target downstream task list; A query analysis is performed on the target downstream task list based on the explanation command to obtain the query analysis result.

5. The data warehouse iterative task testing method according to claim 4, characterized in that: The query analysis result includes a first query analysis result and a second query analysis result, wherein the first query analysis result indicates that the query of the newly added field of the update task is successful, and the second query analysis result indicates that the query of the newly added field of the update task is failed. The method further comprises: If the indicator analysis result is the first query analysis result, determining that the indicator analysis result meets the preset indicator condition; If the indicator analysis result is the second query analysis result, it is determined that the indicator analysis result does not meet the preset indicator condition.

6. The data warehouse iterative task testing method according to claim 3, characterized in that: The downstream leaf task list includes at least one leaf task, The output time is calculated based on the newly added dependent task, the target historical task running time, the target historical task start running time, the target historical task output time and the downstream leaf task list to obtain the output time boundary value of the update task, including: Obtain the running time of the update task, the running time of the newly added dependent task, and the historical leaf task output time of each leaf task; The output time is calculated based on the running time of the update task, the running time of the newly added dependent task, the running time of the target historical task, the start time of the target historical task, the output time of the target historical task and the historical leaf task output time of each leaf task to obtain the test leaf task output time of each leaf task; In the case where the test leaf task output time is greater than the preset leaf task output time, the test leaf task output time greater than the preset leaf task output time is determined as the output time boundary value of the update task.

7. The data warehouse iterative task testing method according to claim 6, characterized in that: The method further comprises: Get the preset demand output time; Compare the preset demand output time with the output time boundary value of the update task to obtain a comparison result; If the comparison result indicates that the output time boundary value of the update task is less than or equal to the preset required output time, it is determined that the indicator analysis result meets the preset indicator condition; If the comparison result indicates that the output time boundary value of the update task is greater than the preset required output time, it is determined that the indicator analysis result does not meet the preset indicator condition.

8. The data warehouse iterative task testing method according to claim 7, characterized in that: The determining the indicator abnormality information in the update task based on the indicator analysis result includes: When the indicator analysis result indicates that the output time boundary value of the update task is greater than the preset required output time, an abnormality analysis is performed based on the running time of the update task and the running time of the newly added dependent task to obtain an abnormality analysis result; When the abnormal analysis result indicates that the running time of the update task is greater than the first preset running time, determining the running time of the update task as the indicator abnormality information in the update task; When the abnormal analysis result indicates that the running time of the newly added dependent task is greater than the second preset running time, the running time of the newly added dependent task is determined as the indicator abnormality information in the update task.

9. The data warehouse iterative task testing method according to claim 1, characterized in that: The step of updating the associated task of the target task in response to the pre-release instruction information for the target task to obtain the updated task after the associated task is updated includes: Obtaining pre-release instruction information for a target task input by the subject; In a development environment, the associated tasks of the target task are updated to obtain updated tasks after the associated tasks are updated.

10. A testing device for data warehouse iterative tasks, characterized in that: The device comprises: A pre-release module, for updating the associated tasks of the target task in response to the pre-release instruction information for the target task, and obtaining an updated task after the associated tasks are updated; the target task is a task after the target historical task is modified, and the target historical task is used to manage the target data warehouse; A test information acquisition module, used to acquire historical metadata and update information of the update task; the historical metadata represents historical operating indicator information of the target historical task; An indicator analysis module, configured to perform an indicator analysis based on the historical metadata and the update information to obtain an indicator analysis result of the update task; the indicator analysis result represents an execution status of the update task; An indicator abnormality information determination module, used to determine the indicator abnormality information in the update task based on the indicator analysis result when the indicator analysis result does not meet the preset indicator condition; A test report generation module is used to generate a test report corresponding to the associated task based on the indicator abnormality information.

Images