A data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a data processing method and device, electronic equipment and storage medium, and belongs to the technical field of computers. The method comprises: obtaining recovery information of a complement event; obtaining data information of at least two task instances; determining at least one problem task instance from the at least two task instances according to a dependency relationship between the at least two task instances and the data information of the at least two task instances; and performing a data generation operation on the at least one problem task instance. In the present disclosure, the electronic equipment can only perform a data generation operation on part of the downstream task instances of a certain task instance, i.e. the problem task instances, specifically the task instances affected by the problem data, which can reduce resource waste and improve the efficiency of data processing.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, when the data generated by a certain task instance has quality problems, the data complementation device can perform a blocking operation on all downstream task instances of that task instance; and when the generated data returns to normal, it can perform a recovery operation on all downstream task instances.

[0003] However, the data generated above may only affect a portion of the downstream task instances within that set of downstream task instances. Therefore, the method of performing blocking or recovery operations on all downstream task instances could consume significant resources, resulting in resource waste. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, electronic device, and storage medium, which solves the technical problem that when the complement device performs blocking or recovery operations on all downstream task instances of a certain task instance, it consumes a lot of resources and causes resource waste.

[0005] The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a data processing method is provided. The method may include: acquiring recovery information of a complement event, the recovery information including identifiers of at least two task instances and dependencies between the at least two task instances; acquiring data information of the at least two task instances, wherein the data information of one task instance includes input information and output information of the task instance; determining at least one problematic task instance from the at least two task instances based on the dependencies between the at least two task instances and the data information of the at least two task instances, the problematic task instance being used to characterize task instances whose generated data has quality problems, the number of the at least one problematic task instance being less than the number of the at least two task instances; and performing a data generation operation on the at least one problematic task instance.

[0007] Optionally, obtaining the data information of the at least two task instances specifically includes: obtaining the task details corresponding to each of the at least two task instances, wherein the task details corresponding to a task instance include the query statement corresponding to the task instance; performing hash processing on the query statements corresponding to each of the at least two task instances to obtain the hash value corresponding to each of the at least two task instances; if a first hash value exists in the stored target correspondence table, obtaining the data information of the first task instance from the target correspondence table, wherein the target correspondence table includes the data information of multiple task instances and the hash value corresponding to each of the multiple task instances, the first hash value is the hash value corresponding to the first task instance, and the first task instance is one of the at least two task instances.

[0008] Optionally, the above data processing method further includes: if the first hash value does not exist in the target correspondence, parsing the query statement corresponding to the first task instance to obtain the data information of the first task instance; and adding the first hash value and the data information of the first task instance to the target correspondence.

[0009] Optionally, the input information of a task instance includes an input field, and the output information of the task instance includes an output field. The above-mentioned determination of at least one problematic task instance from the at least two task instances based on the dependency relationship between the at least two task instances and the data information of the at least two task instances specifically includes: when the current task instance is a problematic task instance, determining whether the input field corresponding to the current task instance's direct downstream task instance contains the output field corresponding to the current task instance, wherein the current task instance is one of the at least two task instances, and the direct downstream task instance is a downstream task instance that has a dependency relationship with the current task instance; when the input field corresponding to the direct downstream task instance contains the output field corresponding to the current task instance, determining that the direct downstream task instance is a problematic task instance.

[0010] Optionally, the above data processing method further includes: when the input field corresponding to the direct downstream task instance does not contain the output field corresponding to the previous task instance, determining to perform a pruning operation on the direct downstream task instance.

[0011] Optionally, the above data processing method further includes: adding the identifier of each problem task instance in the at least one problem task instance and the output field corresponding to each problem task instance to a target queue, the target queue being used to characterize the determined order of each problem task instance.

[0012] Optionally, the above data processing method further includes: generating a directed acyclic graph corresponding to the complement event based on the recovery information, the directed acyclic graph including task instance nodes and edges, the task instance nodes being used to represent task instances, and the edges being used to connect task instance nodes with dependencies; performing a pruning operation on the target task instance nodes to obtain a target directed acyclic graph, the target task instance nodes being task instance nodes other than at least one problem task instance node among at least two task instance nodes, the at least two task instance nodes being task instance nodes included in the directed acyclic graph, and the at least one problem task instance node being the task instance node represented by the at least one problem task instance; the above-mentioned data generation operation on the at least one problem task instance specifically includes: performing a data generation operation on each task instance node included in the target directed acyclic graph, each task instance node being the task instance node represented by each problem task instance among the at least one problem task instance.

[0013] According to a second aspect of the present disclosure, a data processing apparatus is provided. The apparatus may include: an acquisition module, a determination module, and a processing module; the acquisition module is configured to acquire recovery information of a complement event, the recovery information including identifiers of at least two task instances and dependencies between the at least two task instances; the acquisition module is further configured to acquire data information of the at least two task instances, wherein the data information of one task instance includes input information and output information of the task instance; the determination module is configured to determine at least one problematic task instance from the at least two task instances based on the dependencies between the at least two task instances and the data information of the at least two task instances, the problematic task instance being used to characterize task instances whose generated data has quality problems, the number of the at least one problematic task instance being less than the number of the at least two task instances; the processing module is configured to perform a data generation operation on the at least one problematic task instance.

[0014] Optionally, the acquisition module is specifically configured to acquire the task details corresponding to each of the at least two task instances, wherein the task details corresponding to a task instance include the query statement corresponding to the task instance; the processing module is further configured to perform hash processing on the query statements corresponding to each of the at least two task instances to obtain the hash value corresponding to each of the at least two task instances; the acquisition module is further configured to, if a first hash value exists in the stored target correspondence, acquire the data information of the first task instance from the target correspondence table, wherein the target correspondence includes the data information of multiple task instances and the hash value corresponding to each of the multiple task instances, the first hash value is the hash value corresponding to the first task instance, and the first task instance is one of the at least two task instances.

[0015] Optionally, the processing module is further configured to, if the first hash value does not exist in the target correspondence, parse the query statement corresponding to the first task instance to obtain the data information of the first task instance; the processing module is further configured to add the first hash value and the data information of the first task instance to the target correspondence.

[0016] Optionally, the input information of a task instance includes an input field, and the output information of the task instance includes an output field. Specifically, the determining module is configured to, when the current task instance is a problem task instance, determine whether the input field corresponding to the current task instance exists in the input field of the directly downstream task instance, where the current task instance is one of the at least two task instances, and the directly downstream task instance is a downstream task instance that has a dependency relationship with the current task instance; the determining module is further configured to determine the directly downstream task instance as a problem task instance when the input field corresponding to the directly downstream task instance contains the output field corresponding to the current task instance.

[0017] Optionally, the determining module is further configured to determine to perform a pruning operation on the direct downstream task instance when the input field corresponding to the direct downstream task instance does not contain the output field corresponding to the previous task instance.

[0018] Optionally, the processing module is further configured to add the identifier of each problem task instance in the at least one problem task instance and the corresponding output field of each problem task instance to a target queue, the target queue being used to characterize the determined order of each problem task instance.

[0019] Optionally, the processing module is further configured to generate a directed acyclic graph (DAG) corresponding to the complement event based on the recovery information. The DAG includes task instance nodes and edges, where the task instance nodes represent task instances and the edges connect task instance nodes with dependencies. The processing module is also configured to perform pruning operations on the target task instance nodes to obtain a target DAG. The target task instance nodes are task instance nodes other than at least one problem task instance node among at least two task instance nodes. The at least two task instance nodes are the task instance nodes included in the DAG, and the at least one problem task instance node is the task instance node represented by the at least one problem task instance. Specifically, the processing module is configured to perform a data generation operation on each task instance node included in the target DAG, where each task instance node is the task instance node represented by each problem task instance among the at least one problem task instance.

[0020] According to a third aspect of the present disclosure, an electronic device is provided, which may include: a processor and a memory configured to store processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional data processing methods of the first aspect described above.

[0021] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which stores instructions that, when executed by an electronic device, enable the electronic device to perform any of the optional data processing methods described in the first aspect.

[0022] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform any of the optional data processing methods of the first aspect.

[0023] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0024] Based on any of the above aspects, in this disclosure, the electronic device can acquire recovery information of the complement event and data information of at least two task instances. Since the recovery information includes the dependency relationship between at least two task instances, the electronic device can determine all downstream task instances of each of the at least two task instances based on the dependency relationship. Furthermore, since the data information of a task instance includes the input information and output information of the task instance, the electronic device can determine whether there is a data dependency relationship between each task instance and all downstream task instances based on the input and output information of each task instance. Thus, the electronic device can determine at least one problematic task instance from the at least two task instances based on all downstream task instances of each task instance and whether there is a data dependency relationship between each task instance and all downstream task instances, and perform a data generation operation on the at least one problematic task instance to generate data. In the embodiments of this disclosure, the electronic device can perform the data generation operation only on a portion of the downstream task instances (i.e., problematic task instances), specifically the task instances affected by problematic data, which can reduce resource waste and improve data processing efficiency.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0027] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this disclosure is shown.

[0028] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this disclosure is shown;

[0029] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this disclosure is shown;

[0030] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this disclosure is shown;

[0031] Figure 5 A flowchart illustrating another data processing method provided in an embodiment of this disclosure is shown;

[0032] Figure 6 A flowchart illustrating another data processing method provided in an embodiment of this disclosure is shown;

[0033] Figure 7 A flowchart illustrating another data processing method provided in an embodiment of this disclosure is shown;

[0034] Figure 8 A schematic diagram illustrating an embodiment of the present disclosure of performing a pruning operation is shown;

[0035] Figure 9 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure is shown;

[0036] Figure 10 A schematic diagram of the structure of another data processing apparatus provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0037] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0038] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0039] It should also be understood that the term "comprising" indicates the presence of the described feature, whole, step, operation, element and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements and / or components.

[0040] The data disclosed herein may be data authorized by the user or fully authorized by all parties.

[0041] As described in the background section, since the data complement device performs blocking or recovery operations on all downstream task instances of a certain task instance, it consumes a lot of resources and causes resource waste. Based on this, embodiments of this disclosure provide a data processing method in which an electronic device can perform data generation operations only on a portion of the downstream task instances (i.e., problematic task instances), specifically the task instances affected by problematic data, thereby reducing resource waste and improving data processing efficiency.

[0042] The data processing method, apparatus, electronic device, and storage medium provided in this disclosure are applied in scenarios involving data generation (or data recovery). When the electronic device obtains recovery information of a complement event, it can determine at least one problematic task instance from at least two task instances according to the method provided in this disclosure, and perform a data generation operation on the at least one problematic task instance.

[0043] The data processing method provided in the embodiments of this disclosure is illustrated below with reference to the accompanying drawings:

[0044] It is understood that the electronic device executing the data processing method provided in the embodiments of this disclosure can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc., which can install and use content community applications. This disclosure does not impose any special limitations on the specific form of the electronic device. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction, or handwriting device.

[0045] like Figure 1 As shown, the data processing method provided in this embodiment may include S101-S104.

[0046] S101, The electronic device obtains the recovery information of the complement event.

[0047] The recovery information includes the identifiers of at least two task instances and the dependencies between the at least two task instances.

[0048] It should be understood that the at least two task instances are the task instances corresponding to the complement event. This complement event can be applied to scenarios involving the generation of historical data (or the recovery of historical data). Specifically, executing the recovery event performs data generation operations on the at least two task instances, meaning that all task instances within the at least two task instances can generate data.

[0049] In this embodiment of the disclosure, the scenario of a task instance generating historical data can be divided into the following cases:

[0050] In one scenario, the current task instance might be a task instance generated in the current year (e.g., 2021), from which data for the current year can be generated. However, the electronic device might need to obtain data generated in a historical year (e.g., 2019). In this case, the electronic device needs to create and trigger a complement event. The task instance corresponding to this complement event can be linked to a task instance generated in the historical year. The electronic device can then obtain the data generated in that historical year based on the task instance generated in that historical year, thus generating historical data.

[0051] In one scenario, historical data may have already been generated, but it might have been deleted during subsequent data processing. In this case, an electronic device can generate the relevant data—that is, generate historical data—by processing (or operating on) the task instance corresponding to a complement event.

[0052] In another scenario, the electronic device may determine that the data generated by the current task instance has quality issues (hereinafter referred to as problematic data), but this task instance is not the source task instance that generated the problematic data. In this case, the electronic device can create a complement event to generate historical data based on the task instance corresponding to that complement event, thus tracing back to the source task instance that generated the problematic data.

[0053] It is understandable that the relationship between a task and a task instance is similar to the relationship between a program and a process. For a task, each execution of the task will generate a task instance; different task instances will generate different data partitions, which can be understood as partitions that store the data generated (or obtained after operation) by the task instance.

[0054] In this embodiment of the disclosure, the dependency relationship between the at least two task instances is used to characterize the upstream and downstream relationship between the at least two task instances. For example, assuming that the at least two task instances include a first task instance and a second task instance, if there is a dependency relationship between the first task instance and the second task instance, it means that the first task instance is a downstream task instance of the second task instance, or the first task instance is an upstream task instance of the second task instance.

[0055] It should be noted that the task instances corresponding to the aforementioned complement events (i.e., the at least two task instances mentioned above) can be understood as complement instances. That is, if the complement event is executed, the task instance corresponding to the complement event (i.e., the complement instance) will be supplemented, and the task instance corresponding to the complement event will be executed. Specifically, the task instance corresponding to the complement event will generate data, that is, supplement the data of the task instance corresponding to the complement event.

[0056] Optionally, a task instance can be represented by its primary key, which can be a combination of the name of the task corresponding to the task instance and the generation time of the task instance.

[0057] S102, The electronic device acquires data information from at least two task instances.

[0058] The data information of a task instance includes the input information and the output information of that task instance.

[0059] It should be understood that the input information of a task instance is the input data of that task instance, and the output information of that task instance is the output data of that task instance. The electronic device performs a data generation operation on the task instance so that the task instance can obtain the output data based on the input data.

[0060] In one implementation of the embodiments of this disclosure, the data information of a task instance can be a data table of the task instance, which may include an input table and an output table of the task instance.

[0061] S103. The electronic device determines at least one problem task instance from at least two task instances based on the dependencies between at least two task instances and the data information of at least two task instances.

[0062] The problematic task instance is used to characterize a task instance in which the generated data has quality problems, and the number of the at least one problematic task instance is less than the number of the at least two task instances.

[0063] It is understandable that the generated data has quality issues (i.e., problematic task instances), and the output data has quality issues.

[0064] It should be understood that electronic devices can determine all downstream task instances of each of the at least two task instances based on the dependencies between them. However, all downstream task instances of each task instance do not necessarily have a data dependency with each task instance. Specifically, all downstream task instances may not use the data generated by each task instance, that is, the input data of each of the downstream task instances may not be related to the output data of each task instance.

[0065] The electronic device can determine that a downstream task instance is a problematic task instance when the input data of a certain downstream task instance is related to the output data of each of those task instances, and each of those task instances is a problematic task instance. Otherwise, when each of those task instances is a problematic task instance, and the input data of the downstream task instance is not related to the output data of each of those task instances, the electronic device can determine that the downstream task instance is not a problematic task instance. Thus, the electronic device can determine the problematic task instance with quality issues among at least two task instances based on the data information of each task instance, specifically the input and output data of each task instance.

[0066] S104. The electronic device performs a data generation operation on at least one instance of a problem task.

[0067] It should be understood that when an electronic device performs a data generation operation on the at least one problematic task instance, it is to instruct each of the at least one problematic task instances to generate data.

[0068] It is understandable that electronic devices may instruct a problem task instance to generate data in the following two ways:

[0069] In one scenario, the electronic device instructs the problem task instance to generate data when no data has been generated previously, thus instructing the problem task instance to generate data for the first time.

[0070] In another scenario, the problematic task instance has previously generated data (but that data may have been deleted), and the electronic device instructs the problematic task instance to generate the data, which means instructing the problematic task instance to regenerate the data, or instructing the problematic task instance to rerun and recover.

[0071] Optionally, the electronic device may also perform a data blocking operation on the at least one problematic task instance, that is, block each of the at least one problematic task instances from pausing data generation.

[0072] The technical solution provided by the above embodiments can bring at least the following beneficial effects: As shown in S101-S104, the electronic device can obtain the recovery information of the complement event and the data information of at least two task instances; since the recovery information includes the dependency relationship between at least two task instances, the electronic device can determine all downstream task instances of each of the at least two task instances based on the dependency relationship; furthermore, since the data information of a task instance includes the input information and the output information of the task instance, the electronic device can determine whether there is a data dependency relationship between each task instance and all downstream task instances based on the input information and output information of each task instance; thus, the electronic device can determine at least one problematic task instance from the at least two task instances based on all downstream task instances of each task instance and whether there is a data dependency relationship between each task instance and all downstream task instances, and perform a data generation operation on the at least one problematic task instance so that the at least one problematic task instance generates data. In this embodiment of the present disclosure, the electronic device can perform the data generation operation only on a portion of the downstream task instances (i.e., problematic task instances) of a certain task instance, specifically the task instances affected by problematic data, which can reduce resource waste and improve the efficiency of data processing.

[0073] Combination Figure 1 ,like Figure 2 As shown, in one implementation of this disclosure, obtaining data information of at least two task instances may specifically include S1021-S1023.

[0074] S1021. The electronic device obtains the task details corresponding to at least two task instances.

[0075] The task details for a task instance include the query statement corresponding to that task instance.

[0076] It should be understood that the task details corresponding to a task instance are the task details of the task corresponding to that task instance, and the query statement corresponding to that task instance is the query statement corresponding to that task.

[0077] Optionally, the task details corresponding to a task instance may also include the identifier of the task corresponding to that task instance, and the electronic device may periodically obtain the task details corresponding to each of the at least two task instances.

[0078] In one implementation of this disclosure, the electronic device can create multiple threads and then distribute the processing of the at least two task instances based on these multiple threads. Specifically, the task details corresponding to each of the at least two task instances can be obtained based on each of the multiple threads. Alternatively, the at least two task instances can be evenly distributed among the multiple threads, allowing the task details corresponding to each of the at least two task instances to be obtained based on each of the multiple threads.

[0079] The query statement in this embodiment can be a Structured Query Language (SQL) statement. That is, the query statement corresponding to a task instance can be the SQL statement to be executed for that task instance.

[0080] S1022. The electronic device performs hash processing on the query statements corresponding to at least two task instances to obtain the hash values ​​corresponding to at least two task instances.

[0081] It should be understood that the hash value corresponding to a task instance is the hash value of the query statement corresponding to that task instance.

[0082] Understandably, the query statement corresponding to a task instance might be quite long, occupying a significant amount of storage. Hash-processing this query statement to obtain the hash value corresponding to the task instance can reduce its storage footprint, for example, to 32 bytes. Therefore, compared to the query statement itself, querying and storing hash values ​​is much more convenient and faster.

[0083] Optionally, the hash value corresponding to a task instance can be the MD5 value corresponding to that task instance.

[0084] S1023. If a first hash value exists in the stored target correspondence, the electronic device obtains the data information of the first task instance from the target correspondence table.

[0085] The target correspondence includes data information of multiple task instances and hash values ​​corresponding to each of the multiple task instances. The first hash value is the hash value corresponding to the first task instance, which is one of the above at least two task instances.

[0086] It should be understood that the electronic device can obtain the data information of each of the multiple task instances, and generate the target mapping relationship by combining the hash values ​​corresponding to each of the multiple task instances, and then store the target mapping relationship in a cache and / or database. This target mapping relationship can represent the correspondence between the data information of each task instance and the hash value corresponding to each task instance.

[0087] When the target mapping includes the hash value corresponding to a task instance, it means that the electronic device has already parsed the query statement corresponding to the task instance. The electronic device can determine the data information of the task instance from the target mapping based on the hash value corresponding to the task instance.

[0088] Optionally, this target mapping can be stored in a cache and / or database in the form of key-value pairs. Specifically, the key in a key-value pair is the query statement corresponding to a task instance, and the value in the key-value pair is the data information of that task instance.

[0089] For example, Table 1 below is an example of a target correspondence provided in an embodiment of this disclosure. Specifically, the target correspondence includes data information corresponding to each of the four task instances (including task instance 1, task instance 2, task instance 3, and task instance 4), and the hash value corresponding to each of the four task instances.

[0090] Table 1

[0091] Data information corresponding to the task instance Hash value corresponding to the task instance Data Information 1 Hash value 1 Data Information 2 Hash value 2 Data Information 3 Hash value 3 Data Information 4 Hash value 4

[0092] Assuming the first hash value is hash value 3, the electronic device determines the data information of the first task instance as data information 3, and can obtain the data information 3 from the target correspondence.

[0093] The technical solution provided by the above embodiments can bring at least the following beneficial effects: As can be seen from S1021-S1023, the electronic device can obtain the task details corresponding to at least two task instances, and then perform hash processing on the query statement corresponding to the task instance included in the task details to obtain the hash value corresponding to each of the at least two task instances; if the hash value corresponding to one of the at least two task instances exists in the stored target correspondence, the electronic device can directly obtain the data information of the task instance from the stored target correspondence based on the hash value corresponding to the task instance, without parsing the query statement corresponding to the task instance, which can improve the efficiency of data information acquisition and thus improve the efficiency of determining the problem task instance.

[0094] Combination Figure 2 ,like Figure 3 As shown, the data processing method provided in this embodiment further includes S105-S106.

[0095] S105. If the first hash value does not exist in the target correspondence table, the electronic device parses the query statement corresponding to the first task instance to obtain the data information of the first task instance.

[0096] It should be understood that when the first hash value does not exist in the target correspondence table, it means that the electronic device has not parsed the query statement corresponding to the first task instance. In this case, the electronic device can parse the query statement corresponding to the first task instance to obtain the data information of the first task instance.

[0097] Optionally, when the query statement corresponding to the first task instance is the execution SQL statement corresponding to the first task instance, the electronic device can use abstract syntax code (AST) to parse the execution SQL statement to obtain the input and output information that the execution SQL statement depends on, that is, to obtain the data information corresponding to the execution SQL statement.

[0098] S106. The electronic device adds the data information of the first hash value and the first task instance to the target correspondence.

[0099] It is understandable that the electronic device adds the first hash value and the data information of the first task instance to the target correspondence. This can also be understood as adding the correspondence between the first hash value and the data information of the first task instance (i.e., the key-value pair corresponding to the first hash value) to the target correspondence, so that when the electronic device obtains the data information of the first task instance next time, it can directly obtain it from the target correspondence.

[0100] The technical solution provided by the above embodiments can bring at least the following beneficial effects: As shown in S105-S106, when there is no hash value (e.g., a first hash value) corresponding to one of the at least two task instances in the target correspondence, the electronic device can parse the query statement corresponding to the task instance (i.e., the first task instance) to obtain the data information of the first task instance, and can accurately and effectively obtain the data information of each of the at least two task instances. Then, the electronic device can add the first hash value and the data information of the first task instance to the target correspondence, so that when the electronic device obtains the data information of the first task instance next time, it can directly obtain it from the target correspondence, which can improve the efficiency of data information acquisition.

[0101] Combination Figure 1 ,like Figure 4 As shown, in one implementation of this disclosure, the input information of a task instance includes an input field, and the output information of the task instance includes an output field. The above-mentioned determination of at least one problem task instance from at least two task instances based on the dependency relationship between at least two task instances and the data information of at least two task instances may specifically include S1031-S1032.

[0102] S1031. When the current task instance is a problem task instance, the electronic device determines whether the input field corresponding to the current task instance exists in the input field of the direct downstream task instance of the current task instance.

[0103] The current task instance is one of the at least two task instances, and the direct downstream task instance is a downstream task instance that has a dependency relationship with the current task instance.

[0104] It should be understood that the current task instance can be the root node among the at least two task instances, i.e., the upstream task instance; the current task instance can also be a non-root node among the at least two task instances.

[0105] Based on the description of the above embodiments, it should be understood that when the data generated by a task instance has quality problems, the task instance can be identified as a problematic task instance. Specifically, when the data partition corresponding to the current task instance is an empty partition, the electronic device can identify the current task instance as a problematic task instance. In one implementation, the electronic device can also process the data generated by the current task instance; when the processed data shows abnormal fluctuations, the current task instance can also be identified as a problematic task instance.

[0106] In this embodiment of the disclosure, a field (including input field and output field) can be a column (or the name of the column) included in a certain data information (i.e., input information or output information, such as input table and output table) to represent a type of data, such as name, gender or age.

[0107] It should be noted that the number of fields included in a data information in this embodiment of the present disclosure may be one or more, and this embodiment of the present disclosure does not specifically limit the number of fields included in the data information.

[0108] It should be understood that the input fields corresponding to the direct downstream task instance are the fields included in the input information of the direct downstream task instance, used to characterize the input data of the direct downstream task instance; the output fields corresponding to the current task instance are the fields included in the output information of the current task instance, used to characterize the output data of the current task instance.

[0109] In one scenario, if the input field corresponding to the direct downstream task instance contains the output field corresponding to the current task instance, it indicates that the input data of the direct downstream task instance contains at least some data from the output data of the current task instance. Furthermore, since the current task instance is a problematic task instance, meaning the data generated by the current task instance (i.e., the output data) has quality issues, the electronic device can determine that the input data of the direct downstream task instance has quality problems. When the input data of the direct downstream task instance has quality problems, it also indicates that the output data of the direct downstream task instance has quality problems; therefore, the electronic device determines that the direct downstream task instance is a problematic task instance.

[0110] In another scenario, if the input field corresponding to the current task instance does not contain the output field corresponding to the current task instance, it indicates that the input data of the current task instance does not contain the output data of the current task instance. The electronic device can determine that the input data of the current task instance does not have quality issues, and thus determine that the output data of the current task instance does not have quality issues; that is, the electronic device determines that the current task instance is not a problematic task instance.

[0111] S1032. When the input field corresponding to the current task instance contains the output field corresponding to the current task instance, the electronic device determines that the direct downstream task instance is a problem task instance.

[0112] In one implementation of this disclosure, the electronic device can obtain a first field set and a second field set. The first field set includes each output field corresponding to the current task instance, and the second field set includes each input field corresponding to the direct downstream task instance. Then, when there is an intersection between the first field set and the second field set, the electronic device can determine that the direct downstream task instance is a problem task instance.

[0113] The technical solution provided by the above embodiments can bring at least the following beneficial effects: As shown in S1031-S1032, when the current task instance is a problematic task instance, it indicates that the data (i.e., the output data) generated by the current task instance has quality problems. The electronic device can determine whether the input field corresponding to the current task instance exists in the input field corresponding to the direct downstream task instance. When the input field corresponding to the current task instance contains the output field corresponding to the current task instance, it indicates that the input data of the direct downstream task instance contains at least some of the data in the output data of the current task instance. The electronic device can determine that the input data of the direct downstream task instance has quality problems, and thus it indicates that the output data of the direct downstream task instance also has quality problems. That is, the electronic device determines that the direct downstream task instance is a problematic task instance. This allows for the reasonable and accurate determination of each problematic task instance, improving the efficiency of determining problematic task instances.

[0114] Combination Figure 4 ,like Figure 5 As shown, the data processing method provided in this embodiment of the disclosure further includes: S107.

[0115] S107. When the input field corresponding to the current task instance does not contain the output field corresponding to the current task instance, the electronic device determines to perform a pruning operation on the direct downstream task instance.

[0116] Based on the description of the above embodiments, it should be understood that when the input field corresponding to the current task instance does not contain the output field corresponding to the current task instance, it indicates that the output data of the direct downstream task instance does not have quality issues, that is, it is determined that the direct downstream task instance is not a problematic task instance. Thus, the electronic device can determine to perform a pruning operation on the direct downstream task instance.

[0117] In one implementation of the disclosed embodiments, the above-mentioned pruning operation on the direct downstream task instance can be understood as deleting the direct downstream task instance.

[0118] In another implementation of this disclosure, performing a pruning operation on the direct downstream task instance can also be understood as deleting the identifier of the direct downstream task instance and the dependency relationship between the direct downstream task instance and the current task instance from the recovery information of the complement event, so as to obtain target recovery information for a new target complement event. The task instance corresponding to the target complement event does not contain the direct downstream task instance, and the target recovery information does not contain the identifier of the direct downstream task instance. The electronic device can execute the target complement event to complete a process similar to S104 described above.

[0119] In one optional implementation, there may be other task instances upstream of the aforementioned direct downstream task instance. In this case, the electronic device can determine whether there is a problematic task instance among these other task instances, and thus determine whether the direct downstream task instance is a problematic task instance. Specifically, the data processing method provided in this disclosure further includes steps A-B.

[0120] Step A: When the input field corresponding to the current task instance does not exist in the input field corresponding to the current task instance, the electronic device determines whether there are other problematic task instances in the direct upstream task instance of the direct downstream task instance.

[0121] The direct upstream task instance is an upstream task instance that has a dependency relationship with the direct downstream task instance, and the other problem task instance is a task instance other than the current task instance among the above at least one problem task instance.

[0122] Step B: When there are no other problematic task instances among the direct upstream task instances of the direct downstream task instance, the system determines that the current downstream task instance is not a problematic task instance.

[0123] The technical solution provided by the above embodiments can bring at least the following beneficial effects: As shown in S107, when the input field corresponding to the current task instance does not contain the output field corresponding to the current task instance, it indicates that the input data of the current task instance does not contain the output data of the current task instance. The electronic device determines that the output data of the current task instance does not have quality problems, that is, it determines that the current task instance is not a problematic task instance. Then, the electronic device determines to perform a pruning operation on the current task instance so that at least one problematic task instance remains among at least two task instances. This can reasonably and accurately determine each problematic task instance, improving the accuracy of pruning. Furthermore, since the pruned task instance does not need to perform data generation operations, the solution provided by this disclosure can also reduce resource waste.

[0124] Combination Figure 1 ,like Figure 6As shown, the data processing method provided in this embodiment may further include S108.

[0125] S108. The electronic device adds the identifier of each problem task instance and the corresponding output field of each task instance to the target queue.

[0126] The target queue is used to characterize the determined order of each problem task instance.

[0127] It should be understood that when an electronic device determines that a certain task instance (e.g., the current task instance) is a problematic task instance, the electronic device can add the identifier of the current task instance and the corresponding output field to the target queue. This allows the electronic device to retrieve the identifier of the current task instance and the corresponding output field from the target queue, and, in conjunction with the input field corresponding to the aforementioned direct downstream task instance, determine whether the direct downstream task instance is a problematic task instance. Furthermore, when the electronic device determines that the direct downstream task instance is a problematic task instance, it can add the identifier of the direct downstream task instance and the corresponding output field to the target queue, enabling the electronic device to determine whether any problematic task instances exist among the downstream task instances of the direct downstream task instance.

[0128] In one implementation of this disclosure, the electronic device can determine the problematic task instance from the at least two task instances based on a breadth-first search algorithm. Specifically, when the electronic device determines that the current task instance is a problematic task instance, it can determine whether the directly upstream task instance is a problematic task instance, and then determine whether the task instance at the same level as the directly upstream task instance is a problematic task instance. After the task instances at the same level are determined, it then determines whether there is a problematic task instance in the next level (i.e., the level of the downstream task instance that has a dependency relationship with the directly downstream task instance).

[0129] The technical solution provided by the above embodiments can bring at least the following beneficial effects: As shown in S108, the electronic device can add the identifier of each problem task instance in at least one problem task instance and the corresponding output field of each problem task instance to the target queue, so that the electronic device can determine whether the task instances other than each problem task instance in the above at least two task instances are problem task instances in a preset order. This enables the determination of problem task instances in a certain order, improving the efficiency of problem task instance determination.

[0130] Combination Figure 1 ,like Figure 7As shown, the data processing method provided in this embodiment of the disclosure further includes S109-S110.

[0131] S109. The electronic device generates a directed acyclic graph corresponding to the complement event based on the recovery information of the complement event.

[0132] The directed acyclic graph includes task instance nodes and edges. The task instance nodes are used to represent task instances, and the edges are used to connect task instance nodes with dependencies.

[0133] Based on the description of the above embodiments, it should be understood that the dependency relationship between the two task instances is used to characterize the upstream and downstream relationship between the two task instances. For example, suppose the directed acyclic graph includes a first task instance node and a second task instance node. When the first task instance node (which is the endpoint of an edge, or the endpoint in the direction of the arrow on that edge) depends on the second task instance node (which is the starting point of that edge, or the starting point in the direction of the arrow on that edge), it means that the first task instance node is the (direct) downstream task instance node of the second task instance node.

[0134] S110. The electronic device performs a pruning operation on the target task instance node to obtain the target directed acyclic graph.

[0135] Wherein, the target task instance node is a task instance node other than at least one problem task instance node among at least two task instance nodes, the at least two task instance nodes are task instance nodes included in the directed acyclic graph corresponding to the above-mentioned complement event, and the at least one problem task instance node is a task instance node represented by the above-mentioned at least one problem task instance.

[0136] It should be understood that the target directed acyclic graph includes each of the at least one problem task instance nodes, and the edges between each of the problem task instance nodes and other problem task instance nodes, wherein the other problem task instance nodes are the task instance nodes other than each of the at least one problem task instance nodes.

[0137] It is understandable that the electronic device performs a pruning operation on the target task instance node, that is, deletes the target task instance node and the edge corresponding to the target task instance node (the edge can be understood as the edge between the task instance node that has a dependency relationship with the target task instance node and the target task instance node) in the directed acyclic graph corresponding to the above-mentioned complement event, so as to obtain a target directed acyclic graph with fewer task instance nodes and fewer edges. Then the electronic device only needs to perform data generation operation on the task instance nodes included in the target directed acyclic graph.

[0138] For example, Figure 8 This is an example of an electronic device performing a pruning operation on the target task instance nodes included in the directed acyclic graph (i.e., directed acyclic graph 101) corresponding to the complement event, in an embodiment of this disclosure, to obtain a target directed acyclic graph (directed acyclic graph 102). Specifically, Figure 8 In this context, a node represented by a single letter represents a task instance node. For example, a node represented by the letter A represents task instance node A.

[0139] like Figure 8 As shown, the directed acyclic graph 101 includes 8 task instance nodes and 7 edges. Specifically, the 8 task instance nodes include task instance node A, task instance node B, task instance node C, task instance node D, task instance node E, task instance node F, task instance node G, and task instance node H; the 7 edges are used to connect task instance node A with task instance node B, task instance node B with task instance node D, task instance node B with task instance node E, task instance node C with task instance node E, task instance node C with task instance node F, task instance node C with task instance node G, and task instance node F with task instance node H.

[0140] Continue as Figure 8 As shown, since the electronic device determines that task instance nodes A, B, D, E, and G are target task instance nodes, and task instance nodes C, F, and H are problem task instance nodes (i.e., task instance nodes represented by problem task instances), the electronic device performs a pruning operation on these 5 target task instance nodes (specifically, deleting these 5 target task instance nodes and their corresponding 5 edges) to obtain a directed acyclic graph 102. The 3 task instance nodes included in the directed acyclic graph 102 are all problem task instance nodes.

[0141] Continue as Figure 7 As shown, the above-mentioned data generation operation for at least one problem task instance specifically includes S1041.

[0142] S1041. The electronic device performs a data generation operation on each task instance node included in the target directed acyclic graph.

[0143] Each of these task instance nodes represents a task instance node that is the same as the task instance node represented by each of the at least one problem task instances mentioned above.

[0144] Based on the description of the above embodiments, it should be understood that the electronic device performs a data generation operation on each task instance node included in the target directed acyclic graph to instruct each task instance node to generate data.

[0145] The technical solution provided by the above embodiments can bring at least the following beneficial effects: As shown in S109-S110 and S1041, the electronic device can generate a directed acyclic graph corresponding to the complement event based on the recovery information of the complement event; and perform a pruning operation on the target task instance nodes included in the directed acyclic graph to obtain a target directed acyclic graph with fewer task instance nodes and fewer edges. Since the target directed acyclic graph includes task instance nodes represented by each problem task instance in at least one problem task instance, the electronic device can perform a data generation operation on each problem task instance node included in the target directed acyclic graph, so that each problem task instance node generates data. This can avoid redundant or non-quality-prone task instance nodes (i.e., target task instance nodes) generating data, reduce resource waste, and improve resource utilization.

[0146] It is understood that, in practical implementation, the electronic device described in the embodiments of this disclosure may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding data processing methods. These hardware structures and / or software modules can constitute an electronic device. Those skilled in the art should readily recognize that, based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0147] Based on this understanding, the present disclosure also provides a data processing apparatus. Figure 9 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure is shown. Figure 9 As shown, the data processing device 20 may include: an acquisition module 201, a determination module 202, and a processing module 203.

[0148] The acquisition module 201 is configured to acquire recovery information of the complement event, which includes the identifiers of at least two task instances and the dependencies between the at least two task instances.

[0149] The acquisition module 201 is also configured to acquire data information of the at least two task instances, wherein the data information of one task instance includes the input information and the output information of the task instance.

[0150] The determination module 202 is configured to determine at least one problematic task instance from the at least two task instances based on the dependency relationship between the at least two task instances and the data information of the at least two task instances. The problematic task instance is used to characterize the task instance in which the generated data has quality problems. The number of the at least one problematic task instance is less than the number of the at least two task instances.

[0151] Processing module 203 is configured to perform a data generation operation on the at least one problem task instance.

[0152] Optionally, the acquisition module 201 is specifically configured to acquire the task details corresponding to each of the at least two task instances, wherein the task details corresponding to a task instance include the query statement corresponding to the task instance.

[0153] The processing module 203 is also configured to perform hash processing on the query statements corresponding to the at least two task instances to obtain the hash values ​​corresponding to the at least two task instances.

[0154] The acquisition module 201 is further configured to, if a first hash value exists in the stored target correspondence, retrieve the data information of the first task instance from the target correspondence table. The target correspondence includes the data information of multiple task instances and the hash value corresponding to each of the multiple task instances. The first hash value is the hash value corresponding to the first task instance, and the first task instance is one of the at least two task instances.

[0155] Optionally, the processing module 203 is further configured to, if the first hash value does not exist in the target correspondence, parse the query statement corresponding to the first task instance to obtain the data information of the first task instance.

[0156] The processing module 203 is also configured to add the data information of the first hash value and the first task instance to the target correspondence.

[0157] Optionally, the input information of a task instance includes an input field, and the output information of the task instance includes an output field.

[0158] The determination module 202 is specifically configured to, when the current task instance is a problem task instance, determine whether the input field corresponding to the current task instance exists in the input field of the direct downstream task instance of the current task instance, where the current task instance is one of the at least two task instances, and the direct downstream task instance is a downstream task instance that has a dependency relationship with the current task instance.

[0159] The determination module 202 is further configured to determine the direct downstream task instance as a problem task instance when the input field corresponding to the current task instance contains the output field corresponding to the current task instance.

[0160] Optionally, the determining module 202 is further configured to determine to perform a pruning operation on the direct downstream task instance when the input field corresponding to the direct downstream task instance does not contain the output field corresponding to the previous task instance.

[0161] Optionally, the processing module 203 is further configured to add the identifier of each problem task instance in the at least one problem task instance and the output field corresponding to each problem task instance to a target queue, the target queue being used to characterize the determined order of each problem task instance.

[0162] Optionally, the processing module 202 is further configured to generate a directed acyclic graph corresponding to the complement event based on the recovery information. The directed acyclic graph includes task instance nodes and edges, where the task instance nodes are used to represent task instances and the edges are used to connect task instance nodes with dependencies.

[0163] The processing module 202 is further configured to perform a pruning operation on the target task instance nodes to obtain a target directed acyclic graph. The target task instance nodes are task instance nodes other than at least one problem task instance node among at least two task instance nodes. The at least two task instance nodes are task instance nodes included in the directed acyclic graph. The at least one problem task instance node is the task instance node represented by the at least one problem task instance.

[0164] The processing module 202 is specifically configured to perform a data generation operation on each task instance node included in the target directed acyclic graph, wherein each task instance node is a task instance node represented by each problem task instance in the at least one problem task instance.

[0165] As described above, the embodiments of this disclosure can divide the data processing device into functional modules according to the above method examples. The integrated modules can be implemented in hardware or as software functional modules. Furthermore, it should be noted that the module division in these embodiments is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module.

[0166] The specific methods by which each module performs its operations and the beneficial effects of the data processing apparatus in the above embodiments have been described in detail in the foregoing method embodiments, and will not be repeated here.

[0167] Figure 10 This is a schematic diagram of another data processing apparatus provided in this disclosure. Figure 10 The data processing apparatus 30 may include at least one processor 301 and a memory 303 for storing processor-executable instructions. The processor 301 is configured to execute the instructions in the memory 303 to implement the data processing method described in the above embodiments.

[0168] In addition, the data processing device 30 may also include a communication bus 302 and at least one communication interface 304.

[0169] Processor 301 may be a processor (central processing unit, CPU), microprocessor unit, ASIC, or one or more integrated circuits for controlling the execution of programs according to the present disclosure.

[0170] The communication bus 302 may include a path for transmitting information between the aforementioned components.

[0171] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0172] Memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit.

[0173] The memory 303 stores instructions for executing the present invention, and the processor 301 controls the execution of these instructions. The processor 301 executes the instructions stored in the memory 303 to implement the functions of the method disclosed herein.

[0174] In a specific implementation, as one example, processor 301 may include one or more CPUs, for example... Figure 10 CPU0 and CPU1 in the CPU.

[0175] In a specific implementation, as one example, the data processing device 30 may include multiple processors, such as... Figure 10 Processors 301 and 307 are described herein. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0176] In a specific implementation, as one embodiment, the data processing device 30 may further include an output device 305 and an input device 306. The output device 305 communicates with the processor 301 and can display information in various ways. For example, the output device 305 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 306 communicates with the processor 301 and can accept user input in various ways. For example, the input device 306 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0177] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the data processing device 30, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0178] In addition, this disclosure also provides a computer-readable storage medium including instructions that, when executed by a processor, cause the processor to perform the data processing method provided in the above embodiments.

[0179] In addition, this disclosure also provides a computer program product including instructions that, when executed by a processor, cause the processor to perform the data processing method provided in the above embodiments.

[0180] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A data processing method, characterized in that, include: Obtain recovery information for the complement event, the recovery information including the identifiers of at least two task instances and the dependencies between the at least two task instances; Obtain data information from the at least two task instances, wherein the data information of one task instance includes the input information and the output information of the task instance; Based on the dependencies between the at least two task instances and the data information of the at least two task instances, at least one problematic task instance is determined from the at least two task instances. The problematic task instance includes: a task instance whose generated data has quality issues, and a task instance whose input information is associated with the output information of a dependent problematic task instance. The number of the at least one problematic task instance is less than the number of the at least two task instances. The at least one problematic task instance includes: the current task instance and a direct downstream task instance. The output information of the current task instance is associated with the input information of the direct downstream task instance. The direct downstream task instance is a downstream task instance that has a dependency relationship with the current task instance. Perform a data generation operation on the at least one problem task instance.

2. The data processing method according to claim 1, characterized in that, The acquisition of data information from the at least two task instances includes: Obtain the task details corresponding to each of the at least two task instances, wherein the task details corresponding to a task instance include the query statement corresponding to the task instance; Hash the query statements corresponding to each of the at least two task instances to obtain the hash values ​​corresponding to each of the at least two task instances; If a first hash value exists in the stored target correspondence, the data information of the first task instance is obtained from the target correspondence table. The target correspondence includes the data information of multiple task instances and the hash value corresponding to each of the multiple task instances. The first hash value is the hash value corresponding to the first task instance, and the first task instance is one of the at least two task instances.

3. The data processing method according to claim 2, characterized in that, The method further includes: If the first hash value does not exist in the target correspondence, the query statement corresponding to the first task instance is parsed to obtain the data information of the first task instance; Add the data information of the first hash value and the first task instance to the target correspondence.

4. The data processing method according to claim 1, characterized in that, The input information of a task instance includes an input field, and the output information of the task instance includes an output field. The step of determining at least one problem task instance from the at least two task instances based on the dependencies between the at least two task instances and the data information of the at least two task instances includes: When the current task instance is a problem task instance, determine whether there is an output field corresponding to the current task instance in the input field corresponding to the direct downstream task instance of the current task instance. The current task instance is one of the at least two task instances, and the direct downstream task instance is a downstream task instance that has a dependency relationship with the current task instance. When the input field corresponding to the direct downstream task instance contains the output field corresponding to the current task instance, the direct downstream task instance is determined to be a problem task instance.

5. The data processing method according to claim 4, characterized in that, The method further includes: If the input field corresponding to the preceding task instance does not exist in the input field corresponding to the direct downstream task instance, it is determined that a pruning operation will be performed on the direct downstream task instance.

6. The data processing method according to claim 4, characterized in that, The method further includes: The identifier of each problem task instance and the corresponding output field of each problem task instance are added to the target queue, which is used to characterize the determination order of each problem task instance.

7. A data processing apparatus, characterized in that, include: Acquisition module, determination module, and processing module; The acquisition module is configured to acquire recovery information of the complement event, the recovery information including the identifiers of at least two task instances and the dependency relationship between the at least two task instances; The acquisition module is further configured to acquire data information of the at least two task instances, wherein the data information of one task instance includes the input information and the output information of the task instance; The determining module is configured to determine at least one problematic task instance from the at least two task instances based on the dependency relationship between the at least two task instances and the data information of the at least two task instances. The problematic task instance is used to characterize the task instance whose generated data has quality problems. The number of the at least one problematic task instance is less than the number of the at least two task instances. The at least one problematic task instance includes: the current task instance and the direct downstream task instance. The output information of the current task instance is associated with the input information of the direct downstream task instance. The direct downstream task instance is a downstream task instance that has a dependency relationship with the current task instance. The processing module is configured to perform a data generation operation on the at least one problem task instance.

8. An electronic device, characterized in that, The electronic device includes: processor; A memory configured to store processor-executable instructions; The processor is configured to execute the instructions to implement the data processing method as described in any one of claims 1-6.

9. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is able to perform the data processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the data processing method as described in any one of claims 1-6.

Images