Data processing method and device, medium, equipment and computer program product

Abstract

A data processing method and apparatus, a medium, a device and a computer program product, the method comprising: determining whether processing data obtained by data processing by a data processing node is abnormal, the processing data corresponding to the data processing node being stored in a first data table; acquiring original data corresponding to the data processing node under the condition that the processed data is determined to be abnormal; processing the original data according to the processing logic corresponding to the data processing node, and writing the obtained processing data into a second data table; updating the first data table according to the second data table, and generating a change log corresponding to the first data table; and sending a change log to a downstream processing node of the data processing node, wherein the change log is used for triggering the downstream processing node to perform data updating. In this way, the intermediate data in the data processing process can be calibrated, and there is no need to create a repair link which is completely the same as the production environment.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a data processing method, apparatus, medium, device, and computer program product. Background Art

[0002] In the field of big data, a common scenario is the need to perform real-time statistical calculations such as aggregation of the day's data. For example, it is necessary to aggregate and calculate indicators such as conversions and transactions brought about by the advertising plan launched that day so that users can query and determine the effectiveness of the advertising plan.

[0003] However, due to the complex design and implementation of real-time systems, data statistics may deviate due to system or network jitter, leading to statistical errors in data indicators. In related technologies, a repair link is usually built that is exactly the same as the production environment, including real-time computing tasks and data transmission channels. Therefore, when data indicator statistics are incorrect, the repair link can be recalculated based on upstream data, completing the entire upstream and downstream calculations of the data. The final result of the calculation is then overwritten with the data in the production environment, completing the entire repair process. Summary of the Invention

[0004] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a data processing method, the method comprising: Determining whether processed data obtained by a data processing node through data processing has an abnormality, wherein the processed data corresponding to the data processing node is stored in a first data table; When it is determined that the processed data has an anomaly, obtaining original data corresponding to the data processing node, wherein the original data is stored in a data lake; Processing the original data according to the processing logic corresponding to the data processing node, and writing the obtained processed data into a second data table; Update the first data table according to the second data table, and generate a change log corresponding to the first data table; The change log is sent to a downstream processing node of the data processing node, where the change log is used to trigger the downstream processing node to update data. The downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

[0006] In a second aspect, the present disclosure provides a data processing device, the device comprising: a first determining module, configured to determine whether processed data obtained by a data processing node through data processing has an abnormality, wherein the processed data corresponding to the data processing node is stored in a first data table; an acquisition module, configured to acquire original data corresponding to the data processing node when determining that the processed data has an anomaly, wherein the original data is stored in a data lake; a first processing module, configured to process the original data according to the processing logic corresponding to the data processing node, and write the obtained processed data into a second data table; an updating module, configured to update the first data table according to the second data table and generate a change log corresponding to the first data table; The second processing module is used to send the change log to the downstream processing node of the data processing node, and the change log is used to trigger the downstream processing node to update data. The downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

[0007] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processing device.

[0008] In a fourth aspect, the present disclosure provides an electronic device, comprising: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect.

[0009] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0010] Therefore, through the above technical solution, the processed data obtained by the data processing node can be detected for anomalies, so that errors in the intermediate data obtained in the process of data indicator statistics can be discovered, and in the present disclosure, the original data can be stored based on the data lake, so that logical processing can be re-performed based on the original data to calibrate the processed data obtained by the data processing node, and trigger the downstream processing node to make changes in combination with the change log. In addition, in the solution of the present disclosure, the intermediate data obtained by the data processing node can be stored in the form of a data table, so that calibration can be performed on the intermediate data. During the data calibration process, there is no need to create a repair link that is exactly the same as the production environment. By correcting the data in the first data table, the downstream data can be automatically repaired, thereby improving the efficiency of data processing calibration, ensuring the accuracy of the data indicators obtained by data processing, and providing effective data support for users to perform data queries.

[0011] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 It is a flowchart of a data processing method provided according to an embodiment of the present disclosure.

[0013] Figure 2 It is a flowchart of data processing provided according to an embodiment of the present disclosure.

[0014] Figure 3 is a block diagram of a data processing device provided according to one embodiment of the present disclosure.

[0015] Figure 4 A schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0016] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0018] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0026] At the same time, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0027] Figure 1 As shown in FIG, it is a flow chart of a data processing method provided according to an embodiment of the present disclosure. Figure 1 As shown, the method may include: In step 11, it is determined whether there is an abnormality in the processed data obtained by the data processing node, wherein the processed data corresponding to the data processing node is stored in a first data table.

[0028] The data processing node is used to process data received from an upstream node to obtain processed data. The first data table can be a data table stored in a database to store intermediate data during the data processing process. System or network anomalies may cause anomalies in the processed data, affecting the final data statistical results. Therefore, in this embodiment, anomaly detection can be performed on the processed data.

[0029] As an example, after each data processing node completes data processing, the processed data obtained can be automatically detected for anomalies. A data processing chain may include multiple data processing nodes. Anomalies in the processed data obtained by one data processing node may affect the accuracy of the processed data of its downstream nodes. As another example, when anomalies are determined in the final statistical data obtained, anomaly detection can be performed on the processed data obtained by each data processing node along the path to the final statistical data to achieve attribution analysis of the anomaly in the target statistical data and provide data support for subsequent data correction.

[0030] In step 12, when it is determined that there is an abnormality in the processed data, the original data corresponding to the data processing node is obtained, wherein the processed data corresponding to the data processing node is stored in the first data table, and the original data is stored in the data lake.

[0031] A data lake is a data storage architecture used to store, manage, and analyze large-scale, diverse data. Raw data can be reported by terminals, and metrics are derived through data processing. For example, in online shopping scenarios, raw data can include the livestreams users watched, as well as the products they ordered, clicked, or browsed within those livestreams.

[0032] In this embodiment, raw data can be stored in the data lake in the form of Paimon tables, enabling subsequent anomaly detection based on the raw data. Accordingly, the first data table can be a Paimon table, allowing for storage of intermediate data during the processing of the raw data. Paimon tables can support multiple table types, such as tables with primary keys (PKs). These tables have one or more columns that uniquely identify records. The primary key ensures that each record is unique within the table, facilitating insert, update, and delete operations within the table.

[0033] As an example, the original data corresponding to the data processing node is obtained, that is, the original data required for determining the processing data of the data processing node can be obtained from the data lake.

[0034] In step 13, the original data is processed according to the processing logic corresponding to the data processing node, and the obtained processed data is written into the second data table.

[0035] The processing logic corresponding to the data processing node may be the processing logic of the data processing node on the data processing path to which the data processing node belongs for obtaining the processed data. Figure 2 As shown, the process in box A can represent a schematic diagram of the data processing path for obtaining data metrics. For example, data can be received from the cloud-native messaging engine BMQ and processed by Flink stream processing nodes to store the received raw data in a data lake, such as Paimon A. Flink stream processing node J1 then processes the data in Paimon A and stores the processed data in Paimon B. Flink stream processing node J3 processes the data in Paimon B and stores the processed data in Data Table 1. Data Table 1 can be implemented based on Doris in the data lake, an analytical database based on the MPP (Massively Parallel Processing) architecture. Data Tables 1 and 2 are used to provide query services to users. Similarly, Data Table 2 can be obtained by processing the raw data using Flink stream processing nodes J2, J4, and J5. Among them, the upstream and downstream relationships between different data processing nodes can construct a data processing path for processing the original data.

[0036] For example, if the determined data processing node is node J1, the processing logic corresponding to the data processing node is the processing logic of node J1. If the determined data processing node is node J2, the processing logic corresponding to the data processing node is the processing logic of nodes J1 and J2.

[0037] In this step, the code of the processing logic corresponding to the data processing node may be copied, so as to calculate the original data based on the copied code, and the calculated data may be written into the second data table for storage.

[0038] In step 14, the first data table is updated according to the second data table, and a change log corresponding to the first data table is generated.

[0039] Wherein, the second data table contains accurate data obtained after performing the same logical processing on the original data as the data in the first data table, the first data table can be updated so that the data in the first data table and the data in the second data table are the same. Wherein, the second data table can be a Paimon table. Wherein, Paimon supports multiple different Changelog Producers, and after the first data table is updated, a change log can be generated based on the ChangelogProducer supported by Paimon. The change log can include various change records of the updates to the first data table. The implementation method can be based on the generation method in the prior art, and this disclosure is not limited to this.

[0040] by Figure 2 For example, if node J2 consumes Paimon A table to calculate and output Paimon C table, and data error occurs due to system or network reasons, it is determined that the processed data of node J2 is abnormal, that is, there is abnormal data in Paimon C. Then Paimon C can be used as the first data table, and the following can be done: Figure 2 As shown in box B, Paimon C' is obtained based on the original data, and the data in Paimon C is updated based on Paimon C'.

[0041] In step 15, a change log is sent to a downstream processing node of the data processing node. The change log is used to trigger the downstream processing node to update data. The downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

[0042] by Figure 2For example, if it is determined that the processed data of node J2 contains an anomaly, that is, abnormal data exists in Paimon C, and the downstream processing node of node J2 is node J4, then in this step, the change log can be sent to node J4 so that node J4 can reprocess the data based on the change records in the change log and update the data in Paimon D. After the data in Paimon D is updated, a change log corresponding to Paimon D can be generated and further sent to node J5. Accordingly, node J5 can update the data in Data Table 2 based on the change log of Paimon D, thereby achieving data calibration in Data Table 2 and ensuring the accuracy of the statistical data indicators in the data table.

[0043] As an example, after the Paimon table generates a change log, message transmission can be performed through BMQ. That is, after the Paimon table generates a change log, the downstream processing node of the current data processing node can determine that a change log has been generated by monitoring the Paimon table, and then send the change log to the downstream processing node. The downstream processing node can read the change log into its data processing task to perform data processing based on the change log.

[0044] Therefore, through the above technical solution, the processed data obtained by the data processing node can be detected for anomalies, so that errors in the intermediate data obtained in the process of data indicator statistics can be discovered, and in the present disclosure, the original data can be stored based on the data lake, so that logical processing can be re-performed based on the original data to calibrate the processed data obtained by the data processing node, and trigger the downstream processing node to make changes in combination with the change log. In addition, in the solution of the present disclosure, the intermediate data obtained by the data processing node can be stored in the form of a data table, so that calibration can be performed on the intermediate data. During the data calibration process, there is no need to create a repair link that is exactly the same as the production environment. By correcting the data in the first data table, the downstream data can be automatically repaired, thereby improving the efficiency of data processing calibration, ensuring the accuracy of the data indicators obtained by data processing, and providing effective data support for users to perform data queries.

[0045] In some possible embodiments, an exemplary implementation of determining whether processed data obtained by a data processing node through data processing has an abnormality may include: When a data processing checkpoint is determined to have been reached, first data corresponding to the checkpoint in the first data table is obtained. In the data lake-based data processing process, checkpoints can be configured. For example, a checkpoint can be configured at a preset time interval, such as every five minutes. The checkpoint mechanism can be used to write data in the data lake to a stable storage medium and create a recoverable checkpoint.

[0046] As an example, incremental queries can be performed based on checkpoints in a data lake. Each time a checkpoint is reached, a version of the first data table can be generated. After data processing by the data processing node, the processed data is written into the storage structure of the Paimon table and merged with the version corresponding to the previous checkpoint to obtain the version of the first data table corresponding to the checkpoint, that is, a snapshot of the first data table is obtained. In this embodiment, the difference data between the snapshot of the first data table corresponding to the previous checkpoint and the snapshot of the first data table corresponding to the current checkpoint can be used as the first data. As an example, the current checkpoint is 11:05 and the previous checkpoint is 11:00. The first data set corresponding to the current checkpoint is the data written after data processing on the original data from 11:00 to 11:05.

[0047] Afterwards, original data corresponding to the checkpoint is obtained, and the original data is processed according to the processing logic corresponding to the data processing node to obtain second data corresponding to the checkpoint.

[0048] Wherein, each of the checkpoints is associated with a time range, that is, the time range associated with the checkpoint is the range formed by the previous checkpoint to the current checkpoint. Accordingly, the obtaining of the raw data corresponding to the checkpoint may include: The original data within the range from the start time of the time range associated with the checkpoint to the target time is obtained as the original data corresponding to the checkpoint, and the target time is the time for determining abnormal recovery of the data processing node.

[0049] As in the above example, the time range associated with the checkpoint with the exception is 11:00-11:05, then the start time of the time range 11:00 can be used as the start time of the exception. When there is an exception in the processing data, an operation and maintenance operation can be triggered to determine the cause of the exception and repair it. For example, there may be insufficient resource allocation for online tasks, which can be repaired by expanding resources. After that, the corresponding personnel will determine whether the exception has been restored. When it is determined that the exception has been restored, the end of the operation and maintenance operation can be confirmed. Accordingly, the time when the operation and maintenance operation is completed can be used as the time when the exception of the data processing node is recovered. For example, the time when the exception of the data processing node is recovered is 12:00, then the raw data between 11:00 and 12:00 may have anomalies when processing, and the raw data in this time period can be used as the raw data corresponding to the checkpoint for calibration processing.

[0050] If the first data and the second data are different, it is determined that the processed data obtained by the data processing node through data processing has an abnormality.

[0051] If the first data and the second data are different, it means that the data obtained by data processing based on the data processing node is different from the data obtained by calibration processing based on the same original data. At this time, it can be considered that the processed data of the data processing node is abnormal and needs to be corrected.

[0052] Therefore, through the above technical solution, the data in the first data table can be incrementally queried through checkpoints, and by replicating the processing logic of the data processing node, data calibration can be performed based on the original data in the time range where the anomaly exists, so as to generate calibration data at the lowest cost and improve the efficiency of data calibration.

[0053] In some possible embodiments, an exemplary implementation of determining whether processed data obtained by a data processing node through data processing has an abnormality may include: Matching is performed on the processed data according to anomaly detection rules corresponding to the processed data.

[0054] Among them, anomaly detection rules can be pre-configured based on the characteristics of the processed data. As an example, if the value of processed data B1 is stable over a continuous period of time, anomaly detection rule R1 can be set for the processed data B1. If the absolute value of the difference between the processed data B1 of two adjacent checkpoints is greater than a first threshold, the processed data is considered to match the anomaly detection rule. For another example, if the value of processed data B2 is stable during the same period of time every day, anomaly detection rule R2 can be set for the processed data B2. If the absolute value of the difference between the processed data B2 and the processed data B2 of the checkpoint at the same time on the previous day is greater than a second threshold, the processed data is considered to match the anomaly detection rule.

[0055] If the processed data matches the anomaly detection rule, it is determined that the processed data obtained by the data processing node through data processing has an anomaly.

[0056] Therefore, anomaly detection can be performed on processed data by pre-configuring corresponding anomaly detection rules, which can be dynamically configured based on actual application scenarios, thereby expanding the scope of application of the method provided by the present disclosure.

[0057] Accordingly, the obtaining of the original data corresponding to the data processing node includes: A start time is determined based on time information corresponding to abnormal data in the processed data and a preset duration.

[0058] The preset duration can be set based on the actual application scenario, such as 10 minutes. The time corresponding to the checkpoint where the abnormal data is determined can be used as the time information corresponding to the abnormal data, and the preset time before the time information can be used as the start time. For example, 10 minutes before the time information corresponding to the abnormal data can be used as the start time. If the time information corresponding to the abnormal data is 11 o'clock, 10:50 can be used as the start time, so that the original data with processing anomalies can be fully obtained.

[0059] Then, the raw data within the range of the start time to the target time is obtained as the raw data corresponding to the data processing node, wherein the target time is the time when the abnormality recovery of the data processing node is determined. The implementation method of this step has been described above and will not be repeated here.

[0060] Therefore, through the above technical solution, the original data corresponding to the abnormal data can be obtained, thereby effectively reducing the amount of data calculation required for data calibration and improving the efficiency of data calibration.

[0061] In some possible embodiments, processing the original data according to the processing logic corresponding to the data processing node and writing the obtained processed data into the second data table may include: The second data table is created according to the table structure of the first data table, and is used to store calibration data recalculated based on the original data. As an example, the table structure of the second data table is the same as that of the first data table.

[0062] As an example, the second data table can be created based on the table structure of the first data table and the command for creating the data table, such as the creation of the second data table can be implemented based on an SQL statement. As another example, the second data table can be created based on the branch mechanism supported by the data lake. For example, based on the branch mechanism, a branch can be generated in the Paimon table, and a second data table with the same table structure as the first data table can be generated in the branch. The second data table has the same table name as the first data table. When writing the second data table, you can determine which branch the data table is written to by adding a prompt word (such as the branch name). As an example, you can build a calibration branch, and then you can generate the second data table under the calibration branch.

[0063] Batch processing is performed according to the processing logic corresponding to the data processing node and the original data, wherein the processing logic corresponding to the data processing node includes the processing logic of the data processing node in the process from the original data to the acquisition of the processed data.

[0064] The method for determining the processing logic corresponding to the data processing node has been described in detail above and will not be repeated here. In this step, a Flink batch processing task can be submitted based on the processing logic and the original data to perform data calibration in an offline manner. The data obtained from the batch processing is then written to the second data table.

[0065] As an example, if the second data table is created based on the branch mechanism, when writing the second data table, a prompt word (such as a branch name) can be added to determine which data table under which branch to write to, so as to ensure accurate writing of the calibration data.

[0066] In some possible embodiments, updating the first data table according to the second data table may include: An updated primary key is determined according to the primary keys in the first data table and the second data table and the data corresponding to the primary keys.

[0067] Among them, the Paimon table, as a primary key table, can support data update operations with a fixed primary key. Therefore, during the data calibration process, the primary keys and corresponding data in the first and second data tables can be matched to confirm which data in the first data table the correct data in the second data table should be updated to, thereby realizing the calibration of abnormal data in the first data table one by one.

[0068] As an example, determining to update the primary key according to the primary keys and the data corresponding to the primary keys in the first data table and the second data table, respectively, includes: For the primary key in the second data table, if the data corresponding to the primary key in the second data table is different from the data corresponding to the primary key in the first data table, the primary key is determined to be the first updated primary key; if the primary key does not exist in the first data table, the primary key is determined to be the second updated primary key.

[0069] As an example, if the data in the second data table is correct, the primary key in the second data table can be traversed to determine the data that needs to be updated in the first data table. If the data corresponding to the primary key in the second data table is different from the data corresponding to the primary key in the first data table, it means that the data corresponding to the primary key is stored in the first data table, but the data in the first data table is incorrect. In this case, the primary key is determined to be the first update primary key, that is, the primary key that needs to be modified.

[0070] If the primary key does not exist in the first data table, the primary key is determined to be a second updated primary key, that is, data that needs to be inserted into the first data table.

[0071] For the primary key in the first data table, if the primary key does not exist in the second data table, the primary key is determined to be a third updated primary key, and the updated primary key includes the first updated primary key, the second updated primary key and the third updated primary key.

[0072] The first data table may contain data that needs to be deleted, so the primary key in the first data table can be traversed. If the primary key does not exist in the second data table, it means that the data corresponding to the primary key should be deleted, and the primary key is determined to be the third updated primary key, that is, the primary key that needs to be deleted in the first data table.

[0073] Therefore, by comparing the primary keys in the first data table and the second data table, the primary keys in the first data table that need to be calibrated and the data corresponding to the primary keys can be quickly and accurately determined, thereby improving the accuracy and efficiency of data calibration.

[0074] Based on the data corresponding to the updated primary key in the second data table, the data corresponding to the updated primary key in the first data table is updated.

[0075] As an example, the data in the first data table is updated based on a cumulative method. In actual application scenarios, data processing nodes usually perform data processing based on statistical methods such as group-based cumulative summation. In order to improve the efficiency of data processing, the data update method in the data table can be set to cumulative, that is, the updated data is obtained by summing the data in the data table and the written value.

[0076] Accordingly, updating the data corresponding to the updated primary key in the first data table based on the data corresponding to the updated primary key in the second data table may include: For the first updated primary key, the difference between the data corresponding to the first updated primary key in the second data table and the data corresponding to the first data table is used as updated data, and the data corresponding to the first updated primary key in the first data table is updated based on the updated data.

[0077] For a primary key that requires data modification, if the corresponding data in the second data table is 15 and the corresponding data in the first data table is 10, the corresponding data in the first data table needs to be updated to 15. During this process, the updated data can be determined to be 5 (i.e., 15-5). This updated data is then used to update the data in the first data table. Specifically, the data in the first data table, 10, and the updated data, 5, are accumulated to obtain the updated data, 15. This ensures data consistency between the first and second data tables, thus completing the update of the first data table.

[0078] As an example, a FULL JOIN statement may be used to join data with the same primary key in the first data table and the second data table and calculate the difference to obtain the updated data. Thereafter, an INSERT INTO statement may be used to update the data in the first data table.

[0079] For the second updated primary key, the second updated primary key is created in the first data table and data corresponding to the second updated primary key is inserted into the first data table.

[0080] For the primary key that needs to be inserted, a second updated primary key can be created in the first data table, and the data corresponding to the second updated primary key can be used as the update data. Based on the update data, the data of the primary key created in the first data table is updated. If the data corresponding to the primary key created in the first data table is 0 by default, 0 and the data corresponding to the second updated primary key 4 can be accumulated so that the data corresponding to the primary key in the first data table after the update is 4, ensuring the consistency of the data in the first data table and the data in the second data table.

[0081] For the third update primary key, the opposite value of the data corresponding to the third update primary key in the first data table is used as update data, and the data corresponding to the third update primary key in the first data table is updated based on the update data.

[0082] For data that needs to be deleted, that is, online data that should not exist, the opposite value of the data corresponding to the third updated primary key in the first data table can be used as the update data. For example, if the data corresponding to the third updated primary key in the first data table is 6, -6 can be used as the update data. Subsequently, based on the update data, the data corresponding to the third updated primary key in the first data table is updated. That is, the data 6 in the first data table and the update data -6 are accumulated. The updated data in the first data table is 0, which can be considered consistent with the data in the second data table. However, there will be a record with data 0 in the first data table, which will not affect subsequent data statistics.

[0083] Therefore, through the above technical solution, the first data table can be updated based on the original data update logic in the first data table and the data found in the second data table, ensuring the accuracy of the data in the first data table, providing effective data support for data processing in downstream processing nodes, improving the accuracy and efficiency of data processing, and realizing rapid calibration of erroneous data.

[0084] As another example, if the data in the first data table is updated in an overwriting manner, then for the first and second updated primary keys, the data corresponding to the corresponding primary keys in the second data table can be overwritten in the first data table. For the third updated primary key, a default value of 0 can be overwritten in the first data table to achieve rapid calibration of the data in the first data table. This can be implemented based on data modification, insertion, and deletion commands in the database field, and this disclosure is not limited to this.

[0085] Based on the same inventive concept, the present disclosure also provides a data processing device, such as Figure 3 As shown, the device 10 includes: The first determining module 100 is used to determine whether there is an abnormality in the processed data obtained by the data processing node; An acquisition module 200 is configured to acquire original data corresponding to the data processing node when it is determined that the processed data has an anomaly, wherein the processed data corresponding to the data processing node is stored in a first data table, and the original data is stored in a data lake; A first processing module 300 is configured to process the original data according to the processing logic corresponding to the data processing node, and write the obtained processed data into a second data table; An updating module 400 is configured to update the first data table according to the second data table and generate a change log corresponding to the first data table; The second processing module 500 is used to send the change log to the downstream processing node of the data processing node, and the change log is used to trigger the downstream processing node to update data. The downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

[0086] Optionally, the first determining module includes: A first acquisition submodule is configured to, when it is determined that a checkpoint for data processing has been reached, acquire first data corresponding to the checkpoint in the first data table; A second acquisition submodule is used to obtain the original data corresponding to the checkpoint; a first processing submodule, configured to process the original data according to the processing logic corresponding to the data processing node to obtain second data corresponding to the checkpoint; The first determining submodule is configured to determine that an abnormality exists in the processed data obtained by the data processing node through data processing if the first data and the second data are different.

[0087] Optionally, each of said checkpoints is associated with a time range; The second acquisition submodule is further configured to: The original data within the range from the start time of the time range associated with the checkpoint to the target time is obtained as the original data corresponding to the checkpoint, and the target time is the time for determining the abnormal recovery of the data processing node.

[0088] Optionally, the first determining module includes: a matching submodule, configured to match the processed data with an anomaly detection rule corresponding to the processed data; The second determining submodule is configured to determine that an anomaly exists in the processed data obtained by the data processing node through data processing if the processed data matches the anomaly detection rule.

[0089] Optionally, the acquisition module includes: A third determining submodule, configured to determine a start time based on time information corresponding to abnormal data in the processed data and a preset duration; The third acquisition submodule is used to acquire the original data within the range of the start time to the target time as the original data corresponding to the data processing node, wherein the target time is the time for determining the abnormal recovery of the data processing node.

[0090] Optionally, the first processing module includes: A creation submodule, configured to create the second data table according to the table structure of the first data table, wherein the table structure of the second data table is the same as that of the first data table; a second processing submodule, configured to perform batch processing based on the processing logic corresponding to the data processing node and the original data, wherein the processing logic corresponding to the data processing node includes the processing logic of the data processing node in the process from the original data to obtaining the processed data; The third processing submodule is configured to write the data obtained through batch processing into the second data table.

[0091] Optionally, the update module includes: A fourth determining submodule, configured to determine an updated primary key based on the primary keys in the first data table and the second data table and the data corresponding to the primary keys; The first updating submodule is configured to update the data corresponding to the update primary key in the first data table based on the data corresponding to the update primary key in the second data table.

[0092] Optionally, the fourth determining submodule includes: a fifth determination submodule, configured to, for the primary key in the second data table, determine that if the data corresponding to the primary key in the second data table is different from the data corresponding to the primary key in the first data table, the primary key is determined to be a first updated primary key; and if the primary key does not exist in the first data table, the primary key is determined to be a second updated primary key; The sixth determination submodule is used to determine, for the primary key in the first data table, if the primary key does not exist in the second data table, that the primary key is a third updated primary key, and the updated primary key includes the first updated primary key, the second updated primary key and the third updated primary key.

[0093] Optionally, the data in the first data table is updated based on a cumulative manner; The first update submodule includes: a second update submodule, configured to update the data corresponding to the first update primary key in the first data table based on the update data, using the difference between the data corresponding to the first update primary key in the second data table and the data corresponding to the first update primary key in the first data table as update data; A third updating submodule is configured to create the second update primary key in the first data table and insert data corresponding to the second update primary key into the first data table for the second update primary key; The fourth update submodule is used to use the opposite value of the data corresponding to the third update primary key in the first data table as update data for the third update primary key, and update the data corresponding to the third update primary key in the first data table based on the update data.

[0094] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device (e.g., a terminal device or a server) 600 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0095] like Figure 4As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0096] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Figure 4 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0097] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0098] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0099] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0100] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0101] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: determines whether there is an abnormality in the processed data obtained by the data processing node for data processing; when it is determined that the processed data has an abnormality, obtains the original data corresponding to the data processing node, wherein the processed data corresponding to the data processing node is stored in a first data table, and the original data is stored in a data lake; processes the original data according to the processing logic corresponding to the data processing node, and writes the obtained processed data into a second data table; updates the first data table according to the second data table, and generates a change log corresponding to the first data table; sends the change log to a downstream processing node of the data processing node, and the change log is used to trigger the downstream processing node to update the data, and the downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

[0102] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] The modules described in the embodiments of the present disclosure may be implemented in software or hardware. In some cases, the name of a module does not limit the module itself. For example, the first determination module may also be described as "a module for determining whether there are abnormalities in the processed data obtained by the data processing node during data processing."

[0105] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0106] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0107] According to one or more embodiments of the present disclosure, Example 1 provides a data processing method, the method including: Determine whether there is any abnormality in the processed data obtained by the data processing node; When it is determined that the processed data has an anomaly, obtaining original data corresponding to the data processing node, wherein the processed data corresponding to the data processing node is stored in a first data table, and the original data is stored in a data lake; Processing the original data according to the processing logic corresponding to the data processing node, and writing the obtained processed data into a second data table; Update the first data table according to the second data table, and generate a change log corresponding to the first data table; The change log is sent to a downstream processing node of the data processing node, where the change log is used to trigger the downstream processing node to update data. The downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

[0108] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein determining whether processed data obtained by data processing by a data processing node is abnormal includes: When it is determined that a checkpoint for data processing has been reached, obtaining first data corresponding to the checkpoint in the first data table; Obtaining original data corresponding to the checkpoint; Processing the original data according to the processing logic corresponding to the data processing node to obtain second data corresponding to the checkpoint; If the first data and the second data are different, it is determined that the processed data obtained by the data processing node through data processing has an abnormality.

[0109] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein each of the checkpoints is associated with a time range; The obtaining of original data corresponding to the checkpoint includes: The original data within the range from the start time of the time range associated with the checkpoint to the target time is obtained as the original data corresponding to the checkpoint, and the target time is the time for determining abnormal recovery of the data processing node.

[0110] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 1, wherein determining whether processed data obtained by data processing by a data processing node is abnormal includes: matching the processed data with an anomaly detection rule corresponding to the processed data; If the processed data matches the anomaly detection rule, it is determined that the processed data obtained by the data processing node through data processing has an anomaly.

[0111] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 4, wherein obtaining the original data corresponding to the data processing node includes: Determining a start time based on time information corresponding to abnormal data in the processed data and a preset duration; Original data within a range from the start time to a target time is acquired as original data corresponding to the data processing node, wherein the target time is a time for determining abnormal recovery of the data processing node.

[0112] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, wherein processing the original data according to the processing logic corresponding to the data processing node and writing the obtained processed data into the second data table includes: Creating the second data table according to the table structure of the first data table, wherein the table structure of the second data table is the same as that of the first data table; performing batch processing according to the processing logic corresponding to the data processing node and the original data, wherein the processing logic corresponding to the data processing node includes the processing logic of the data processing node in the process from the original data to obtaining the processed data; The data obtained by batch processing is written into the second data table.

[0113] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 1, wherein updating the first data table according to the second data table includes: Determine to update the primary key according to the primary keys in the first data table and the second data table and the data corresponding to the primary keys; Based on the data corresponding to the updated primary key in the second data table, the data corresponding to the updated primary key in the first data table is updated.

[0114] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 7, wherein determining to update the primary key based on the primary keys and the data corresponding to the primary keys in the first data table and the second data table, respectively, includes: For the primary key in the second data table, if the data corresponding to the primary key in the second data table is different from the data corresponding to the primary key in the first data table, then the primary key is determined to be a first updated primary key; if the primary key does not exist in the first data table, then the primary key is determined to be a second updated primary key; For the primary key in the first data table, if the primary key does not exist in the second data table, the primary key is determined to be a third updated primary key, and the updated primary key includes the first updated primary key, the second updated primary key and the third updated primary key.

[0115] According to one or more embodiments of the present disclosure, Example 9 provides the method of Example 8, wherein the data in the first data table is updated based on a cumulative manner; The updating of the data corresponding to the updated primary key in the first data table based on the data corresponding to the updated primary key in the second data table includes: For the first update primary key, use the difference between the data corresponding to the first update primary key in the second data table and the data corresponding to the first update primary key in the first data table as update data, and update the data corresponding to the first update primary key in the first data table based on the update data; For the second updated primary key, create the second updated primary key in the first data table and insert data corresponding to the second updated primary key into the first data table; For the third update primary key, the opposite value of the data corresponding to the third update primary key in the first data table is used as update data, and the data corresponding to the third update primary key in the first data table is updated based on the update data.

[0116] According to one or more embodiments of the present disclosure, Example 10 provides a data processing device, the device including: A first determining module is used to determine whether there is an abnormality in the processed data obtained by the data processing node; An acquisition module, configured to acquire original data corresponding to the data processing node when it is determined that the processed data has an anomaly, wherein the processed data corresponding to the data processing node is stored in a first data table, and the original data is stored in a data lake; a first processing module, configured to process the original data according to the processing logic corresponding to the data processing node, and write the obtained processed data into a second data table; an updating module, configured to update the first data table according to the second data table and generate a change log corresponding to the first data table; The second processing module is used to send the change log to the downstream processing node of the data processing node, and the change log is used to trigger the downstream processing node to update data. The downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

[0117] According to one or more embodiments of the present disclosure, Example 11 provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in any one of Examples 1-9 when executed by a processing device.

[0118] According to one or more embodiments of the present disclosure, Example 12 provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method described in any one of Examples 1-9.

[0119] According to one or more embodiments of the present disclosure, Example 13 provides a computer program product, including a computer program, which implements the steps of any one of the methods of Examples 1-9 when executed by a processor.

[0120] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the scope of the above disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0121] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0122] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. A data processing method, characterized in that: The method comprises: Determining whether processed data obtained by a data processing node through data processing has an abnormality, wherein the processed data corresponding to the data processing node is stored in a first data table; When it is determined that the processed data has an anomaly, obtaining original data corresponding to the data processing node, wherein the original data is stored in a data lake; Processing the original data according to the processing logic corresponding to the data processing node, and writing the obtained processed data into a second data table; Update the first data table according to the second data table, and generate a change log corresponding to the first data table; The change log is sent to a downstream processing node of the data processing node, where the change log is used to trigger the downstream processing node to update data. The downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

2. The method according to claim 1, characterized in that Determining whether the processed data obtained by the data processing node through data processing is abnormal includes: When it is determined that a checkpoint for data processing has been reached, obtaining first data corresponding to the checkpoint in the first data table; Obtaining original data corresponding to the checkpoint; Processing the original data according to the processing logic corresponding to the data processing node to obtain second data corresponding to the checkpoint; If the first data and the second data are different, it is determined that the processed data obtained by the data processing node through data processing has an abnormality.

3. The method according to claim 2, characterized in that Each of said checkpoints is associated with a time range; The obtaining of original data corresponding to the checkpoint includes: The original data within the range from the start time of the time range associated with the checkpoint to the target time is obtained as the original data corresponding to the checkpoint, and the target time is the time for determining the abnormal recovery of the data processing node.

4. The method according to claim 1, wherein Determining whether the processed data obtained by the data processing node through data processing is abnormal includes: matching the processed data with an anomaly detection rule corresponding to the processed data; If the processed data matches the anomaly detection rule, it is determined that the processed data obtained by the data processing node through data processing has an anomaly.

5. The method according to claim 4, characterized in that The obtaining of original data corresponding to the data processing node includes: Determining a start time based on time information corresponding to abnormal data in the processed data and a preset duration; Original data within a range from the start time to a target time is acquired as original data corresponding to the data processing node, wherein the target time is a time for determining abnormal recovery of the data processing node.

6. The method according to claim 1, characterized in that The processing of the original data according to the processing logic corresponding to the data processing node and writing the obtained processed data into the second data table includes: Creating the second data table according to the table structure of the first data table, wherein the table structure of the second data table is the same as that of the first data table; performing batch processing according to the processing logic corresponding to the data processing node and the original data, wherein the processing logic corresponding to the data processing node includes the processing logic of the data processing node in the process from the original data to obtaining the processed data; The data obtained by batch processing is written into the second data table.

7. The method according to claim 1, characterized in that The updating of the first data table according to the second data table includes: Determine to update the primary key according to the primary keys in the first data table and the second data table and the data corresponding to the primary keys; Based on the data corresponding to the updated primary key in the second data table, the data corresponding to the updated primary key in the first data table is updated.

8. The method according to claim 7, characterized in that The determining, based on the primary keys and the data corresponding to the primary keys in the first data table and the second data table, to update the primary keys, includes: For the primary key in the second data table, if the data corresponding to the primary key in the second data table is different from the data corresponding to the primary key in the first data table, then the primary key is determined to be a first updated primary key; if the primary key does not exist in the first data table, then the primary key is determined to be a second updated primary key; For the primary key in the first data table, if the primary key does not exist in the second data table, the primary key is determined to be a third updated primary key, and the updated primary key includes the first updated primary key, the second updated primary key and the third updated primary key.

9. The method according to claim 8, characterized in that The data in the first data table is updated based on the cumulative method; The updating of the data corresponding to the updated primary key in the first data table based on the data corresponding to the updated primary key in the second data table includes: For the first update primary key, use the difference between the data corresponding to the first update primary key in the second data table and the data corresponding to the first update primary key in the first data table as update data, and update the data corresponding to the first update primary key in the first data table based on the update data; For the second updated primary key, create the second updated primary key in the first data table and insert data corresponding to the second updated primary key into the first data table; For the third update primary key, the opposite value of the data corresponding to the third update primary key in the first data table is used as update data, and the data corresponding to the third update primary key in the first data table is updated based on the update data.

10. A data processing device, characterized in that: The device comprises: a first determining module, configured to determine whether processed data obtained by a data processing node through data processing has an abnormality, wherein the processed data corresponding to the data processing node is stored in a first data table; an acquisition module, configured to acquire original data corresponding to the data processing node when determining that the processed data has an anomaly, wherein the original data is stored in a data lake; a first processing module, configured to process the original data according to the processing logic corresponding to the data processing node, and write the obtained processed data into a second data table; an updating module, configured to update the first data table according to the second data table and generate a change log corresponding to the first data table; The second processing module is used to send the change log to the downstream processing node of the data processing node, and the change log is used to trigger the downstream processing node to update data. The downstream processing node is a node that processes data based on the processed data obtained by the data processing node.

11. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 9 are implemented.

12. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images