Data processing method and device, electronic equipment and storage medium
By reacquisitioning and processing unfinished target batch data in the streaming engine after the data processing task is restarted, the data inconsistency problem is solved to ensure the integrity and consistency of the data processing task.
Patent Information
- Application Number
- CN202411390351.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-02-25
AI Technical Summary
On the data source side of the streaming processing engine, when an exception occurs in the data processing task, the restart may cause the data received by the data destination to be incomplete and data loss, which will lead to data inconsistency problems.
After the data processing task is restarted, the location identification and traversal number of the target batch data are re-acquipped from the external data source, the remaining data is traversed and obtained, and the remaining data is processed and sent according to the data processing task.
Ensure that the data received by the destination end of the data is complete and without loss, and maintain data consistency between the source end and the destination end.
Smart Images
Figure CN119512701A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art
[0002] At present, after an exception occurs in a data processing task in the data source end of a streaming processing engine, it is usually necessary to restart the data processing task. After the data processing task is restarted, the data source end learns that the data processing task has consumed the target batch of data based on the data consumption record of the data processing task, and obtains the next batch of data after the target batch of data from the external data source based on the data consumption record, and processes each data in the next batch of data in sequence through the data processing task, and sends the processed data in sequence to the data destination end corresponding to the data processing task. However, when an exception occurs in the data processing task, if the data processing task has not yet processed all the data in the target batch of data, this method of directly obtaining the next batch of data based on the data consumption record after the data processing task is restarted will make the data received by the data destination end incomplete, and there will be data loss, resulting in data inconsistency problems between the data destination end and the external data source. Summary of the invention
[0003] The present application proposes a data processing method, device and electronic equipment.
[0004] The first aspect of the present application provides a data processing method, which includes: after a data processing task is restarted, obtaining a target data consumption record corresponding to the data processing task, wherein the target data consumption record includes: a location identifier of a target batch data being processed before the data processing task is restarted and a traversal sequence number of the target data in the target batch data, wherein the target data is the last data successfully processed by the data processing task before the data processing task is restarted; re-obtaining the target batch data corresponding to the location identifier from an external data source; traversing the target batch data to obtain the remaining data located after the traversal sequence number; and processing each data in the remaining data in turn according to the data processing task, and sending the processed data to the data destination corresponding to the data processing task.
[0005] The data processing method of the embodiment of the present application, after the data processing task is restarted, re-acquires the target batch data that was not processed last time by the data processing task from the external data source, and obtains the target data that was successfully processed for the target batch data before the data processing task is restarted, and obtains the remaining data after the target data from the target batch data, and processes each data in the remaining data according to the data processing task, and sends the processed data to the data destination corresponding to the data processing task. In this way, the unprocessed data in the target batch data that was not processed last time can continue to be processed, which will make the data received by the data destination complete, without data loss, and ensure the data consistency between the data destination and the external data source.
[0006] The second aspect of the present application proposes a data processing device, which includes: a first acquisition module, used to obtain a target data consumption record corresponding to the data processing task after the data processing task is restarted, wherein the target data consumption record includes: a location identifier of the target batch data being processed before the data processing task is restarted and a traversal sequence number of the target data in the target batch data, wherein the target data is the last data successfully processed by the data processing task before the restart; a second acquisition module, used to re-acquire the target batch data corresponding to the location identifier from an external data source; a traversal module, used to traverse the target batch data to obtain the remaining data located after the traversal sequence number; a processing module, used to process each data in the remaining data in turn according to the data processing task, and send the processed data to the data destination corresponding to the data processing task.
[0007] The data processing device of the embodiment of the present application, after the data processing task is restarted, re-acquires the target batch data that was not processed last time by the data processing task from the external data source, and obtains the target data that was successfully processed for the target batch data before the data processing task is restarted, and obtains the remaining data after the target data from the target batch data, and processes each data in the remaining data according to the data processing task, and sends the processed data to the data destination corresponding to the data processing task. In this way, the unprocessed data in the target batch data that was not processed last time can continue to be processed, which will make the data received by the data destination complete, without data loss, and ensure the data consistency between the data destination and the external data source.
[0008] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method as described in the first aspect when executing the computer program.
[0009] The fourth aspect of the present application provides a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the data processing method described in the first aspect.
[0010] The fifth aspect of the present application provides a computer program product, including a computer program, which implements the data processing method described in the first aspect when executed by a processor.
[0011] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0013] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;
[0014] Figure 2 A flowchart of another data processing method provided in an embodiment of the present application;
[0015] Figure 3 A flowchart of another data processing method provided in an embodiment of the present application;
[0016] Figure 4 A flowchart of another data processing method provided in an embodiment of the present application;
[0017] Figure 5 A flowchart of another data processing method provided in an embodiment of the present application;
[0018] Figure 6 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0019] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0020] In order to clearly illustrate the embodiments of the present invention, the terms involved in the embodiments of the present application are first explained.
[0021] Among them, the stream processing engine refers to an engine that can process and analyze data streams in real time.
[0022] The data source end, which may also be referred to as the Source end, refers to a component in the stream processing engine that is responsible for reading data from an external data source end, that is, a Source component.
[0023] The data destination refers to the terminal where the data needs to be processed by the data processing task.
[0024] The external data source end refers to a data storage end outside the stream processing engine, and the data storage end is used to store data. For example, the external data source end may be a data source end including a TGene time series database.
[0025] Among them, TEngi ne time series database is a high-performance open source time series database. TEngi ne time series database supports reading data in batches to improve the efficiency of data reading.
[0026] The batch data file refers to a file including multiple batches of data, and the location identifier corresponding to each batch of data is different.
[0027] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application.
[0028] It should be noted that the execution subject of the data processing method in the embodiment of the present application is a data processing device, which can be implemented by software and / or hardware, and the data processing device can be a data source end (also referred to as a Source end) in a streaming processing engine, wherein the Source end can be an electronic device, or can be configured in an electronic device to implement a data processing function. In this embodiment, the data processing device is taken as the Source end for an exemplary description.
[0029] It should be noted that the electronic device in this embodiment can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.
[0030] like Figure 1 As shown, the data processing method includes the following steps:
[0031] Step 101, after the data processing task is restarted, obtain the target data consumption record corresponding to the data processing task.
[0032] The target data consumption record includes: the location identifier of the target batch data being processed before the data processing task is restarted and the traversal sequence number of the target data in the target batch data.
[0033] The target data is the last data successfully processed by the data processing task before it is restarted.
[0034] It should be noted that, in this embodiment, successful processing means that the target data is processed before the data processing task is restarted, and is finally successfully sent to the data destination corresponding to the data processing task.
[0035] The location identifier is used to uniquely identify the target batch data, and the target batch data can be obtained based on the location identifier.
[0036] In this embodiment, in order to accurately obtain the target data consumption record, correspondingly, a possible implementation method of step 101 is: after the data processing task is restarted, according to the identification information and current time information of the data destination corresponding to the data processing task, the target checkpoint status data of the data destination is obtained from the external storage device, wherein the target checkpoint status data refers to the checkpoint status data of the data destination before the current time information and closest to the current time information; from the target checkpoint status data, the target data consumption record of the data consumer corresponding to the data processing task is obtained.
[0037] The identification information refers to information that can be used to uniquely identify the destination of data, and this embodiment does not limit the identification information.
[0038] In this embodiment, after receiving the data, the data destination may store the received data. As an example, the data destination may store the received data in a streaming data lake platform.
[0039] Among them, the streaming data lake platform in this embodiment can be a Hud i streaming data lake platform, or other types of streaming data lake platforms, which is not specifically limited in this embodiment.
[0040] Step 102: reacquire target batch data corresponding to the location identifier from an external data source.
[0041] In some exemplary embodiments, in order to conveniently and quickly obtain the target batch data from the external data source, the target data consumption record may also include the subject to which the target batch data belongs. Correspondingly, a possible implementation method for re-obtaining the target batch data corresponding to the location identifier from the external data source is: obtaining the batch data file under the subject from the external data source; and re-obtaining the target batch data corresponding to the location identifier from the batch data file.
[0042] The external data source in this embodiment may be a data source including a time series database.
[0043] Among them, the time series database in this embodiment can be a TDeng i ne time series database, or can be other types of time series databases, which is not specifically limited in this embodiment.
[0044] As an example, the position identifier is used to identify the position of the target batch data in the batch data file, for example, the position identifier may be the offset information of the target batch data, wherein the offset information of the target batch data refers to the offset of the target batch data relative to the beginning of the batch data file.
[0045] Step 103, traverse the target batch data to obtain the remaining data after the traversal sequence number.
[0046] In this embodiment, the target batch data is traversed in the order from front to back or from back to front according to the position of each data in the target batch data. That is to say, in this embodiment, the target batch data can be traversed in the order from front to back or from back to front according to the position of each data in the target batch data in the target batch data to obtain the remaining data after the traversal sequence number.
[0047] It should be noted that in order to accurately obtain the data that has not been processed by the data processing task before restart from the target batch data, correspondingly, in some embodiments, the order of traversing the target batch data is the same after and before the data processing task is restarted.
[0048] As an example, after obtaining the target batch data, each data in the target batch data can be cached in sequence into a data queue according to the position of each data in the target batch data from front to back, or from back to front, and the remaining data after the traversal sequence number can be obtained from the data queue.
[0049] Step 104, according to the data processing task, process each data in the remaining data in turn, and send the processed data to the data destination.
[0050] In this embodiment, each data in the remaining data may be serialized in sequence according to the data processing task, and the serialized data may be sent to the data destination in sequence.
[0051] The data processing method provided by the implementation of the present application, after the data processing task is restarted, re-acquires the target batch data that was not processed last time by the data processing task from the external data source, and obtains the target data that was successfully processed for the target batch data before the data processing task is restarted, and obtains the remaining data after the target data from the target batch data, and processes each data in the remaining data according to the data processing task, and sends the processed data to the data destination corresponding to the data processing task. In this way, the unprocessed data in the target batch data that was not processed last time can continue to be processed, which will make the data received by the data destination complete, without data loss, and ensure the data consistency between the data destination and the external data source.
[0052] Based on the above embodiments, in order to clearly understand the present application, Figure 2 The data processing method of this embodiment is further described exemplarily.
[0053] Figure 2 A flowchart of another data processing method provided in an embodiment of the present application. It should be noted that in this embodiment, the target data consumption record includes the location identifier of the target batch data being processed before the data processing task is restarted, the subject to which the target batch data belongs, and the traversal sequence number of the target data in the target batch data for exemplary description.
[0054] like Figure 2 As shown, before obtaining the target data consumption record corresponding to the data processing task, the data processing method may further include:
[0055] Step 201, determine the target data consumption record according to the subject, location identifier and traversal sequence number.
[0056] In this embodiment, in order to accurately obtain the target data consumption record, a possible implementation method for determining the target data consumption record according to the subject, location identifier and traversal sequence number is: before the data processing task is restarted, an initial data consumption record including the subject and location identifier is constructed; when traversing to the target data, the target data is processed according to the data processing task, and the processed target data is sent to the data destination; the traversal sequence number of the target data is added to the initial data consumption record to obtain the target data consumption record.
[0057] It should be noted that, in addition to the subject of the target batch data, the location identifier of the target batch data and the traversal sequence number of the target data in the target batch data, the target data consumption record in this embodiment may also include other information. For example, the target data consumption record may also include: information about the consumer group to which the data consumer of the data processing task belongs.
[0058] Step 202, receiving checkpoint status data submitted by a target checkpoint for indicating the status of a data destination, wherein the target checkpoint is the first checkpoint created by the data destination after receiving the target data.
[0059] In this embodiment, the checkpoint coordinator in the streaming processing engine can create checkpoints in the data destination at preset time intervals, and obtain the current status information of the data destination through the checkpoints, and form the checkpoint status data of the checkpoint based on the current status information, and send the checkpoint status data to the data source end in the streaming computing engine, so as to facilitate the subsequent data destination to obtain the checkpoint closest to the current time after an exception occurs, and restore the status and position of the data destination based on the checkpoint status data of the checkpoint.
[0060] Step 203: Add the target data consumption record to the checkpoint status data to obtain the target checkpoint status data.
[0061] Step 204, saving the target checkpoint state data to an external storage device.
[0062] In this embodiment, the target data consumption record is added to the checkpoint state data to obtain the target checkpoint state data, and the target checkpoint state data is saved to the external storage device, so that after the data processing task is restarted, the target checkpoint state data can be obtained from the external storage device, and the target data consumption record can be obtained from the target checkpoint state data. Thus, the dependence on the internal storage in the data source end is reduced, and the storage capacity can be easily increased by using the external storage device without replacing or upgrading the data source end.
[0063] In order to clearly understand this application, Figure 3 , an exemplary description is given of the process of adding a target data consumption record to the checkpoint state data and saving the updated checkpoint state data to an external storage device after a data processing task is started.
[0064] Figure 3 A flowchart of another data processing method provided in an embodiment of the present application.
[0065] like Figure 3 As shown, the data processing method may also include:
[0066] Step 301, after a data processing task is started, the Source end in the streaming processing engine can initialize the data consumer corresponding to the data processing task, so that the initialized data consumer subscribes to the corresponding topic 1 in the external data source end according to the data processing task, and obtains a batch of data data1 from the batch data file of the topic 1 on the external data source end.
[0067] It should be noted that, in this embodiment, a streaming processing engine is taken as a Flink streaming processing engine for exemplary description.
[0068] Step 302, the Source end obtains the location identifier 1 of the batch of data data1 and the consumer group to which the data consumer belongs, and constructs a data consumption record according to the topic 1, the location identifier 1 and the consumer group.
[0069] The location identifier 1 in this embodiment is used to uniquely identify the batch of data data1, and the location identifier may be the offset information of the batch of data data1.
[0070] In this embodiment, the data consumption record may be saved in the memory of the Source end.
[0071] Step 303: The Source end puts the batch of data data1 into the data queue.
[0072] In this embodiment, the data in the batch data data1 are put into the data queue according to the order of the positions of the data in the batch data data1 from front to back.
[0073] In step 304, the Source end traverses the data in the data sequence one by one, processes the traversed current data according to the data processing task, and sends the processed data to the data destination end corresponding to the data processing task.
[0074] In this embodiment, the traversed current data can be serialized, and the serialized data can be sent to the downstream operator corresponding to the data processing task for processing, and the processed data can be sent to the data destination corresponding to the data processing task.
[0075] Step 305, after sending the processed data to the data destination, the Source end may add the traversal sequence number 1 of the current data to the data consumption record.
[0076] Among them, the data destination end in this embodiment can also be called a Si nk end.
[0077] Step 306: After receiving the data, the data destination writes the received data to the specified Hudi table in the Hudi streaming data lake platform.
[0078] In this embodiment, after the data destination ends the data to be written to the specified Hud i table in the Hud i streaming data lake platform, the data is sent to the Hud i worker workpoint of the Hud i streaming data lake platform for a write operation to write the data to the specified Hud i table in the Hud i streaming data lake platform.
[0079] For a specific description of how the Hud i worker writes the data into the specified Hud i table, please refer to the relevant description in the related technology, which will not be repeated here.
[0080] Step 307, after the checkpoint in the data destination is triggered, the checkpoint coordinator in the stream processing engine generates the checkpoint status data of the checkpoint based on the current status data of the data destination, and sends the checkpoint status data of the checkpoint to the Source.
[0081] Among them, a checkpoint in this embodiment is created at every preset time interval.
[0082] As an example, the stream processing engine may create a checkpoint in the data destination at a preset time interval, so as to obtain the status data of the data destination through the checkpoint.
[0083] Step 308: The Source receives the checkpoint status data and adds the data consumption record to the checkpoint status data to obtain the target checkpoint status data.
[0084] Step 309: The Source end saves the target checkpoint state data to an external storage device.
[0085] Step 310, after the Si nk end determines that all data in the batch of data data1 are written to the specified Hud i table, the Si nk end can send a commit operation to the Hud i streaming data lake platform to make the specified Hud i table visible to the outside world.
[0086] In this embodiment, for the Source end, in the process of consuming data from the external data source end, by recording the offset information of each batch of data, and recording the serial number index of the data that has been processed in the current batch of data during the serialization process, the recorded offsets and index are persisted in the checkpoint status data of the checkpoint checkpoint, so that after the data processing task is restarted, the offsets information can be obtained from the checkpoint status data, and the target batch data that was not processed last time can be re-read based on the offsets information, and the unprocessed data in the target batch data can be obtained based on index, and the unprocessed data can be continued to be processed to ensure that the data is not lost or duplicated, so that the data consistency between the data destination end and the external data source end is achieved.
[0087] Among them, it should be noted that in order to further ensure the data consistency between the data destination end and the external data source end, the data destination end in this embodiment can use a two-stage submission and recommit mechanism to prevent data loss between the data destination end and the Hudi streaming data lake platform, thereby ensuring data consistency between the data destination end and the Hudi streaming data lake platform.
[0088] Among them, it should be noted that regarding the process of two-stage submission and repeated submission of the data destination to the Hudi streaming data lake platform, please refer to the relevant description in the relevant technology, which will not be repeated here.
[0089] In order to clearly understand this application, Figure 4 , an exemplary description is given of the data processing process after a data processing task is restarted. It should be noted that in this embodiment, an exception occurs in the data processing task when processing a certain data in the batch data data1, and the restart of the data processing task is used as an example for an exemplary description.
[0090] Figure 4 A flowchart of another data processing method provided in an embodiment of the present application.
[0091] like Figure 4 As shown, the data processing method may also include:
[0092] Step 401, after the data processing task is restarted, the Source end obtains the target checkpoint status data of the data destination end from the external storage device according to the identification information and current time information of the data destination end corresponding to the data processing task, and obtains the data consumption record of the data consumer corresponding to the data processing task from the target checkpoint status data.
[0093] The target checkpoint status data refers to the checkpoint status data of the data destination before the current time information and closest to the current time information.
[0094] The data consumption record includes: the subject of the batch data data1 acquired by the data consumer, the location identifier 1 of the batch data data1, and the traversal sequence number of the target data in the batch data data1.
[0095] The target data is the data that is successfully sent to the data destination corresponding to the data processing task before the data processing task is restarted.
[0096] Among them, the topic is subscribed by the data consumer in the external data source according to the data processing task it undertakes.
[0097] Step 402, the Source end obtains location identifier 1 from the data consumption record, replaces the data reading position in the data consumer's data reading function with location identifier 1, and calls the data reading function to re-obtain the batch data data1 corresponding to location identifier 1 from the batch data file of the subject of the external data source end.
[0098] The data consumer in this embodiment corresponds to the data processing task.
[0099] The data reading function in this embodiment may be a seek() function.
[0100] Step 403: The Source end puts the batch of data data1 into the data queue.
[0101] In this embodiment, the data in the batch data data1 are put into the data queue according to the order of the positions of the data in the batch data data1 from front to back.
[0102] In step 404, the Source end obtains the traversal sequence number 1 from the data consumption record, and traverses the remaining data in the data queue after the traversal sequence number 1 in sequence, processes the traversed current data according to the data processing task, and sends the processed data to the data destination end corresponding to the data processing task.
[0103] Among them, for the specific description of processing the traversed current data according to the data processing task and sending the processed data to the data destination corresponding to the data processing task, please refer to the relevant description in other embodiments, which will not be repeated here.
[0104] In addition, regarding the processing process performed by the data destination on the received data after receiving the data, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.
[0105] In this embodiment, after the data processing task is restarted, the unprocessed data in the batch data data1 that was not processed last time is continued to be processed through the data processing task, which will make the data received by the data destination complete, without data loss and duplication, and ensure the data consistency between the data destination and the external data source.
[0106] Figure 5 Schematic diagram of the structure of a data processing device according to an embodiment of the present application.
[0107] like Figure 5 As shown, the data processing device 500 may include: a first acquisition module 501, a second acquisition module 502, a traversal module 503 and a processing module 504, wherein:
[0108] The first acquisition module 501 is used to acquire the target data consumption record corresponding to the data processing task after the data processing task is restarted, wherein the target data consumption record includes: the location identifier of the target batch data being processed before the data processing task is restarted and the traversal sequence number of the target data in the target batch data, wherein the target data is the last data successfully processed by the data processing task before the data processing task is restarted;
[0109] The second acquisition module 502 is used to reacquire the target batch data corresponding to the location identifier from the external data source;
[0110] A traversal module 503 is used to traverse the target batch data to obtain the remaining data after the traversal sequence number;
[0111] The processing module 504 is used to process each data in the remaining data in turn according to the data processing task, and send the processed data to the data destination corresponding to the data processing task.
[0112] In one embodiment of the present application, the first acquisition module 501 is specifically used to: after the data processing task is restarted, obtain the target checkpoint status data of the data destination end from the external storage device according to the identification information and current time information of the data destination end corresponding to the data processing task, wherein the target checkpoint status data refers to the checkpoint status data of the data destination end before the current time information and closest to the current time information; obtain the target data consumption record corresponding to the data processing task from the target checkpoint status data.
[0113] In one embodiment of the present application, the target data consumption record also includes: the topic to which the target batch data belongs, and the second acquisition module 502 is specifically used to: obtain the batch data file under the topic from the external data source; and re-acquire the target batch data corresponding to the location identifier from the batch data file.
[0114] In one embodiment of the present application, the device may further include:
[0115] A first determination module is used to determine the target data consumption record according to the subject, location identifier and traversal sequence number;
[0116] A receiving module, used to receive checkpoint status data submitted by a target checkpoint for indicating the status of a data destination, wherein the target checkpoint is the first checkpoint created by the data destination after receiving the target data;
[0117] A second determination module is used to add the target data consumption record to the checkpoint state data to obtain the target checkpoint state data;
[0118] The saving module is used to save the target checkpoint state data to an external storage device.
[0119] In one embodiment of the present application, the first determination module is specifically used to: construct an initial data consumption record including a subject and a location identifier; when traversing to the target data, process the target data according to the data processing task, and send the processed target data to the data destination; add the traversal sequence number of the target data to the initial data consumption record to obtain the target data consumption record.
[0120] In one embodiment of the present application, the processing module 504 is specifically used to: according to the data processing task, sequentially serialize each data in the remaining data, and sequentially send each serialized data to the data destination.
[0121] In one embodiment of the present application, the order of traversing the target batch data is from front to back, or from back to front, according to the position of each data in the target batch data.
[0122] It should be noted that the aforementioned explanation of the data processing device embodiment is also applicable to the data processing device of this embodiment and will not be repeated here.
[0123] The data processing device provided in the embodiment of the present application, after the data processing task is restarted, re-acquires the target batch data that was not processed last time by the data processing task from the external data source, and obtains the target data that was successfully processed for the target batch data before the data processing task is restarted, and obtains the remaining data after the target data from the target batch data, and processes each data in the remaining data according to the data processing task, and sends the processed data to the data destination corresponding to the data processing task. In this way, the unprocessed data in the target batch data that was not processed last time can continue to be processed, which will make the data received by the data destination complete, without data loss, and ensure the data consistency between the data destination and the external data source.
[0124] In order to implement the above embodiments, the present application also proposes an electronic device.
[0125] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0126] The electronic device includes:
[0127] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .
[0128] When the processor 602 executes the program, the data processing method provided in the above embodiment is implemented.
[0129] Furthermore, the electronic device further comprises:
[0130] The communication interface 603 is used for communication between the memory 601 and the processor 602 .
[0131] The memory 601 is used to store computer programs that can be executed on the processor 602 .
[0132] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.
[0133] The processor 602 is used to implement the data processing method of the above embodiment when executing a program.
[0134] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0135] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.
[0136] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0137] In order to implement the above embodiments, the embodiments of the present application also provide a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data processing method provided in the above embodiments is implemented.
[0138] In order to realize the above-mentioned embodiment, the embodiment of the present application also proposes a computer program product, and when the instruction processor in the computer program product executes, the data processing method provided in the above-mentioned embodiment is realized. In the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above-mentioned terms does not necessarily need to be directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradicting each other.
[0139] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0140] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0142] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0143] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.
[0144] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0145] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A data processing method, characterized in that: The method comprises: After the data processing task is restarted, a target data consumption record corresponding to the data processing task is obtained, wherein the target data consumption record includes: a location identifier of the target batch data being processed before the data processing task is restarted and a traversal sequence number of the target data in the target batch data, wherein the target data is the last data successfully processed by the data processing task before the data processing task is restarted; Reacquiring the target batch data corresponding to the location identifier from an external data source; Traversing the target batch of data to obtain remaining data after the traversal sequence number; According to the data processing task, each data in the remaining data is processed in turn, and the processed data is sent to the data destination corresponding to the data processing task.
2. The method according to claim 1, characterized in that After the data processing task is restarted, obtaining the target data consumption record corresponding to the data processing task includes: After the data processing task is restarted, according to the identification information and current time information of the data destination corresponding to the data processing task, the target checkpoint state data of the data destination is obtained from the external storage device, wherein the target checkpoint state data refers to the checkpoint state data of the data destination before the current time information and closest to the current time information; A target data consumption record corresponding to the data processing task is obtained from the target checkpoint status data.
3. The method according to claim 2, characterized in that The target data consumption record further includes: the subject to which the target batch data belongs, and the reacquiring the target batch data corresponding to the location identifier from the external data source includes: Acquire batch data files under the subject from the external data source; The target batch data corresponding to the location identifier is reacquired from the batch data file.
4. The method according to claim 3, characterized in that Before acquiring the target data consumption record corresponding to the data processing task, the method further includes: Determining the target data consumption record according to the subject, the location identifier and the traversal sequence number; Receiving checkpoint status data submitted by a target checkpoint for indicating the status of the data destination, wherein the target checkpoint is the first checkpoint created by the data destination after receiving the target data; Adding the target data consumption record to the checkpoint state data to obtain the target checkpoint state data; The target checkpoint state data is saved in the external storage device.
5. The method according to claim 4, characterized in that The determining the target data consumption record according to the subject, the location identifier and the traversal sequence number includes: Constructing an initial data consumption record including the subject and the location identifier; When the target data is traversed, the target data is processed according to the data processing task, and the processed target data is sent to the data destination; The traversal sequence number of the target data is added to the initial data consumption record to obtain the target data consumption record.
6. The method according to claim 1, characterized in that The step of processing each of the remaining data in sequence according to the data processing task and sending the processed data to the data destination corresponding to the data processing task includes: According to the data processing task, each data in the remaining data is serialized in turn, and each serialized data is sent to the data destination in turn.
7. A data processing device, characterized in that: The device comprises: A first acquisition module is used to acquire a target data consumption record corresponding to the data processing task after the data processing task is restarted, wherein the target data consumption record includes: a location identifier of the target batch data being processed before the data processing task is restarted and a traversal sequence number of the target data in the target batch data, wherein the target data is the last data successfully processed by the data processing task before the data processing task is restarted; A second acquisition module, used to reacquire the target batch data corresponding to the location identifier from an external data source; A traversal module, used for traversing the target batch data to obtain the remaining data after the traversal sequence number; The processing module is used to process each data in the remaining data in turn according to the data processing task, and send the processed data to the data destination corresponding to the data processing task.
8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.