Data processing method, data processing device, storage medium and electronic equipment

Abstract

The embodiment of the invention relates to a data processing method and device, a storage medium and electronic equipment, and relates to the technical field of data processing. The method comprises the following steps: receiving second data sent by a first data end, and recording batch processing time corresponding to the second data; the second data is data formed by batch processing of the first data by the first data end, and the batch processing time corresponding to the second data is time for batch processing of the first data corresponding to the second data by the first data end; in response to the received exception information sent by the first data end, forward offset is carried out based on the latest batch processing time in the recorded batch processing time, and exception restart time is obtained; and sending the abnormal restart time to the first data end, so that the first data end performs batch processing on the first data after the abnormal restart time again. According to the data processing method and device, when the first data end is abnormal, data processing can be effectively and accurately restarted.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology. More specifically, the embodiments of the present disclosure relate to a data processing method, a data processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no description herein is admitted to be prior art by inclusion in this section.

[0003] At present, in the process of data processing, for example, when using Flink (stream computing framework) to perform real-time data stream aggregation processing, data processing failures may occur and data restart or job recovery may be required. In related technologies, technicians usually use Checkpoints to recover jobs. Checkpoints can periodically save snapshots of the current data processing status to persistent storage. When a data processing failure triggers a failover redeployment, the data end will read the most recent snapshot information and try to recover from the checkpoint. Summary of the invention

[0004] However, in data processing scenarios with large traffic and excessive intermediate data, related technologies have the following problems:

[0005] On the one hand, when the snapshot file is large, it is impossible to take into account both fault tolerance and real-time performance; on the other hand, when the state structure, logic, parallel state, etc. of the data end change, data recovery through Checkpoint may not be successful; on the other hand, when the snapshot file is large, the pressure and overhead of resource storage are large, and the difficulty of data processing and the cost of management and maintenance are also high.

[0006] Therefore, a data processing method is highly needed to improve the accuracy and convenience of data recovery.

[0007] In view of one or more of the above problems, embodiments of the present disclosure are intended to provide a data processing method, a data processing device, a computer-readable storage medium, and an electronic device.

[0008] According to a first aspect of the present disclosure, a data processing method is provided, including: receiving second data sent by a first data end, and recording a batch processing time corresponding to the second data; the second data is data formed by the first data end performing batch processing on the first data, and the batch processing time corresponding to the second data is the time when the first data end performs batch processing on the first data corresponding to the second data; in response to receiving abnormal information sent by the first data end, shifting forward based on the most recent batch processing time in the recorded batch processing time to obtain an abnormal restart time; sending the abnormal restart time to the first data end, so that the first data end re-batch processes the first data located after the abnormal restart time.

[0009] According to a second aspect of the present disclosure, a data processing method is provided, including: batch processing first data to form second data; sending the second data and a batch processing time corresponding to the second data to a second data terminal; in response to an exception occurring during the batch processing of the first data, sending exception information to the second data terminal; receiving an exception restart time fed back by the second data terminal in response to the exception information, and re-batch processing the first data after the exception restart time.

[0010] According to a third aspect of the present disclosure, a data processing device is provided, including: a data receiving module, used to receive second data sent by a first data end, and record a batch processing time corresponding to the second data; the second data is data formed by the first data end performing batch processing on the first data, and the batch processing time corresponding to the second data is the time when the first data end performs batch processing on the first data corresponding to the second data; a time determination module, used to obtain an abnormal restart time by shifting forward based on the most recent batch processing time in the recorded batch processing time in response to receiving abnormal information sent by the first data end; and a time sending module, used to send the abnormal restart time to the first data end, so that the first data end re-batch processes the first data located after the abnormal restart time.

[0011] According to a fourth aspect of the present disclosure, a data processing device is provided, including: a data batch processing module, used to batch process first data to form second data; a batch processing time sending module, used to send the second data and the batch processing time corresponding to the second data to a second data end; an exception information sending module, used to send exception information to the second data end in response to an exception occurring in the process of batch processing the first data; a restart time receiving module, used to receive the abnormal restart time fed back by the second data end in response to the abnormal information, and re-batch process the first data located after the abnormal restart time.

[0012] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method of the first and second aspects and possible implementation methods thereof are implemented.

[0013] According to the sixth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the data processing methods of the first and second aspects and possible implementation methods thereof by executing the executable instructions.

[0014] In the technical solution based on the first aspect of the present disclosure, on the one hand, a new data processing method is proposed, which can record and manage the batch processing time corresponding to the second data sent by the first data end, so as to shift forward according to the latest batch processing time in the recorded batch processing time, so as to determine the abnormal restart time. This method can make full use of the data processing information of the existing resources, calculate the effective and accurate abnormal restart time, and provide a reliable time node for the first data end to restart the data processing; on the other hand, since the abnormal restart time is obtained by shifting forward based on the latest batch processing time, the first data end can re-batch the first data from an accurate and effective time node based on the abnormal restart time, so as to avoid the situation that the batch processing of the first data is directly restarted from the interruption node, causing data omission or data confusion; on the other hand, compared with the existing method of realizing data restart based on the Checkpoint strategy, this exemplary embodiment consumes less computing resources and storage resources, can quickly and efficiently determine the abnormal restart time, and enables the first data end to restart the batch processing of the first data in a timely and accurate manner.

[0015] Based on the technical solution of the second aspect of the present disclosure, on the one hand, during the process of batch processing the first data, the first data end sends the formed second data and the batch processing time corresponding to the second data to the second data end, so that the second data end can record and manage the batch processing time, and determine the abnormal restart time by making full use of the data processing information of the existing resources; on the other hand, the first data end can accurately and effectively determine the node of data restart according to the abnormal restart time determined by the second data end, and re-batch process the first data after the abnormal restart time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment;

[0017] Figure 2 A flow chart showing a data processing method in this exemplary embodiment;

[0018] Figure 3A schematic diagram showing an architecture of data processing in this exemplary embodiment;

[0019] Figure 4 A schematic diagram showing another data processing architecture in this exemplary embodiment;

[0020] Figure 5 A flowchart showing another data processing method in this exemplary embodiment;

[0021] Figure 6 A schematic diagram showing the structure of a data processing device in this exemplary embodiment is shown;

[0022] Figure 7 A schematic diagram showing the structure of another data processing device in this exemplary embodiment;

[0023] Figure 8 A schematic structural diagram of an electronic device in this exemplary embodiment is shown.

[0024] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0025] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0026] The embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0027] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0029] The inventors have found that in the prior art, when data processing is performed, when a job failure causes an interruption, it is necessary to restart or recover the data. Specifically, in the related art, job recovery is usually performed through Checkpoint. The function of Checkpoint is to help the data end achieve fault tolerance and ensure that data processing can be effectively recovered when a failure occurs. Its working principle is to save a snapshot of the current data processing state to persistent storage at a regular interval. When a data processing failure triggers Failover redeployment, the data end will read the most recent snapshot information and try to recover from the checkpoint instead of completely restarting the job, thereby achieving fault tolerance of Flink jobs. However, in data processing scenarios with large traffic and too much intermediate data, the Checkpoint method is not very practical. On the one hand, when the snapshot file is large, it is impossible to take into account both fault tolerance and real-time performance; on the other hand, when the state structure, logic, parallel state, etc. of the data end change, data recovery through Checkpoint may not be successful; on the other hand, when the snapshot file is large, the pressure and overhead of resource storage are large, and the difficulty of data processing and the cost of management and maintenance are also high.

[0030] In view of the above, the present disclosure provides a data processing method, a program running device, a computer program product and an electronic device. On the one hand, a new data processing method is proposed, which can record and manage the batch processing time corresponding to the second data sent by the first data end, so as to shift forward according to the latest batch processing time in the recorded batch processing time, so as to determine the abnormal restart time. This method can make full use of the data processing information of the existing resources, calculate the effective and accurate abnormal restart time, and provide a reliable time node for the first data end to restart the data processing; on the other hand, since the abnormal restart time is obtained by shifting forward based on the latest batch processing time, the first data end can re-batch the first data from an accurate and effective time node based on the abnormal restart time, avoiding the situation that the batch processing of the first data is directly restarted from the interruption node, causing data omission or data confusion; on the other hand, compared with the existing method of realizing data restart based on the Checkpoint strategy, this exemplary embodiment consumes less computing resources and storage resources, can quickly and efficiently determine the abnormal restart time, and enables the first data end to restart the batch processing of the first data in a timely and accurate manner.

[0031] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0032] Application Scenario Overview

[0033] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0034] The embodiments of the present disclosure can be applied to scenarios related to data aggregation on the Flink side for data in a data stream. The application scenarios are described in detail below in conjunction with the system architecture.

[0035] Figure 1 A schematic diagram of a system architecture of a data processing method is shown. The system architecture 100 includes a first data terminal 110 and a second data terminal 120. The first data terminal 110 may be a Flink terminal, which is used to batch process the first data to generate the second data, such as performing data aggregation processing on the original data in the user interaction data stream such as playing music, clicking music, forwarding music, etc., to obtain the aggregation result; the second data terminal 120 may be a service terminal, which is used to receive the second data sent by the first data terminal and the batch processing time of the second data, and perform statistical analysis on the data based on the aggregation result, such as receiving the aggregation result of the user interaction data for statistical analysis to determine the user portrait, user preference or user configuration information, etc. The second data terminal 120 can maintain and manage the batch processing time of the second data, determine the abnormal restart time according to the batch processing time recorded therein, and send the abnormal restart time to the first data terminal 110 that needs to perform abnormal restart to perform data recovery. In addition, the second data terminal 120 can also receive abnormal information from the first data terminal 110, such as Flink identification or Failover number information, etc., to determine the abnormal restart time according to the abnormal information and batch processing time.

[0036] Exemplary Methods

[0037] An exemplary embodiment of the present disclosure provides a data processing method. Figure 2 As shown, the method may include steps S210 to S230. Figure 2 Provide detailed instructions for each step.

[0038] refer to Figure 2 In step S210, the second data sent by the first data end is received, and the batch processing time corresponding to the second data is recorded; the second data is data formed by the first data end performing batch processing on the first data, and the batch processing time corresponding to the second data is the time when the first data end performs batch processing on the first data corresponding to the second data.

[0039] Among them, the first data end can be used to perform data aggregation processing. For example, the first data end can be a Flink end. The Flink end can be used to process large-scale data streams and realize real-time data processing and analysis. Data processing may include aggregation analysis of data, etc. The data stream processed in the Flink end can be an infinite sequence, and each element can be regarded as a data record. The second data refers to the data formed by the first data end performing batch processing on the first data. Batch processing can refer to aggregation processing on data. The first data can be the original data in the data stream, and the second data can refer to the aggregation result data obtained by aggregation processing on the first data. The first data can be obtained from a variety of data sources, such as Kafka (an open source stream processing platform) or HDFS (Hadoop distributed file system), etc.

[0040] In some application scenarios, such as audio-visual applications, the first data may be data under a specific indicator, such as the browsing data or playback data of the target video or target music by the user within one hour. When the first data end processes the first data, there may be a corresponding batch processing time, and the second data is the aggregated data obtained by batch processing the first data. In this exemplary embodiment, the batch processing time corresponding to the second data may be the time when the first data end batch processes the first data corresponding to the second data. For example, the batch processing time corresponding to the second data may be the time for performing aggregate processing on the first data constituting the second data, that is, the aggregation time, or it may be the time for obtaining the aggregated result data, or it may be the batch processing time for processing the most recent first data when the first data end batch processes the second data, or it may be one of the batch processing times of the most recent first data, etc.

[0041] In step S220, in response to receiving the abnormal information sent by the first data end, the abnormal restart time is obtained by shifting forward based on the most recent batch processing time in the recorded batch processing time.

[0042] Among them, the exception information can be information about abnormal situations that cause data processing or data transmission to fail to be performed normally, such as job failures on the Flink side, system restarts, or other abnormalities that cause job failovers. The abnormal restart time refers to the start time of resuming the data processing process in abnormal situations such as failures on the data side, interruptions in data processing or transmission, for example, when resuming data aggregation, the time corresponding to which data is started is the abnormal restart time, that is, according to the abnormal restart time, it can be determined from which data the data batch processing process should be continued.

[0043] In this exemplary embodiment, a batch processing time management information may be maintained to record the batch processing time corresponding to the second data that has been sent previously. This exemplary embodiment may be applied to the server, and when the server receives the abnormal information sent by the first data terminal, it may perform forward offset processing based on the most recent batch processing time in the recorded batch processing time to obtain the abnormal restart time.

[0044] In step S230, the abnormal restart time is sent to the first data end, so that the first data end re-batch-processes the first data located after the abnormal restart time.

[0045] After determining the abnormal restart time, the abnormal restart time may be sent to the first data end, so that the first data end receives the abnormal restart time and re-performs the batch processing process starting from the first data corresponding to the abnormal restart time.

[0046] In an exemplary embodiment, the second data is data generated by the first data end performing a first aggregation process on the first data according to a first unit time.

[0047] Among them, batch processing can be the first aggregation processing, and the first unit time can be an aggregation window for performing the first aggregation processing on the first data. The first aggregation processing is performed on the first data according to the first unit time, that is, the first data in the aggregation window is aggregated. The first unit time can be processed according to actual needs, for example, it can be set to 2 minutes, 5 minutes, etc., and the present disclosure does not make specific limitations on this.

[0048] In an exemplary embodiment, the abnormal restart time includes a first abnormal restart time; the abnormal restart time is obtained by shifting forward based on the latest batch processing time in the recorded batch processing time, which may include:

[0049] Determine a first offset according to a first unit time;

[0050] The most recent batch processing time is shifted forward based on the first offset to obtain a first abnormal restart time.

[0051] The first abnormal restart time may be a theoretically preliminarily determined time point for starting to consume data, and the first offset refers to an offset calculation amount used for performing offset processing on the batch processing time.

[0052] In this exemplary embodiment, the first offset may be determined according to the first unit time, for example, the first unit time is used as the first offset, or calculation is performed based on the first unit time, such as a slight increase or decrease in the first unit time, to determine the first offset, etc. After determining the first offset, the most recent batch processing time may be shifted forward according to the first offset, for example, the first abnormal restart time may be determined by subtracting the first offset from the most recent batch processing time.

[0053] In an exemplary embodiment, determining the first offset according to the first unit time may include:

[0054] The first unit time is taken as the first offset.

[0055] That is, in this exemplary embodiment, the first unit time can be directly used as the first offset. For example, when the first unit time is set to 5 minutes, 5 minutes can be directly used as the first offset.

[0056] In this exemplary embodiment, the first abnormal restart time can be expressed as LogicConsumeTime, the latest batch time can be expressed as LatestAggregateTime, and the first unit time can be expressed as Period. Then, the first abnormal restart time can be determined by the following formula:

[0057] LogicConsumeTime=LatestAggregateTime–Period

[0058] In an exemplary embodiment, the abnormal restart time further includes a second abnormal restart time; the abnormal restart time is obtained by shifting forward based on the latest batch processing time in the recorded batch processing time, and may also include:

[0059] The first abnormal restart time is shifted forward based on a preset second offset to obtain a second abnormal restart time; the second abnormal restart time is used to be sent to the first data end.

[0060] The second abnormal restart time may be a physically determined time point for starting to consume data, and the second offset refers to the amount of calculation used to offset the first abnormal restart time. The second offset may be a custom time offset, and the value may be freely defined by the user based on the first data end, such as the amount of data processed by the Flink job and the size of the aggregation window. The second offset may be used to ensure that when consumption starts from the second abnormal restart time, the complete data starting from the first abnormal restart time can be consumed.

[0061] This exemplary embodiment can shift the first abnormal restart time forward according to the second offset, determine the second abnormal restart time, and send the second abnormal restart time to the first data end so that the first data end can perform data consumption according to the second abnormal restart time.

[0062] In this exemplary embodiment, the second abnormal restart time can be expressed as PhysicalConsumeTime, the preset second offset corresponds to customOffsetTime, and the first abnormal restart time is shifted forward based on the preset second offset to obtain the second abnormal restart time, which can be expressed as:

[0063] PhysicalConsumeTime=LogicConsumeTime–customOffsetTime

[0064] The server may return the first abnormal restart time and / or the second abnormal restart time to the first data end to ensure that the first data end can sense where the data should be consumed starting from this restart.

[0065] In another exemplary embodiment, after determining the first abnormal restart time and sending it to the first data end, the first data end can shift the first abnormal restart time forward based on a preset second offset to obtain a second abnormal restart time, and re-batch the first data located after the second abnormal restart time.

[0066] Figure 3 A schematic diagram of a data processing architecture in this exemplary embodiment is shown, which may specifically include the following steps:

[0067] Step S302: the first data end 310, such as the Flink end, may perform batch processing, such as aggregation processing, on the first data according to a first unit time (such as 5 minutes) to obtain an aggregation result of the first data, that is, second data 320;

[0068] Step S304, when an exception occurs during the process of batch processing of the first data by the first data end 310, the second data 320 is sent to the second data end 330, such as the service end;

[0069] Step S306, the first data terminal 310 saves the second data 320 to the memory 340;

[0070] Step S308, the first data terminal 310 sends the abnormal information 350 including the task identifier, the number of abnormalities, etc. to the second data terminal 330;

[0071] Step S310, the second data terminal 330 receives the second data 320 sent by the first data terminal, records the batch processing time corresponding to the second data, and maintains a time list 360 of the batch processing time;

[0072] Step S312, the second data terminal 330 marks and processes the invalid aggregated data generated during the last run according to the time information maintained by the time list 360;

[0073] Step S314, after receiving the abnormal information 350 sent by the first data end, the second data end 330 may shift forward based on the latest batch processing time 370 in the recorded batch processing time to obtain an abnormal restart time 380; the abnormal restart time 380 may include a first abnormal restart time 381 and a second abnormal restart time 382;

[0074] Step S316, the second data end 330 may return the calculated abnormal restart time 380 to the first data end 310;

[0075] Step S318, the first data terminal 310 completes data restart according to the abnormal restart time;

[0076] Step S320 , the second data in the memory can be visually queried in the second data terminal 330 .

[0077] It should be noted that the above steps and drawings are only exemplary. In actual applications, the specific execution steps are not limited to the above order. For example, step S320 can be executed at any stage, or step S306 and step S308 can be executed simultaneously or sequentially. Figure 3 The various modules are for exemplary purposes only. In fact, the memory and the time list may be configured in the second data terminal or independently configured, and the present disclosure does not make any specific limitation on this.

[0078] In an exemplary embodiment, the data processing method may further include:

[0079] A second aggregation process is performed on the second data according to a second unit time to form third data; the second unit time is greater than the first unit time.

[0080] In the case of large traffic, in order to improve the efficiency of data batch processing and avoid aggregation rollback during aggregation processing, the rollback time is too long, and too much historical data needs to be consumed, thereby causing the consumption tasks of data processing to pile up. This exemplary embodiment proposes that the second data can also be batch processed according to a second unit time, that is, a second aggregation processing, that is, a unit time with a larger granularity different from the first unit time is set, and the server-side downsampling strategy is used to perform the data processing process. The third data refers to the aggregation result obtained by performing the second aggregation processing on the second data.

[0081] The difference between the second aggregation processing and the first aggregation processing is that, on the one hand, the aggregation unit time used is different, and the second unit time is greater than the first unit time. For example, when performing the first aggregation processing, a 5-minute time window is used for data aggregation, and when performing the second aggregation time, a 1h (hour) time window is used for data aggregation, etc.; on the other hand, the aggregation objects are different. The first aggregation processing aggregates the first data, and the second aggregation processing aggregates the second data. For example, the first aggregation processing aggregates the data in the original data stream, and the second aggregation processing aggregates the aggregation result of the data aggregation processing based on the original data stream again.

[0082] The second unit time can be set according to actual needs, or customized according to the size of the first unit time. For example, the first unit time can be set to 2 minutes and the second unit time can be set to 30 minutes, the first unit time can be set to 5 minutes and the second unit time can be set to 1 hour, and so on.

[0083] For example, Flink performs the first aggregation processing in the first unit time, such as a 5-minute aggregation window, to aggregate the first data; and the server triggers downsampling execution at a certain time interval to perform aggregation in the second unit time, such as 1 hour. The core is that the aggregation result of the second unit time comes from the secondary aggregation of the aggregation result of the first unit time on the Flink side, rather than directly aggregating the original data.

[0084] In an exemplary embodiment, performing the second aggregation process on the second data according to the second unit time to form the third data may include:

[0085] In response to the most recent batch processing time and the second unit time satisfying a preset relationship, the second data that has not been subjected to the second aggregation processing is subjected to the second aggregation processing according to the second unit time to form third data.

[0086] In practical applications, there may be a lag in the generation time of the aggregation result, or a situation where it takes some time to read the aggregation result. For example, when counting the data for the one-hour period from 3 o'clock to 4 o'clock, and the first unit time is 5 minutes. Usually, at 4 o'clock, all the data from 3 o'clock to 4 o'clock may not be readable, and it can only be read at 4:05. Therefore, that is, it is not until 4:05 that the second aggregation process can be performed on the second data according to the second unit time. Therefore, in order to ensure that the second aggregation process can be accurately and effectively performed on the second data that has not undergone the second aggregation process according to the second unit time, this exemplary embodiment can configure the response condition for the second aggregation process. When the most recent batch processing time and the second unit time satisfy a preset relationship, the second aggregation process is performed on the second data that has not undergone the second aggregation process according to the second unit time to form the third data.

[0087] Among them, the preset relationship may include that the remainder value of the most recent batch processing time with respect to the second unit time exceeds a preset value. The preset value can be set according to actual needs. In this exemplary embodiment, the preset value can be k times the first unit time.

[0088] For example, in this exemplary embodiment, the remainder value of the most recent batch processing time with respect to the second unit time exceeds the preset value, and the preset value is k times the first unit time.

[0089] The server can reuse the logic of maintaining the aggregation time list for the aggregation of the unit time to determine whether downsampling can be performed. Specifically, the most recent batch processing time can be determined from the aggregation time list. Considering the possible aggregation rollback triggered by the Flink job, when LatestAggregateTime - (LatestAggregateTime - LatestAggregateTime % MaxPeriod) > k * MinPeriod, the second aggregation process is performed on the second data that has not undergone the second aggregation process according to the second unit time to form the third data. Here, LatestAggregateTime represents the most recent batch processing time, MaxPeriod represents the second unit time, that is, the aggregation window for server downsampling, MinPeriod represents the first unit time, and k is a coefficient, 2 < k < MaxPeriod. The purpose is to ensure that when the first aggregation process of the first unit time undergoes aggregation rollback, it will not affect the execution result of downsampling. Through the above method, the second aggregation process through the second unit time can be achieved.

[0090] Based on the above implementation, in this exemplary embodiment, when an exception occurs at the first data end, whether it is the first aggregation process of the first unit time or the second aggregation process of the second unit time, it can ensure that the aggregated result data will not be lost.

[0091] In an exemplary embodiment, the batch processing time corresponding to the second data may include:

[0092] The batch processing time corresponding to the second data is stored in a time list in chronological order.

[0093] This exemplary embodiment may manage and save the batch processing time corresponding to the second data through a time list.

[0094] The first aggregation processing of the first data end can be performed based on event time, processing time and intake time. This type of time-related aggregation can usually use the first unit time to perform the first aggregation processing, such as data aggregation based on a specified window size, and the final output result can contain time window information, and the output of the aggregation result is incremented by time size. These aggregation results can be delivered to the server through messages or other means. When the server consumes these aggregation result data, due to inconsistent consumption rates or implementation methods, the received time may be temporarily partially disordered, but the overall time trend is increasing. In this exemplary embodiment, the server can save the batch processing time corresponding to the second data in a time list in chronological order. It should be noted that the batch processing time maintained in the time list can be time data that increases by time size and is deduplicated. In addition, the most recent batch processing time can also be recorded as LatestAggregateTime.

[0095] In an exemplary embodiment, after obtaining the abnormal restart time, the data processing method may further include:

[0096] The second data whose batch processing time is later than the abnormal restart time is discarded.

[0097] In this exemplary embodiment, in order to improve the efficiency of data statistics, this exemplary embodiment can discard the second data whose batch processing time is later than the abnormal restart time. Specifically, the server can mark the previous second data and the data after the first abnormal restart time as invalid according to the job identification information reported in the last run. This part of data can adopt an asynchronous cleanup strategy or filter during query. The core is to mark this part of data as discarded and no longer need to participate in the subsequent real data statistics.

[0098] In an exemplary embodiment, the second data includes second data under different services; the abnormal information includes a service identifier;

[0099] In response to receiving the abnormal information sent by the first data terminal, obtaining the abnormal restart time based on the most recent batch processing time in the recorded batch processing time by shifting forward may include:

[0100] In response to receiving the abnormal information sent by the first data terminal, determining the target service where the abnormality occurs according to the service identifier in the abnormal information;

[0101] The abnormal restart time is obtained by shifting forward the most recent batch processing time among the recorded batch processing times of the target business.

[0102] In this exemplary embodiment, when an abnormal situation occurs at the first data end, for example, when the job triggers a failover due to a fault or other abnormality at the Flink end, the abnormal information can be sent to the server when the job is restarted, wherein the abnormal information can include a service identifier and can also include information such as the start time. The service identifier can include the service name and service abnormality times information, such as the job name and the number of times the current job fails, etc., which can be used to identify the Flink task. Based on the service identifier, the Flink task where the abnormality occurs can be determined. For example, Flink can process multiple tasks, and there is no intersection between the tasks of each Flink. For example, Flink1 can be used to process two tasks, namely, the number of plays and the number of visits, and Flink2 can be used to process three tasks, namely, the play time, the browsing time, and the browsing time. When Flink fails, it can be determined whether Flink1 or Flink2 fails based on the service identifier.

[0103] This exemplary embodiment can determine the target business where the exception occurs based on the business identifier in the exception information, and obtain the exception restart time based on the forward offset of the most recent batch processing time in the recorded batch processing time under the target business. For example, the most recent batch processing time in the recorded batch processing time under the target business is determined from the time list, and the most recent batch processing time is offset forward to determine the exception restart time.

[0104] Figure 4 A schematic diagram of another data processing architecture in this exemplary embodiment is shown, which may specifically include the following steps:

[0105] Step S402: the first data terminal 410 may perform a first aggregation process on the first data according to a first unit time to obtain second data 420;

[0106] Step S404, the first data terminal 410 sends the second data 420 to the second data terminal 430;

[0107] Step S406, adding the second data to the memory 440;

[0108] Step S408, the second data end 430 maintains a time list 450 of the batch processing time of the second data, and the second data end 430 shifts forward according to the most recent batch processing time among the batch processing times recorded in the time list 450 of the batch processing time, to obtain an abnormal restart time;

[0109] Step S410, the time list 450 can read the aggregate time from the working module;

[0110] Step S412, the working module 460 writes the second unit time into the memory 440;

[0111] Step S414, when the most recent batch processing time of the second data 420 and the second unit time satisfy a preset relationship, performing a second aggregation process on the second data that has not been subjected to the second aggregation process according to the second unit time to form third data;

[0112] In step S416 , the second data in the memory 440 may be displayed in a chart at the second data terminal 430 .

[0113] It should be noted that the above steps and figures are only exemplary. In actual applications, the specific execution steps are not limited to the above order. For example, step S416 can be executed at any stage. Figure 4 The various modules are for exemplary purposes only. In fact, the memory and the time list may be configured in the second data terminal or independently configured, and the present disclosure does not make any specific limitation on this.

[0114] The exemplary embodiment of the present disclosure also provides a data processing method. Figure 5 As shown, the method may include steps S510 to S530. Figure 5 Provide detailed instructions for each step.

[0115] Step S510: batch process the first data to form second data.

[0116] The first data refers to the data in the original data stream to be processed, and the second data refers to the data after batch processing. For example, the first data may be the original user behavior statistics data, and the second data refers to the aggregation result of the aggregation processing of the first data. The above method can be applied to the first data end, which may be the Flink end, for performing aggregation processing on the first data.

[0117] Step S520: Send the second data and the batch processing time corresponding to the second data to the second data terminal.

[0118] Among them, the batch processing time corresponding to the second data refers to the time for batch processing the first data to obtain the second data. For example, the time of the last processed first data in the first data used to obtain the aggregation result can be used as the batch processing time, or the time of the most recent data in the first data used to obtain the aggregation result can also be used as the batch processing time, or the time when the batch processing is actually completed can also be used as the batch processing time, and so on.

[0119] After batch processing based on the first data, the first data terminal may send the second data and the batch processing time corresponding to the second data to the second data terminal. The second data terminal may be a data terminal for data processing or statistics, such as a server for analyzing user behavior statistical data. The second data may be sent to the second data terminal according to a preset period. For example, if the preset period is set to 10 minutes, the second data may be sent to the second data terminal once every 10 minutes. After receiving the batch processing time corresponding to the second data, the first data terminal may maintain it in the form of a time list to calculate the abnormal restart time using the batch processing time stored therein.

[0120] Step S530, in response to an exception occurring during the batch processing of the first data, sending exception information to the second data terminal.

[0121] When the first data end is subjected to an exception during batch processing of the first data, the exception may include a job failure, a system restart, or other situations that cause the job to trigger a failover on the Flink end. At this time, the first data end may send an exception message to the second data end to store the exception message persistently. The exception message may include a business identifier, such as the current job identifier information, and / or a start time information. Among them, the business identifier may be a job identifier information, such as the job name, the number of failovers of the current job, which may be obtained from the context information of the Flink job operation. At the same time, when the aggregation result is output, the job identifier information may also be carried to facilitate subsequent processing. The start time information may be the start reporting time. In actual applications, there may be a time difference between the first data end and the second data end. For example, the time of the failure or restart for the first data end is 10:35, but the time of the failure or restart for the second data end is 10:39. The start reporting time refers to the timestamp information sent by the first data end to the second data end, which is based on the actual time of the second data end. In this exemplary embodiment, the second data end can determine the abnormal restart time based on the recorded batch processing time, or determine a more accurate abnormal restart time based on the recorded batch processing time combined with the start time information sent by the first data end, or the start time information can also be used as a restart notification information to notify the second data end that data restart is required, etc.

[0122] Step S540: receiving the abnormal restart time fed back by the second data end in response to the abnormal information, and re-batch-processing the first data located after the abnormal restart time.

[0123] Specifically, since each first data has corresponding timestamp information, after receiving the abnormal restart time, the first data end can search for a time node that matches or has the same time as the abnormal restart time in the timestamp of the first data, and use the first data corresponding to the found time node as the restarted data node to restart the batch processing process. After receiving the abnormal restart time sent by the second data end, the first data end can re-batch process the first data located after the abnormal restart time. For example, after receiving the abnormal restart time sent by the server, the Flink end can reset the data consumption site according to the abnormal restart time, and restart the data consumption and real-time aggregation of the first data, and output the aggregation result normally.

[0124] In an exemplary embodiment, the abnormal restart time includes a first abnormal restart time and a second abnormal restart time, and re-batch processing of the first data located after the abnormal restart time may include:

[0125] Determine a batch processing time corresponding to the first data after the second abnormal restart time;

[0126] If the batch processing time corresponding to the first data is earlier than the first abnormal restart time, then the first data is discarded;

[0127] If the batch processing time corresponding to the first data is later than the first abnormal restart time, the first data is batch processed again.

[0128] The first abnormal restart time indicates the starting time corresponding to the first data that the first data end actually starts to participate in the aggregation, which can be expressed as LogicConsumeTime. The second abnormal restart time indicates the actual time from which the first data end should start consuming data after this startup, which can be expressed as PhysicalConsumeTime.

[0129] In this exemplary embodiment, if the batch processing time corresponding to the first data is earlier than the first abnormal restart time, it is discarded; if the batch processing time corresponding to the first data is later than the first abnormal restart time, the first data is re-batch processed, for example, the Flink end resets the consumption point to PhysicalConsumeTime and starts consuming data and starting the job according to the abnormal restart time information returned by the server. When the timestamp of the first data is less than or equal to LogicConsumeTime, the consumed first data is discarded because this part of the data may have participated in the calculation in the meantime; when the timestamp of the consumed message is greater than LogicConsumeTime, this part of the data will flow to the downstream operator to participate in the real real-time aggregation window calculation.

[0130] Exemplary Devices

[0131] The exemplary embodiment of the present disclosure also provides a data processing device. Figure 6 As shown, the data processing device 600 may include the following program modules: a data receiving module 610, used to receive the second data sent by the first data end, and record the batch processing time corresponding to the second data; the second data is data formed by the first data end performing batch processing on the first data, and the batch processing time corresponding to the second data is the time when the first data end performs batch processing on the first data corresponding to the second data; a time determination module 620, used to respond to the abnormal information sent by the first data end, and shift forward based on the most recent batch processing time in the recorded batch processing time to obtain the abnormal restart time; a time sending module 630, used to send the abnormal restart time to the first data end, so that the first data end re-batch processes the first data located after the abnormal restart time.

[0132] In one implementation, the second data is data generated by the first data end performing a first aggregation process on the first data according to a first unit time.

[0133] In one embodiment, the abnormal restart time includes a first abnormal restart time; the time determination module includes: a first offset determination unit, used to determine a first offset according to a first unit time; a first time determination unit, used to shift the most recent batch processing time forward based on the first offset to obtain a first abnormal restart time.

[0134] In one implementation, the time determination module is configured to use a first unit time as a first offset.

[0135] In one embodiment, the abnormal restart time also includes a second abnormal restart time; the time determination module also includes: a second time determination unit, used to shift the first abnormal restart time forward based on a preset second offset to obtain a second abnormal restart time; the second abnormal restart time is used to send to the first data end.

[0136] In one embodiment, the data processing device further includes: a third data forming module, configured to perform a second aggregation process on the second data according to a second unit time to form third data; the second unit time is greater than the first unit time.

[0137] In one embodiment, the third data forming module includes: a second aggregation processing unit, which is used to perform second aggregation processing on second data that has not been subjected to second aggregation processing according to the second unit time in response to the most recent batch processing time and the second unit time satisfying a preset relationship, to form third data.

[0138] In one implementation, the data receiving module includes: a time saving unit, configured to save the batch processing time corresponding to the second data in a time list in chronological order.

[0139] In one implementation, after obtaining the abnormal restart time, the data processing device further includes:

[0140] The data discarding module is used to discard the second data whose batch processing time is later than the abnormal restart time.

[0141] In one embodiment, the second data includes second data under different services; the exception information includes a service identifier; the time determination module includes: an exception information receiving unit, used to respond to the exception information sent by the first data end, and determine the target service where the exception occurs according to the service identifier in the exception information; a time offset unit, used to offset forward based on the most recent batch processing time in the recorded batch processing time under the target service to obtain the exception restart time.

[0142] The exemplary embodiment of the present disclosure also provides a data processing device. Figure 7 As shown, a data batch processing module 710 is used to batch process the first data to form second data; a batch processing time sending module 720 is used to send the second data and the batch processing time corresponding to the second data to the second data terminal; an exception information sending module 730 is used to send exception information to the second data terminal in response to an exception in the process of batch processing the first data; a restart time receiving module 740 is used to receive the abnormal restart time fed back by the second data terminal in response to the abnormal information, and re-batch the first data after the abnormal restart time.

[0143] In one embodiment, the abnormal restart time includes a first abnormal restart time and a second abnormal restart time, and the restart time receiving module includes: a batch processing time determination unit, used to determine the batch processing time corresponding to the first data after the second abnormal restart time; a first processing unit, used to discard the batch processing time corresponding to the first data if it is earlier than the first abnormal restart time; and a second processing unit, used to re-batch the first data if the batch processing time corresponding to the first data is later than the first abnormal restart time.

[0144] In addition, other specific details of the embodiments of the present disclosure have been described in detail in the embodiments of the above method and will not be repeated here.

[0145] Exemplary Storage Media

[0146] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method of the present disclosure when the computer program is executed by a processor. The above-mentioned method can be implemented by a program product, such as a portable compact disk read-only memory (CD-ROM) and including program code, and can be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0147] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0148] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0149] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RE, etc., or any suitable combination of the foregoing.

[0150] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0151] Exemplary Electronic Devices

[0152] The exemplary embodiment of the present disclosure also provides an electronic device. The electronic device includes a processor and a memory, and the memory is used to store executable instructions of the processor. The processor is configured to execute the above method of the present disclosure by executing the executable instructions.

[0153] refer to Figure 8 An electronic device according to an exemplary embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0154] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0155] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 The method steps shown, etc.

[0156] The storage unit 820 may include a volatile storage unit, such as a random access storage unit (RAM) 821 and / or a cache storage unit 822 , and may further include a read-only storage unit (ROM) 823 .

[0157] The storage unit 820 may also include a program / utility 824 having a set (at least one) of program modules 825, such program modules 825 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0158] The bus 830 may include a data bus, an address bus, and a control bus.

[0159] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and such communication may be performed via an input / output (I / O) interface 840. The electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 850. As shown, the network adapter 850 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0160] It should be noted that, although several modules or submodules of the device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0161] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0162] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: receiving second data sent by the first data end, and recording a batch processing time corresponding to the second data; the second data is data formed by the first data end performing batch processing on the first data, and the batch processing time corresponding to the second data is the time when the first data end performs batch processing on the first data corresponding to the second data; In response to receiving the abnormal information sent by the first data end, shifting forward based on the latest batch processing time in the recorded batch processing time to obtain the abnormal restart time; The abnormal restart time is sent to the first data end, so that the first data end re-batch-processes the first data located after the abnormal restart time.

2. The method according to claim 1, characterized in that The second data is data generated by the first data end performing a first aggregation process on the first data according to a first unit time.

3. The method according to claim 2, characterized in that The abnormal restart time includes a first abnormal restart time; the abnormal restart time is obtained by shifting forward based on the latest batch processing time in the recorded batch processing time, including: Determine a first offset according to the first unit time; The most recent batch processing time is shifted forward based on the first offset to obtain the first abnormal restart time.

4. The method according to claim 3, characterized in that The determining the first offset according to the first unit time includes: The first unit time is used as the first offset.

5. The method according to claim 3, characterized in that: The abnormal restart time also includes a second abnormal restart time; the abnormal restart time is obtained by shifting forward based on the latest batch processing time in the recorded batch processing time, and further includes: The first abnormal restart time is shifted forward based on a preset second offset to obtain the second abnormal restart time; the second abnormal restart time is used to be sent to the first data end.

6. A data processing method, characterized in that: The method comprises: batch processing the first data to form second data; Sending the second data and a batch processing time corresponding to the second data to a second data terminal; In response to an exception occurring during batch processing of the first data, sending exception information to the second data terminal; The abnormal restart time fed back by the second data end in response to the abnormal information is received, and the first data located after the abnormal restart time is re-batch-processed.

7. A data processing device, characterized in that: include: A data receiving module, used for receiving second data sent by the first data terminal, and recording a batch processing time corresponding to the second data; The second data is data generated by batch processing the first data by the first data end, and the batch processing time corresponding to the second data is the time when the first data end batch processes the first data corresponding to the second data; A time determination module, configured to obtain an abnormal restart time by shifting forward the most recent batch processing time in the recorded batch processing time in response to receiving the abnormal information sent by the first data terminal; The time sending module is used to send the abnormal restart time to the first data end, so that the first data end re-batch processes the first data located after the abnormal restart time.

8. A data processing device, characterized in that: The device comprises: A data batch processing module, used for batch processing the first data to form second data; A batch processing time sending module, used for sending the second data and the batch processing time corresponding to the second data to the second data terminal; an exception information sending module, configured to send exception information to the second data terminal in response to an exception occurring during the batch processing of the first data; The restart time receiving module is used to receive the abnormal restart time fed back by the second data end in response to the abnormal information, and re-batch process the first data located after the abnormal restart time.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 6 by executing the executable instructions.

Images