Data processing method and apparatus, and electronic device

Abstract

The embodiment of the application provides a kind of data processing method, device and electronic equipment, the method includes the real-time acquisition target service corresponding to the data to be processed, wherein the data to be processed includes data generation time, according to data generation time, the data to be processed is divided into different time windows, and according to the division result, the window data corresponding to different time windows is obtained, wherein the window data of each time window includes a boundary time, each boundary time includes a start time and an end time, the earliest generation time included in the data to be processed allocated to the time window is start time, and the latest generation time included in the data to be processed allocated to the time window is end time, according to the window data corresponding to different time windows, the target service is realized.The application improves the real-time performance of data processing, improves the range of mass data applicable business scenarios, and thus guarantees the application experience of user.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus and electronic device. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually transforming into Fintech. Big data technology is no exception. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on big data technology. To meet the growing needs of various financial businesses, the application of big data processing frameworks is becoming increasingly widespread.

[0003] In existing technologies, massive amounts of data can be processed in batches using big data processing frameworks to achieve related business objectives. When processing massive amounts of data in batches, data to be processed corresponding to different business operations can be collected first. Once the number of days or items of data to be processed has been collected, the collected data can then be processed in batches.

[0004] However, this processing method has a long latency, which reduces the real-time performance of data processing. It can only be applied to businesses with low real-time requirements, and cannot be applied to businesses with high real-time requirements. This reduces the scope of business scenarios in which massive data processing can be applied, and thus affects the user's application experience. Summary of the Invention

[0005] This application provides a data processing method, apparatus, and electronic device to improve the real-time performance of data processing.

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] Real-time acquisition of pending data corresponding to the target business, wherein the pending data includes the data generation time;

[0008] The data to be processed is divided into different time windows according to the data generation time, and the window data corresponding to different time windows is obtained according to the division results. Each time window contains a boundary time, and each boundary time contains a start time and an end time. The start time is the earliest generation time of the data to be processed allocated to the time window, and the end time is the latest generation time of the data to be processed allocated to the time window.

[0009] The target service is implemented based on the window data corresponding to the different time windows.

[0010] Optionally, the step of dividing the data to be processed into different time windows according to the data generation time, and obtaining window data corresponding to different time windows according to the division results, includes:

[0011] When the data partitioning permission lock is obtained, historical window data is obtained. The historical window data includes a first end time. The historical window data is the data corresponding to the time window where at least one window is in the executed state. The first end time is the latest end time among the boundary times included in the historical window data.

[0012] The data to be processed is extracted and processed according to the first end time and the target time to obtain the initial target data to be processed, wherein the target time is determined according to the current time and the first preset delay duration;

[0013] Extract target data corresponding to the number of targets from the initial target data in the order of data generation time, wherein the number of targets is the product of a preset number threshold and a preset time window threshold;

[0014] Construct time windows corresponding to the number of time window thresholds, and divide the target data to be processed into time windows corresponding to the number of time window thresholds based on the data generation time of the target data to be processed and the number thresholds, and obtain window data corresponding to different time windows according to the division results.

[0015] Optionally, dividing the target data to be processed into time windows corresponding to the number of windows corresponding to the window threshold, based on the data generation time of the target data to be processed and the preset number threshold, includes:

[0016] For any target time window, if the end time of the target time window corresponds to multiple target data to be processed with the same data generation time, then the multiple target data to be processed with the same data generation time are allocated to the target time window.

[0017] Optionally, if each synchronization processing permission lock corresponds to at least two window data points, then implementing the target service based on the window data corresponding to the different time windows includes:

[0018] After obtaining the target synchronization processing permission lock, acquire at least two first target window data corresponding to the target synchronization processing permission lock;

[0019] Data is extracted from the data to be processed based on the second start time and the second end time contained in the at least two first target window data, wherein the second start time is the earliest start time contained in the at least two first target window data, and the second end time is the latest end time contained in the at least two first target window data.

[0020] The newly extracted data to be processed is executed synchronously, and the data of the at least two first target windows are updated according to the execution result of the newly extracted data to be processed.

[0021] Optional, also includes:

[0022] After obtaining the asynchronous processing permission lock, acquire the exception data to be processed that meets the preset conditions;

[0023] The abnormal pending data is executed asynchronously, and the window data corresponding to the abnormal pending data is updated according to the processing result of the abnormal pending data.

[0024] Optionally, obtaining abnormal pending data that meets preset conditions includes:

[0025] Obtain the second target window data corresponding to the time window of processing failure, wherein the time window of processing failure is the time window in which the number of synchronous processing is greater than the first preset number threshold and the number of asynchronous processing is less than the second preset number threshold within the first preset time period;

[0026] Abnormal pending data is extracted from the pending data table based on the third start time and the third end time contained in the second target window data, wherein the third start time is the earliest start time contained in the second target window data, and the third end time is the latest end time contained in the second target window data.

[0027] Optionally, if the data to be processed includes a data status, then obtaining abnormal data to be processed that meets preset conditions includes:

[0028] Abnormal pending data is obtained by acquiring pending data whose data generation time is within a second preset time period and whose data status is a processing failure status from the pending data.

[0029] Optionally, if the window data includes a window state, then obtaining the abnormal pending data that meets the preset conditions includes:

[0030] Data to be processed that whose data generation time meets the delay time period is obtained from the data to be processed, and abnormal data to be processed is obtained. The start time of the delay time period is determined by the current time and the second preset delay duration, and the end time of the delay time period is the earliest time of the time window corresponding to the window state of pending state, processing state and processing failure state.

[0031] Optionally, if the window data includes a window state, then obtaining the abnormal pending data that meets the preset conditions includes:

[0032] An abnormal time window is defined as the window state being processed, and the duration of the processing state exceeds a preset duration threshold.

[0033] The synchronous processing count and asynchronous processing count corresponding to the abnormal time window are updated to zero, wherein the synchronous processing count and the asynchronous processing count are stored in the abnormal window data;

[0034] Abnormal pending data is extracted from the pending data based on the fourth start time and the fourth end time contained in the abnormal window data, wherein the fourth start time is the earliest start time contained in the abnormal window data, and the fourth end time is the latest end time contained in the abnormal window data.

[0035] Secondly, embodiments of this application provide a data processing apparatus, including:

[0036] The acquisition module is used to acquire the data to be processed corresponding to the target business in real time, wherein the data to be processed includes the data generation time;

[0037] The processing module is used to divide the data to be processed into different time windows according to the data generation time, and to obtain the window data corresponding to different time windows according to the division results. Each time window's window data includes a boundary time, and each boundary time includes a start time and an end time. The start time is the earliest generation time included in the data to be processed allocated to the time window, and the end time is the latest generation time included in the data to be processed allocated to the time window.

[0038] The processing module is also used to implement the target service based on the window data corresponding to the different time windows.

[0039] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0040] The memory stores computer-executed instructions;

[0041] The processor executes computer execution instructions stored in the memory to implement the data processing method described in the first aspect and various possible designs of the first aspect.

[0042] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method described in the first aspect and various possible designs of the first aspect.

[0043] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect and various possible designs of the first aspect.

[0044] This application provides a data processing method, apparatus, and electronic device. By adopting the above scheme, data to be processed corresponding to the target business, including its data generation time, can be acquired in real time. Then, the data to be processed is divided into different time windows based on its data generation time, resulting in window data corresponding to each time window. Each time window's window data includes a boundary time, and each boundary time includes a start time and an end time. The start time is the earliest generation time among the data to be processed allocated to the time window, and the end time is the latest generation time among the data to be processed allocated to the time window. The target business is then implemented based on the window data corresponding to different time windows. By dividing the real-time acquired data to be processed into different time windows based on its data generation time, and then processing the data according to the time windows, the target business can be achieved. This allows for streaming processing of the real-time acquired data to be processed, eliminating the need to wait for the data to be processed to reach a certain number of days or quantities before batch processing. This improves the real-time performance of data processing and can be applied to businesses with high real-time requirements, expanding the scope of business scenarios applicable to massive amounts of data, thereby ensuring a better user experience. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A schematic diagram illustrating the application of existing big data processing frameworks;

[0047] Figure 2A schematic diagram of the application system architecture for the data processing method provided in the embodiments of this application;

[0048] Figure 3 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0049] Figure 4 A schematic diagram illustrating the principle of the data processing method provided in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0051] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can also include other sequential examples besides those illustrated or described. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0054] Traditional big data processing architectures can include streaming processing architectures (such as Spark Streaming and Flink), batch processing architectures (such as Hive), and integrated streaming and batch processing architectures (such as Flink). These architectures involve numerous components, are incompatible with commonly used relational databases (such as MySQL), and cannot effectively support both OLAP (Online Analytical Processing) and OLTP (Online Transaction Processing) tasks simultaneously, especially in fintech applications. Furthermore, relational databases like MySQL cannot support the processing of massive amounts of data.

[0055] Against this backdrop, new big data processing frameworks (such as TiDB) have emerged. Currently, these new frameworks can provide batch processing of massive amounts of data in fintech businesses, and also offer data query services after processing. However, when batch processing massive amounts of data, current big data processing frameworks lack a streaming data processing platform, making it impossible to use existing streaming processing technologies, i.e., near real-time data processing is not possible. Typically, it's necessary to first collect data to be processed from different business areas, and then batch process the collected data after the required number of days or data entries have been collected. For example... Figure 1 A schematic diagram illustrating the application of existing big data processing frameworks, such as... Figure 1 As shown, existing batch processing methods typically use a T+1 day approach. This means the big data processing framework waits until day T, after all the data to be processed on day T has been imported into the framework, before ceasing the import of new data. Then, on day T+1, all newly acquired data is processed in batches. However, this method has a long latency, reducing the real-time performance of data processing. It can only be applied to businesses with low real-time requirements, and cannot be used for businesses with high real-time requirements. This limits the scope of business scenarios applicable to massive data processing, thus impacting the user experience.

[0056] In addition, existing technologies can also target hourly tasks, i.e., process data imported from a previous hour, thereby improving real-time performance, but the data processing latency is still relatively high.

[0057] Based on the aforementioned technical issues, this application divides the real-time acquired data to be processed into different time windows according to the data generation time of the data to be processed, and then processes the data to be processed according to the time windows to achieve the target business. This method enables streaming processing of real-time acquired data to be processed, eliminating the need to wait for the data to be processed to reach the required number of days or data items before batch processing. This improves the real-time performance of data processing and can be applied to businesses with high real-time requirements. It also expands the scope of business scenarios applicable to massive data (i.e., it can be applied to both scenarios with low and high real-time requirements), thereby ensuring the user's application experience.

[0058] Figure 2 This is a schematic diagram of the architecture of the application system for the data processing method provided in the embodiments of this application, such as... Figure 2 As shown, this application system may include: a distributed database, a big data processing framework, and terminal devices. The distributed database can synchronize the data to be processed to the big data processing framework in real time, and the terminal devices can stream the data to be processed in the big data processing framework, thereby improving the real-time performance of the data processing.

[0059] The distributed database can utilize existing databases, and the data to be processed within these databases can correspond to different business operations, such as financial business data. The big data processing framework can be TiDB, a converged distributed open database product that supports both online transaction processing and online analytical processing. It features horizontal scaling, real-time HTAP (hybrid OLTP and OLAP processing), and combines the characteristics of both relational and non-relational databases. Real-time synchronization of the data to be processed from the distributed database to the big data processing framework can be achieved using TiDB's data synchronization tool, DM (Data Migrator). Terminal devices can include smartphones, personal computers, tablets, servers, or server clusters.

[0060] In addition, big data processing frameworks can be deployed on standalone devices or on terminal devices.

[0061] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0062] Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method in this embodiment can be executed by a terminal device. Figure 3As shown, the method in this embodiment may include:

[0063] S301: Real-time acquisition of pending data corresponding to the target business, where the pending data includes the data generation time.

[0064] In this embodiment, when implementing the target business, data to be processed related to the target business can be acquired first. To improve the real-time performance of business data processing, a streaming processing approach can be adopted, that is, acquiring and processing the data to be processed corresponding to the target business in real time. The data to be processed may contain basic information related to the target business, the data generation time, and a data status to identify the current state of the data to be processed. For example, the data status can be pending, processing, processing failed, or processing successful. The default data status is pending (e.g., represented by 0), and the status of the data to be processed can be updated subsequently according to the specific processing procedure.

[0065] Furthermore, there can be one or more data items to be processed. Optionally, when acquiring the data items corresponding to the target business in real time, they can be retrieved from the distributed database corresponding to the target business. That is, after each business system generates the data items corresponding to the target business, it can store the data items in the distributed database. Then, different distributed databases can synchronize the data items to be processed to the big data processing framework for storage. The terminal device can sequentially retrieve the data items to be processed from the big data processing framework according to the data generation time contained in the data items and execute the data items to achieve the target business.

[0066] S302: Divide the data to be processed into different time windows according to the data generation time, and obtain the window data corresponding to different time windows according to the division results. Each time window's window data contains a boundary time, and each boundary time contains a start time and an end time. The start time is the earliest generation time of the data to be processed allocated to the time window, and the end time is the latest generation time of the data to be processed allocated to the time window.

[0067] In this embodiment, after acquiring the data to be processed in real time, since the data to be processed corresponds to different distributed databases and the amount of data is large, directly processing the acquired data could easily lead to data omissions or errors in the data processing order. Therefore, the data to be processed can be divided into different time windows based on the data generation time, resulting in window data corresponding to different time windows. The amount of data to be processed corresponding to each time window can be customized according to the actual application scenario.

[0068] Furthermore, the step of dividing the data to be processed into different time windows based on the data generation time, and obtaining window data corresponding to different time windows based on the division results, may specifically include:

[0069] When the data partitioning permission lock is obtained, historical window data is obtained. The historical window data includes a first end time. The historical window data is the data corresponding to the time window where at least one window is in the executed state. The first end time is the latest end time among the boundary times included in the historical window data.

[0070] The data to be processed is extracted and processed according to the first end time and the target time to obtain the initial target data to be processed, wherein the target time is determined according to the current time and the first preset delay duration.

[0071] According to the data generation time, target data corresponding to the number of targets are extracted sequentially from the initial target data to be processed, wherein the number of targets is the product of a preset number threshold and a preset time window threshold.

[0072] Construct time windows corresponding to the number of time window thresholds, and divide the target data to be processed into time windows corresponding to the number of time window thresholds based on the data generation time of the target data to be processed and the number threshold, and obtain window data corresponding to different time windows according to the division results.

[0073] Specifically, when dividing the data to be processed into different time windows, it is necessary to first acquire a data division permission lock. Only after acquiring the data division permission lock can the process of dividing the data to be processed into different time windows be executed. Moreover, after one thread acquires the data division permission lock, other threads cannot acquire the data division permission lock, thus achieving the effect that each piece of data to be processed can only belong to one window, ensuring the correctness of data division.

[0074] Furthermore, historical window data consists of data corresponding to time windows with at least one window in the "executed" state. Each executed time window corresponds to a boundary time, and each boundary time can include a start time (Timestamp_From) and an end time (Timestamp_Till). The start time is the data generation time of the first data in the corresponding window, and the end time is the data generation time of the last data in the corresponding window. The first end time is the latest end time among the boundary times included in the historical window data, i.e., the data generation time of the last data. After acquiring the data partitioning permission lock, the first end time can be obtained and then set as the start time of a new time window. If it is the first window partitioning, the start time of the first time window can be set to 0. Then, based on the first end time and the subsequently determined target time, the data to be processed can be extracted and processed to obtain the initial target data to be processed. The target time can be determined by subtracting the current time from the first preset delay duration. The first preset delay duration can be customized according to the actual application scenario. Optionally, the first preset delay duration can be determined based on the number of initial target data to be processed in the current time period. For example, if the number of initial target data to be processed in the current time period is small, the first preset delay time can be set to a small value or to 0. If the number of initial target data to be processed in the current time period is large, the first preset delay time can be set to a large value, which corresponds to a shorter time period for the data to be processed. Therefore, the amount of data to be processed each time can be appropriately reduced, thereby reducing the pressure on the terminal device to process data.

[0075] Optionally, after obtaining the initial target data to be processed based on the first end time and the target time, all initial target data whose generation time meets the time interval between the first end time and the target time can be divided into different time windows. Alternatively, a portion of the initial target data to be processed can be selected from all the initial target data that meet the conditions, based on the actual processing capability of the terminal device. When selecting a portion of the initial target data to be processed, it can be obtained and processed sequentially according to the data generation time of the initial target data, avoiding data processing errors caused by disordered processing order.

[0076] Furthermore, when dividing the acquired initial target data to be processed into different time windows, to accommodate the actual computational load of the terminal device, a pre-set threshold for the number of data points and a threshold for the time window can be obtained first. Then, target data to be processed can be extracted sequentially from the initial target data based on the threshold for the number of data points and the threshold for the time window, and different time windows can be constructed according to the time window threshold (for example, if the time window threshold is 3, three time windows can be constructed). The extracted target data to be processed can then be divided into different time windows. The threshold for the number of data points can be the maximum length of the encoded blocks contained in each window (also known as chunk_size), and the threshold for the time window can be the maximum number of time windows that the terminal device can provide (also known as window_divide_max).

[0077] Optionally, after obtaining the time window threshold and the number threshold, the time window threshold and the number threshold can be multiplied to obtain the target number. Then, the target pending data corresponding to the target number can be obtained from the pending data. To ensure the normal implementation of the target business, when obtaining the target pending data, it can be based on the data generation time of the pending data, that is, the target pending data corresponding to the target number can be obtained from the initial target pending data in chronological order of data generation time. Furthermore, when the initial number of target pending data is small, i.e., when there are few concurrently executed businesses, the number of target pending data may be the same as the initial number of target pending data, meaning that all pending data is obtained.

[0078] Furthermore, after dividing the target data to be processed into time windows corresponding to the number of time window thresholds, window data corresponding to different time windows can be obtained based on the division results, i.e., new window data corresponding to each window can be obtained. The window data can include the window state (initially unprocessed, which can be updated to processing, executed, etc., based on actual operations), and the boundary times of the time windows (such as the start and end times of each time window). The target data to be processed in each time window is arranged in chronological order, i.e., the first target data to be processed is generated earliest, and the last target data to be processed is generated latest. The start time of each time window is the data generation time of the first target data to be processed assigned to that time window, and the end time of each time window is the data generation time of the last target data to be processed assigned to that time window.

[0079] For example, if the time window threshold is 3 and the number threshold is 2, then the initial target data to be processed can be filtered from the data to be processed based on the first end time and the target time. Then, 3 data points can be filtered from the initial target data to be processed. Two target data points are to be processed, and the data is divided into three time windows, with two target data points per time window. The start and end times of each time window can be determined based on the data generation time of the data to be processed within that window, thus defining the window size.

[0080] For example, Table 1 is an information table corresponding to the initial target data to be processed. Table 1 contains 6 initial target data to be processed. Each initial target data to be processed contains the data generation time and basic information related to the target business.

[0081] Table 1 Information table corresponding to the initial target data to be processed

[0082]

[0083] Table 2 is an information table corresponding to the target data to be processed. In Table 2, there are three time windows, and each time window contains two rows of target data to be processed.

[0084] Table 2 Information table corresponding to the target data to be processed

[0085]

[0086] Table 3 is a window data table corresponding to the target data to be processed. In Table 3, there are three time windows, each time window corresponding to a start time and an end time.

[0087] Table 3. Window data table corresponding to the target data to be processed

[0088]

[0089] In summary, by first filtering the data to be processed to obtain a preset number of target data to be processed, and then dividing the filtered target data into different time windows, the processing efficiency of the data to be processed is improved, and it also conforms to the actual computing load of the terminal device, avoiding terminal device failures due to large computing loads, thereby ensuring the normal operation of the business. Furthermore, the boundary times of the time windows are determined based on the generation time of the data to be processed allocated to that time window, increasing the flexibility of the boundary time determination method.

[0090] Furthermore, the step of dividing the target data to be processed into time windows corresponding to the number of windows corresponding to the window threshold, based on the data generation time of the target data to be processed and the preset number threshold, may specifically include:

[0091] For any target time window, if the end time of the target time window corresponds to multiple target data to be processed with the same data generation time, then the multiple target data to be processed with the same data generation time are allocated to the target time window.

[0092] Specifically, when dividing the target data to be processed into windows, the division is based on the data generation time of the target data. Since multiple data entries may exist at the same time, to ensure that target data entries with the same generation time (also known as the same timestamp) are not placed in different time windows (if they are placed in different time windows, data that should be processed according to a certain business-defined processing order may be processed in an incorrect order, leading to data that cannot be processed normally), if the end time of a time window corresponds to multiple data entries, all target data entries can be placed in that time window instead of being placed in the next time window, thus improving the accuracy of data processing.

[0093] S303: Implement the target business based on the window data corresponding to different time windows.

[0094] In this embodiment, after dividing the data to be processed into different time windows, the window data corresponding to different time windows can be obtained according to the division results, and then the target business can be realized according to the window data corresponding to different time windows.

[0095] Optionally, the window data may include boundary times, which may include a start time and an end time. The start time is the data generation time of the first data to be processed contained in the window corresponding to the window data, and the end time is the data generation time of the last data to be processed contained in the window corresponding to the window data. Then, data within the time period whose generation time satisfies the start time and end time can be extracted from the data to be processed, and the target business can be achieved based on the extracted data.

[0096] By adopting the above scheme, the pending data corresponding to the target business, including the data generation time, can be acquired in real time. Then, the pending data is divided into different time windows according to the data generation time, resulting in window data corresponding to different time windows. The target business is then implemented based on the window data corresponding to different time windows. This method of dividing the real-time acquired pending data into different time windows based on the data generation time, and then processing the pending data according to the time windows, can achieve streaming processing of the real-time acquired pending data. It eliminates the need to wait for the pending data to reach the collection days or number of items before batch processing, improving the real-time performance of data processing. This method can be applied to businesses with high real-time requirements, expands the scope of business scenarios applicable to massive data, and thus ensures the user's application experience.

[0097] based on Figure 2 In addition to the method described herein, this specification also provides some specific implementation schemes of the method, which will be described below.

[0098] In another embodiment, if each synchronization processing permission lock corresponds to at least two window data points, then implementing the target service based on the window data corresponding to the different time windows includes:

[0099] After obtaining the target synchronization processing permission lock, acquire at least two first target window data corresponding to the target synchronization processing permission lock.

[0100] Data is extracted from the data to be processed based on a second start time and a second end time contained in the at least two first target window data, wherein the second start time is the earliest start time contained in the at least two first target window data, and the second end time is the latest end time contained in the at least two first target window data.

[0101] The newly extracted data to be processed is executed synchronously, and the data of the at least two first target windows are updated according to the execution result of the newly extracted data to be processed.

[0102] In this embodiment, existing synchronous processing permission locks are coarse-grained, for example, allocating a permission lock to an entire window of data, thus ensuring that each task can only acquire one window for subsequent processing. However, streaming processing involves large amounts of data and requires near real-time performance, making conventional processing solutions impractical. Therefore, fine-grained locks can be used to increase concurrency. This means allocating a permission lock to a portion of the data in a window (such as a row of data), dividing a window of data into multiple lock granularities according to actual needs. Each fine-grained lock can also correspond to at least two time windows. After acquiring a synchronous processing permission lock (i.e., a fine-grained lock), the data required within multiple time windows can be read at once, thereby reducing the number of data reads.

[0103] Optionally, you can first obtain the maximum number of windows (which can be called window_query_max) and the maximum number of synchronization processing permission locks (which can be called window_mutex_num) that are pre-set for each synchronization processing permission lock. Then, starting from number 0, the lock number is incremented by 1 every time window corresponding to the maximum number of windows of the synchronization processing permission lock. The maximum lock number is the maximum number of synchronization processing permission locks minus 1. After reaching the maximum value, it returns to 0 and starts again.

[0104] Table 4 shows the window data after locking. Continuing with Table 3 as an example, a synchronization processing permission lock can be assigned to every two time windows, and a lock number can be assigned to each synchronization processing permission lock.

[0105] Table 4. Window Data Table After Locking

[0106]

[0107] By allocating at least two consecutive time windows to a synchronous processing permission lock, multiple window data can be obtained simultaneously after acquiring a synchronous processing permission lock. Then, by using the minimum and maximum times (i.e., minimum start time and maximum end time) of multiple time windows, as much data as possible can be retrieved in a single operation. This not only effectively ensures the continuity of data in time but also effectively reduces interaction with the database and improves data processing efficiency.

[0108] In addition, after allocating the synchronization processing permission lock to the time window, the corresponding data to be processed can be obtained according to the allocated synchronization processing permission lock. The obtained data to be processed and the corresponding window data are then encapsulated into a data block (also called a chunk), and the encapsulated data block is submitted to the data processing thread to achieve the relevant target business.

[0109] For example, if the current thread acquires synchronization processing permission lock 0 (i.e., the target synchronization processing permission lock is lock 0), it can obtain the window data corresponding to synchronization processing permission lock 0 from all time windows (the obtained window data corresponds to at least two time windows). Then, based on the second start time and second end time contained in the at least two window data, it extracts data from the data to be processed and implements the relevant target business based on the extracted data. Specifically, the data to be processed corresponding to each window is arranged in chronological order of data generation time, and each window data contains a boundary time. The boundary time can correspond to a start time and an end time. The start time is the data generation time of the first data to be processed contained in the window corresponding to the window data, and the end time is the data generation time of the last data to be processed contained in the window corresponding to the window data. The second start time is the earliest start time among the boundary times contained in the windows corresponding to the at least two window data, and the second end time is the latest end time among the boundary times contained in the windows corresponding to the at least two first target window data.

[0110] Furthermore, the data processing methods in this application can be divided into two types: synchronous processing mode (also known as MainRoad) and asynchronous processing mode (also known as SideTrack). To improve the processing efficiency of the data to be processed, synchronous processing mode can be used when implementing the target business based on window data corresponding to different time windows, while asynchronous processing mode can be used when handling various abnormal scenarios. For example, abnormal scenarios can include window processing failure and re-pull, business data processing failure and re-pull, processing of delayed data arrival, and window heartbeat loss and re-pull, etc. In addition, to avoid mutual interference between synchronous and asynchronous processing modes, the data corresponding to synchronous and asynchronous processing modes can be isolated by resources.

[0111] For example, in synchronous processing mode, the data processing thread can be a DataProcessPoolService. A preset lock counter can determine the acquireable lock number, then acquire the synchronous processing permission lock corresponding to that lock number. Based on the acquired synchronous processing permission lock, it obtains the processable time window data and the corresponding data to be processed, and encapsulates this into a data block. Simultaneously, the state of the window corresponding to the time window data can be set to "processing," and the synchronous processing permission lock can be released. The lock counter increments, and the encapsulated data block can then be submitted to the DataProcessPoolService for processing to achieve the target business. The data block can be considered an independent data processing unit, including window data, the data to be processed corresponding to the time window, and the data block type (e.g., synchronous processing, asynchronous processing, business failure re-pull, delayed arrival data re-pull, and window loss heartbeat re-pull, corresponding to various processing methods in MainRoad and SideTrack modes, respectively). Through data block encapsulation, not only can complete data (data to be processed and window data, etc.) be provided for subsequent processing, but different processing can also be performed according to different types (e.g., different processing logic for synchronous and asynchronous processing), improving the efficiency and security of data processing.

[0112] In another embodiment, the method may further include: after obtaining the asynchronous processing permission lock, acquiring abnormal pending data that meets preset conditions.

[0113] The abnormal pending data is executed asynchronously, and the window data corresponding to the abnormal pending data is updated according to the processing result of the abnormal pending data.

[0114] In this embodiment, when executing newly extracted data to be processed to achieve the target business, there may be situations such as data processing errors. In order to ensure the normal processing of other data and the normal operation of the business, an error handling method can be used to handle abnormal situations.

[0115] Furthermore, there can be various abnormal situations, and different handling methods can be used to deal with different abnormal situations.

[0116] Optionally, there may be cases where time window processing fails. In such cases, obtaining abnormal pending data that meets preset conditions may include:

[0117] Obtain the second target window data corresponding to the time window of processing failure, wherein the time window of processing failure is the time window in which the number of synchronous processing is greater than the first preset number threshold and the number of asynchronous processing is less than the second preset number threshold within the first preset time period.

[0118] Abnormal pending data is extracted from the pending data table based on the third start time and the third end time contained in the second target window data, wherein the third start time is the earliest start time contained in the second target window data, and the third end time is the latest end time contained in the second target window data.

[0119] Specifically, during synchronous processing of data to be processed, there may be instances where encapsulated data blocks fail to process. In such cases, the abnormal data blocks can be reprocessed (the maximum number of retries can be configured). When the maximum number of retries is reached, the state of the time window corresponding to that data block can be set to a processing failure state. Then, after acquiring a specific lock (e.g., a lock specifically for window processing failures), the second target window data corresponding to the time window in the processing failure state can be retrieved. The range of window data retrieved can be time-limited; for example, it can retrieve time windows within a first preset duration (any value between 3 and 5 days) where processing failures occurred. Furthermore, it is necessary to ensure that the synchronous processing count of the window has reached the maximum (i.e., the first preset threshold) and the asynchronous processing count is less than the maximum processing count (i.e., the second preset threshold). Then, the window state of the time window corresponding to the retrieved second target window data can be updated to a processing state, and the lock can be released. Next, based on the third start time and third end time contained in the retrieved second target window data, abnormal data to be processed is extracted from the data table to be processed and encapsulated into a window abnormal data block (also called a chunk, which can include the second target window data and the abnormal data to be processed corresponding to the second target window data). The encapsulated window exception data block is then passed to the processing task, and the task is submitted to the task processing thread (such as DataProcessPoolService) to wait for execution.

[0120] Optionally, there may be cases where some pending data within the time window fails to be processed. In such cases, the pending data includes data status. Therefore, obtaining the abnormal pending data that meets preset conditions may include:

[0121] Abnormal pending data is obtained by acquiring pending data whose data generation time is within a second preset time period and whose data status is a processing failure status from the pending data.

[0122] Specifically, it can also be done based on the return result of the business layer interface (such as StreamedProcessor.process):

[0123] If no result is returned, it means that all source data in the task window has been processed successfully. Call the preset interface (such as the batchUpdate interface in TiDB) to update the data status of all pending data to the processing success status (for example, the value of the corresponding field can be updated to 9).

[0124] If a result is returned, all data corresponding to the returned result is considered to have failed to process. A preset interface (such as the batchUpdate interface in TiDB) is called to update the data processing status to a processing failure status (for example, the value of the corresponding field can be incremented by 1 each time). At the same time, the status of the data that has not been returned within the time window is updated to a processing success status (that is, the value of the corresponding field is updated to 9).

[0125] Regardless of the returned result, the processing status of the time window is updated to a successful processing status (value S). Even if business data processing fails, the status of the time window can still be updated to a successful processing status. In this case, the processing status of the failed pending data is a non-successful status (9 for success, and a non-zero value less than 9 for failure). It can then be processed again through the SideTrack mode task (reprocessing failed business data).

[0126] When reprocessing failed exception data using the SideTrack mode, reprocessing can be performed on a subset of failed data within a time window. Specifically, a lock (instantaneous, designed for business data processing failures) can be acquired. After acquiring this lock, the failed exception data can be extracted. For example, each piece of data has a processing status: 0 represents a pending status, 1-8 represent the number of retries on failure, and 9 represents a success status (customizable). Under this premise, only data with a processing status containing any number from 1 to 8 needs to be filtered for failed processing. Furthermore, the query range can be limited by time; for example, data generated within a second preset time period can be retrieved. This data can then be encapsulated into a window exception data block (also called a chunk, which can include failed processing data). The encapsulated window exception data block is then passed to the processing task, and the task is submitted to a task processing thread (such as DataProcessPoolService) for execution.

[0127] Furthermore, after a task is submitted to a task processing thread (such as DataProcessPoolService) for execution, only the processing status of the pending data needs to be updated: if successful, the status of the pending data is set to 9; if it fails, the status of the pending data is incremented by 1 each time until it is successfully processed. If it still fails after reaching the maximum number of retries, the status can be modified after manual confirmation by operations personnel, and the program will automatically execute the aforementioned process.

[0128] Optionally, there may also be cases where there is delayed arrival of abnormal pending data. In this case, the window data includes the window state. Therefore, obtaining the abnormal pending data that meets the preset conditions may specifically include:

[0129] Data to be processed that whose data generation time meets the delay time period is obtained from the data to be processed, and abnormal data to be processed is obtained. The start time of the delay time period is determined by the current time and the second preset delay duration, and the end time of the delay time period is the earliest time of the time window corresponding to the window state of pending state, processing state and processing failure state.

[0130] Specifically, since this application supports a near real-time data processing method that supports streaming processing, it needs to support the processing of delayed arrival of abnormal pending data. Correspondingly, a specific lock can be acquired first (for example, a dedicated lock for delayed data arrival), and then the delayed abnormal pending data can be retrieved. When retrieving delayed abnormal pending data, the delay period can be determined first, and then the abnormal pending data that meets the delay period can be retrieved. Generally, the delay time of pending data will not exceed 3 days; therefore, the start time of the delay period can be set to 3 days prior to the current time. If the time window status is pending or processing, it means that the time window has not yet been processed or has not been fully processed. Therefore, the pending data within this time window cannot be considered delayed and does not need to be processed separately; it can be handled uniformly by the synchronous task. If the time window status is processing failure, the asynchronous task can also restart the time window for processing. Therefore, the start time of the smallest time window among the time windows with processing failure, processing, or pending status can be taken as the end time of data delay. Then, the acquired abnormal pending data can be encapsulated into a chunk (mainly the data that arrives late), and the encapsulated chunk can be passed to the processing task. The task can then be submitted to the task processing thread (such as DataProcessPoolService) to wait for execution, and the lock can be released after execution.

[0131] Furthermore, after the task processing thread completes its execution, it can update only the processing status of the pending data: if successful, set the status of the pending data to 9; if it fails, increment the status of the pending data by 1 each time until successful. If it still fails after reaching the maximum number of retries, the status can be modified after manual confirmation by operations personnel, and the program will automatically execute the aforementioned process.

[0132] Optionally, it may also include a window processing timeout situation, in which case the window data contains the window state. In this case, obtaining the abnormal pending data that meets the preset conditions may specifically include:

[0133] An abnormal time window is defined as one where the window is in a processing state and the duration of the processing state exceeds a preset duration threshold.

[0134] The synchronous processing count and asynchronous processing count corresponding to the abnormal time window are updated to zero, wherein the synchronous processing count and the asynchronous processing count are stored in the abnormal window data.

[0135] Abnormal pending data is extracted from the pending data based on the fourth start time and the fourth end time contained in the abnormal window data, wherein the fourth start time is the earliest start time contained in the abnormal window data, and the fourth end time is the latest end time contained in the abnormal window data.

[0136] Specifically, for any given time window, if the task is interrupted during processing, the window will remain in a processing state, requiring reprocessing of the lost heartbeat time window. Correspondingly, a specific lock can be acquired first (for example, a lock specifically for lost heartbeats). If the lock cannot be acquired, an exception message can be generated. Then, the abnormal time window for the lost heartbeat can be retrieved, i.e., a window whose state has been in a processing state for an extended period exceeding a preset duration threshold. For example, a window that started processing 10 minutes ago is still in a processing state; this time can be customized based on experience. Then, the synchronous and asynchronous processing counts for the abnormal time window can be updated to 0, and subsequent reprocessing can be run, while releasing the lock. Alternatively, abnormal pending data can be retrieved from the pending data based on the boundary times (i.e., the fourth start time and the fourth end time) in the abnormal window data corresponding to the abnormal time window, encapsulated into a chunk (which may include window data and pending data within the window), and passed to the processing task. The task is then submitted to the task processing thread (such as DataProcessPoolService) for execution, and the lock is released after execution.

[0137] In summary, this system supports automatic restart and retry of some abnormal pending data for asynchronous tasks in various abnormal scenarios. That is, while normal window data is executed in synchronous tasks, when some data execution fails, the abnormal data is marked as failed, and then the asynchronous task will automatically retry these abnormal data. This improves data processing efficiency, has good near real-time performance, and only reprocesses the failed data. Successful data does not need to be reprocessed, reducing the amount of data to be processed and further improving data processing efficiency.

[0138] Figure 4 This is a schematic diagram illustrating the principle of the data processing method provided in the embodiments of this application, such as... Figure 4 As shown, in this embodiment, the database can be a TDSQL database, which contains data to be processed. The data to be processed can be synchronized to TiDB via DM. The window splitting thread in the terminal device (exemplarily, it can be WindowDivisionThread) can first acquire the window splitting lock (i.e., the data partitioning permission lock), then read the first end time in the historical window splitting data, and perform window splitting processing on the data to be processed according to the first end time to obtain new window data, and then release the window splitting lock. The window data processing thread (exemplarily a WindowedProcessorThread) can first acquire the window extraction lock (also known as a synchronization processing permission lock), then determine the window data corresponding to the target time window to be processed, and release the window extraction lock. Then, based on the window data corresponding to the target time window, it determines the target data to be processed, and encapsulates the window data corresponding to the target time window and the target data to be processed into data blocks (also known as Chunks). The encapsulated data blocks are sent to the data processing thread pool (also known as DataProcessPoolService). The data processing thread pool DataProcessPoolService schedules the data processing task StreamedProcessorRunner. The data processing task StreamedProcessorRunner calls the corresponding business logic (the specific processing logic implemented by the business developers) to process the data to be processed within the time window, and performs related follow-up operations based on the processing results, such as retrying the task if it fails, updating the data processing status, etc.

[0139] In summary, the TiDB-based streaming architecture in this application can support near real-time processing of financial-grade data. Furthermore, business developers only need to implement the business processing-related interface StreamedProcessor, and all other functions can be implemented based on the streaming architecture layer: data source integration, task scheduling, reliability assurance, etc., which reduces the workload of business developers.

[0140] Based on the same idea, this specification also provides an apparatus corresponding to the above method. Figure 5 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application, such as... Figure 5 As shown, the apparatus provided in this embodiment may include:

[0141] The acquisition module 501 is used to acquire the data to be processed corresponding to the target business in real time, wherein the data to be processed includes the data generation time.

[0142] The processing module 502 is used to divide the data to be processed into different time windows according to the data generation time, and to obtain window data corresponding to different time windows according to the division results. Each time window's window data includes a boundary time, and each boundary time includes a start time and an end time. The start time is the earliest generation time included in the data to be processed allocated to the time window, and the end time is the latest generation time included in the data to be processed allocated to the time window.

[0143] In this embodiment, the processing module 502 is further configured to:

[0144] When the data partitioning permission lock is obtained, historical window data is obtained. The historical window data includes a first end time. The historical window data is the data corresponding to the time window where at least one window is in the executed state. The first end time is the latest end time among the boundary times included in the historical window data.

[0145] The data to be processed is extracted and processed according to the first end time and the target time to obtain the initial target data to be processed, wherein the target time is determined according to the current time and the first preset delay duration.

[0146] According to the data generation time, target data corresponding to the number of targets are extracted sequentially from the initial target data to be processed, wherein the number of targets is the product of a preset number threshold and a preset time window threshold.

[0147] Construct time windows corresponding to the number of time window thresholds, and divide the target data to be processed into time windows corresponding to the number of time window thresholds based on the data generation time of the target data to be processed and the number threshold, and obtain window data corresponding to different time windows according to the division results.

[0148] Furthermore, the processing module 502 is also used for:

[0149] For any target time window, if the end time of the target time window corresponds to multiple target data to be processed with the same data generation time, then the multiple target data to be processed with the same data generation time are allocated to the target time window.

[0150] The processing module 502 is also used to implement the target service based on the window data corresponding to the different time windows.

[0151] Furthermore, in another embodiment, if each synchronization processing permission lock corresponds to at least two window data points, then the processing module 502 is further configured to:

[0152] After obtaining the target synchronization processing permission lock, acquire at least two first target window data corresponding to the target synchronization processing permission lock.

[0153] Data is extracted from the data to be processed based on a second start time and a second end time contained in the at least two first target window data, wherein the second start time is the earliest start time contained in the at least two first target window data, and the second end time is the latest end time contained in the at least two first target window data.

[0154] The newly extracted data to be processed is executed synchronously, and the data of the at least two first target windows are updated according to the execution result of the newly extracted data to be processed.

[0155] Furthermore, in another embodiment, the processing module 502 is also used for:

[0156] After obtaining the asynchronous processing permission lock, acquire the exception data to be processed that meets the preset conditions.

[0157] The abnormal pending data is executed asynchronously, and the window data corresponding to the abnormal pending data is updated according to the processing result of the abnormal pending data.

[0158] In this embodiment, the processing module 502 is further configured to:

[0159] Obtain the second target window data corresponding to the time window of processing failure, wherein the time window of processing failure is the time window in which the number of synchronous processing is greater than the first preset number threshold and the number of asynchronous processing is less than the second preset number threshold within the first preset time period.

[0160] Abnormal pending data is extracted from the pending data table based on the third start time and the third end time contained in the second target window data, wherein the third start time is the earliest start time contained in the second target window data, and the third end time is the latest end time contained in the second target window data.

[0161] In this embodiment, the data to be processed includes a data state, and the processing module 502 is further configured to:

[0162] Abnormal pending data is obtained by acquiring pending data whose data generation time is within a second preset time period and whose data status is a processing failure status from the pending data.

[0163] In this embodiment, the window data includes the window state, and the processing module 502 is further configured to:

[0164] Data to be processed that whose data generation time meets the delay time period is obtained from the data to be processed, and abnormal data to be processed is obtained. The start time of the delay time period is determined by the current time and the second preset delay duration, and the end time of the delay time period is the earliest time of the time window corresponding to the window state of pending state, processing state and processing failure state.

[0165] In this embodiment, the window data includes the window state, and the processing module 502 is further configured to:

[0166] An abnormal time window is defined as one where the window is in a processing state and the duration of the processing state exceeds a preset duration threshold.

[0167] The synchronous processing count and asynchronous processing count corresponding to the abnormal time window are updated to zero, wherein the synchronous processing count and the asynchronous processing count are stored in the abnormal window data.

[0168] Abnormal pending data is extracted from the pending data based on the fourth start time and the fourth end time contained in the abnormal window data, wherein the fourth start time is the earliest start time contained in the abnormal window data, and the fourth end time is the latest end time contained in the abnormal window data.

[0169] The apparatus provided in this application embodiment can achieve the above-mentioned... Figure 2 The methods in the embodiments shown are similar in principle and technical effect, and will not be described again here.

[0170] Figure 6 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the device 600 provided in this embodiment includes a processor 601 and a memory communicatively connected to the processor. The processor 601 and the memory 602 are connected via a bus 603.

[0171] In a specific implementation, the processor 601 executes the computer execution instructions stored in the memory 602, causing the processor 601 to execute the method in the above method embodiment.

[0172] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0173] In the above Figure 6 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0174] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0175] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0176] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method described in the above method embodiments.

[0177] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method described above.

[0178] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0179] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0180] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: obtaining target business corresponding to to-be-processed data in real time, wherein the to-be-processed data contains data generation time; dividing the to-be-processed data into different time windows according to the data generation time, and obtaining window data corresponding to different time windows according to the division result, wherein the window data of each time window contains a boundary time, each boundary time contains a start time and an end time, the start time is the earliest generation time contained in the to-be-processed data allocated to the time window, and the end time is the latest generation time contained in the to-be-processed data allocated to the time window; implementing the target business according to the window data corresponding to the different time windows; wherein dividing the to-be-processed data into different time windows according to the data generation time, and obtaining window data corresponding to different time windows according to the division result, comprises: when the data division permission lock is obtained, obtaining historical window data, wherein the historical window data contains a first end time, the historical window data is data corresponding to at least one time window with an executed state, and the first end time is the latest end time in the boundary time contained in the historical window data; performing extraction processing on the to-be-processed data according to the first end time and a target time to obtain initial target to-be-processed data, wherein the target time is determined according to the current time and a first preset delay duration; extracting target to-be-processed data corresponding to a target number from the initial target to-be-processed data in sequence according to the data generation time, wherein the target number is the product of a preset number threshold and a preset time window threshold; constructing time windows corresponding to the number of the time window threshold, and dividing the target to-be-processed data into the time windows corresponding to the number of the time window threshold based on the data generation time of the target to-be-processed data and the number threshold, and obtaining window data corresponding to different time windows according to the division result.

2. The method of claim 1, wherein, the dividing of the target to-be-processed data into the time windows corresponding to the number of the window threshold based on the data generation time of the target to-be-processed data and the preset number threshold comprises: for any target time window, if the end time of the target time window corresponds to a plurality of target to-be-processed data with the same data generation time, the plurality of target to-be-processed data with the same data generation time are allocated to the target time window.

3. The method according to any of claims 1-2, characterized in that, if the window data corresponding to each synchronization processing permission lock is at least two, the implementation of the target business according to the window data corresponding to the different time windows comprises: after obtaining a target synchronization processing permission lock, obtaining at least two first target window data corresponding to the target synchronization processing permission lock; extracting data from the to-be-processed data according to a second start time and a second end time contained in the at least two first target window data, wherein the second start time is the earliest start time contained in the at least two first target window data, and the second end time is the latest end time contained in the at least two first target window data; synchronously executing the newly extracted to-be-processed data, and updating the at least two first target window data according to an execution result of the newly extracted to-be-processed data.

4. The method of claim 3, wherein, Further comprising: obtaining abnormal to-be-processed data satisfying a preset condition after obtaining an asynchronous processing permission lock; asynchronously executing the abnormal to-be-processed data, and updating window data corresponding to the abnormal to-be-processed data according to a processing result of the abnormal to-be-processed data.

5. The method of claim 4, wherein, The obtaining of the abnormal to-be-processed data satisfying the preset condition comprises: obtaining second target window data corresponding to a time window in which a synchronous processing number is greater than a first preset number threshold and an asynchronous processing number is less than a second preset number threshold within a first preset time length; extracting abnormal to-be-processed data from the to-be-processed data table according to a third start time and a third end time contained in the second target window data, wherein the third start time is the earliest start time contained in the second target window data, and the third end time is the latest end time contained in the second target window data.

6. The method of claim 4, wherein, The obtaining of the abnormal to-be-processed data satisfying the preset condition comprises: obtaining to-be-processed data in which a data generation time is within a second preset time length and a data state is a processing failure state from the to-be-processed data to obtain abnormal to-be-processed data.

7. The method of claim 4, wherein, The obtaining of the abnormal to-be-processed data satisfying the preset condition comprises: obtaining to-be-processed data in which a data generation time satisfies a delay time period from the to-be-processed data to obtain abnormal to-be-processed data, wherein a start time of the delay time period is determined by a current time and a second preset delay time length, and an end time of the delay time period is the earliest time of a time window in which a window state is a to-be-processed state, a processing state or a processing failure state.

8. The method of claim 4, wherein, The obtaining of the abnormal to-be-processed data satisfying the preset condition comprises: determining an abnormal time window in which the window state is the processing state and a duration in the processing state exceeds a preset time length threshold; updating a synchronous processing number and an asynchronous processing number corresponding to the abnormal time window to zero, wherein the synchronous processing number and the asynchronous processing number are stored in abnormal window data; extracting abnormal to-be-processed data from the to-be-processed data according to a fourth start time and a fourth end time contained in the abnormal window data, wherein the fourth start time is the earliest start time contained in the abnormal window data, and the fourth end time is the latest end time contained in the abnormal window data.

9. A data processing apparatus, characterized by, Further comprising: The acquisition module is configured to acquire, in real time, to-be-processed data corresponding to a target service, wherein the to-be-processed data comprises a data generation time; The processing module is configured to divide the to-be-processed data into different time windows according to the data generation time, and obtain window data corresponding to different time windows according to a division result, wherein the window data of each time window comprises a boundary time, each boundary time comprises a start time and an end time, the start time is the earliest generation time included in the to-be-processed data allocated to the time window, and the end time is the latest generation time included in the to-be-processed data allocated to the time window; The processing module is further configured to implement the target service according to the window data corresponding to the different time windows. The processing module is specifically configured to, when a data division permission lock is acquired, acquire historical window data, wherein the historical window data comprises a first end time, the historical window data is data corresponding to at least one time window in an executed state, and the first end time is the latest end time in the boundary time included in the historical window data; perform extraction processing on the to-be-processed data according to the first end time and a target time to obtain initial target to-be-processed data, wherein the target time is determined according to a current time and a first preset delay duration; extract target to-be-processed data corresponding to a target number from the initial target to-be-processed data in sequence according to the data generation time, wherein the target number is a product of a preset number threshold and a preset time window threshold; construct time windows corresponding to the number of the time window threshold, divide the target to-be-processed data into the time windows corresponding to the number of the time window threshold based on the data generation time of the target to-be-processed data and the number threshold, and obtain window data corresponding to different time windows according to a division result.

10. An electronic device, comprising: Comprise: A processor and a memory in communication connection with the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the data processing method in any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the data processing method in any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the data processing method in any one of claims 1 to 8.

Images