Batch processing task triggering method, system and computer readable storage medium

Abstract

The present application discloses a triggering method, system and computer-readable storage medium for a batch processing task. The data processing system is provided with a data processing architecture, which includes: a real-time processing layer and a batch processing layer. The real-time processing layer is used to receive the first data to be processed in real time, copy the first data to be processed to obtain the second data to be processed; determine whether all the second data to be processed corresponding to the current batch processing task have been received, and if all the second data to be processed corresponding to the current batch processing task have been received, an event notification message is generated; the batch processing layer is connected to the real-time processing layer, and is used to process all the received second data to be processed after receiving the event notification message to execute the current batch processing task. In the above manner, the present application can shorten the delay time of the batch processing task.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a batch processing task triggering method, system and computer-readable storage medium. Background Art

[0002] At present, big data technology has become more and more mature, and the frameworks of real-time computing and offline computing are also becoming mature. Real-time computing can get results in the first time, but due to many factors such as disorder, repeated transmission or omission of real-time data, the results of real-time computing may be inaccurate; offline computing is just the opposite, with very low timeliness, and some even delay calculations for 1 day, 1 week or one month. Some problems such as disorder or repeated transmission of data can be handled within this buffer time with high processing accuracy. Therefore, stream-batch integration has become the development direction, but the current stream-batch integration has the problems of isolation between real-time computing and offline computing and long delays in offline computing. Summary of the invention

[0003] The present application provides a batch processing task triggering method, system and computer-readable storage medium, which can shorten the delay time of the batch processing task.

[0004] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide a data processing system, which is provided with a data processing architecture, and the data processing architecture includes: a real-time processing layer and a batch processing layer, the real-time processing layer is used to receive the first data to be processed in real time, and copy the first data to be processed to obtain the second data to be processed; determine whether all the second data to be processed corresponding to the current batch processing task have been received, if all the second data to be processed corresponding to the current batch processing task have been received, then generate an event notification message; the batch processing layer is connected to the real-time processing layer, and is used to process all the received second data to be processed after receiving the event notification message, so as to execute the current batch processing task.

[0005] To solve the above technical problems, another technical solution adopted in the present application is: to provide a triggering method for a batch processing task, the method is applied to a data processing system, the data processing system is arranged with a data processing architecture, the data processing architecture includes a real-time processing layer and a batch processing layer, the method includes: using the real-time processing layer to receive the first data to be processed in real time, copying the first data to be processed to obtain the second data to be processed; determining whether all the second data to be processed corresponding to the current batch processing task have been received, if all the second data to be processed corresponding to the current batch processing task have been received, then generating an event notification message; using the batch processing layer to process all the received second data to be processed after receiving the event notification message to execute the current batch processing task.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the triggering method of the batch processing task in the above technical solution.

[0007] Through the above scheme, the beneficial effect of the present application is: the real-time processing layer is responsible for receiving the first data to be processed in real time, copying the first data to be processed to obtain the second data to be processed, and the real-time processing layer can determine whether the second data to be processed required by the batch layer to execute the current batch processing task is fully acquired. If all the second data to be processed corresponding to the current batch processing task are acquired, an event notification message is generated and sent to the batch layer, so that the batch layer executes the current batch processing task according to all the acquired second data to be processed. By establishing a connection between the real-time processing layer and the batch layer, the real-time processing layer has the ability to trigger the batch layer to execute the current batch processing task, rather than using a triggering method of timing to trigger the current batch processing task, breaking the isolation between the real-time processing layer and the batch layer, so that the real-time processing layer and the batch layer can be combined, which can adapt to more scenarios, shorten the delay of offline computing of the traditional Lambda architecture, and make the execution of offline batch processing tasks more real-time; and the first data to be processed and the second data to be processed are synchronized, and there will be no data inconsistency problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a structural diagram of the Lambda architecture provided by this application;

[0010] Figure 2 It is a schematic diagram of the structure of the Kappa architecture provided by this application;

[0011] Figure 3 It is a structural schematic diagram of an embodiment of a data processing system provided by the present application;

[0012] Figure 4 is a structural schematic diagram of another embodiment of the data processing system provided by the present application;

[0013] Figure 5 It is a flowchart of an embodiment of a triggering method for batch processing tasks provided by the present application;

[0014] Figure 6 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The current mainstream stream-batch integration solutions are Lambda architecture and Kappa architecture. Lambda architecture is Figure 1 As shown in the figure, the real-time branch directly consumes data to produce real-time, inaccurate results, while the offline branch consumes data and persists it to the database, waiting for the scheduled offline task to trigger to produce more accurate results, and then corrects the real-time results based on the results. However, this scheduled triggering mode has some disadvantages:

[0017] 1) The same business requirement requires the development of two sets of identical codes, which is not only difficult to develop, but also difficult to maintain.

[0018] 2) Scheduled tasks cannot sense whether the data required by the user is complete, resulting in inaccurate data.

[0019] 3) The data may be ready very early, but it will take a long time to get the data until the next offline scheduled task, resulting in a high delay.

[0020] Kappa architecture Figure 2 As shown in the figure, it strongly relies on the persistence capability of the message middleware and realizes offline computing capability through data playback. However, this model has some disadvantages:

[0021] 1) Strong reliance on message middleware, the amount of data cached by the message middleware and the backtracking data have performance bottlenecks.

[0022] 2) Kappa abandons the offline data processing module and also abandons the more stable and reliable characteristics of offline calculations, resulting in inaccurate calculation results.

[0023] 3) When processing real-time data and associating a large number of different real-time streams, it is very dependent on the capabilities of the real-time computing system, and it is very likely that data will be lost due to the order of the data streams.

[0024] In order to break the isolation between real-time computing and offline computing, let real-time computing trigger offline computing, combine real-time and offline computing, adapt to more scenarios, and solve more problems, this application adopts an event notification mechanism. Through the active event notification mechanism, it effectively reduces the offline computing delay of the traditional Lambda architecture, and also solves the problem of Kappa architecture's strong dependence on message middleware.

[0025] See also Figure 3 , Figure 3 It is a structural diagram of an embodiment of a data processing system provided in the present application. The data processing system is provided with a data processing architecture, and the data processing architecture includes: a real-time processing layer 10 and a batch processing layer 20.

[0026] The real-time processing layer 10 is used to receive the first data to be processed in real time, and copy the first data to be processed to obtain the second data to be processed; specifically, the first data to be processed is newly acquired data, which can be image, sound or text data. After receiving the first data to be processed, the real-time processing layer 10 copies the first data to be processed to generate the second data to be processed, and sends the second data to be processed to the batch processing layer 20.

[0027] The real-time processing layer 10 is also used to determine whether all the second data to be processed corresponding to the current batch processing task have been received. If all the second data to be processed corresponding to the current batch processing task have been received, an event notification message is generated; specifically, the real-time processing layer 10 is used to count whether the number of the second data to be processed corresponding to the current batch processing task (i.e., the offline task) has reached a preset number. If it is determined that the number of the second data to be processed corresponding to the current batch processing task has reached a preset number, it indicates that the second data to be processed required to execute the current batch processing task is sufficient, and the batch processing layer 20 can execute the current batch processing task at this time; or after receiving the execution instruction issued by the user, an event notification message can be generated and sent to the batch processing layer 20 to remind the batch processing layer 20 that the current batch processing task can be executed at this time.

[0028] The real-time processing layer 10 can also process the first data to be processed after receiving it to obtain an execution result; for example, the first data to be processed can be a face image, and the real-time processing layer 10 can extract features from the face image to obtain face features, and then perform clustering processing based on the face features.

[0029] The batch processing layer 20 is connected to the real-time processing layer 10, and is used for processing all received second data to be processed after receiving the event notification message, so as to execute the current batch processing task.

[0030] First, there is only one data stream on the data source side (not shown in the figure). The real-time processing layer 10 is the only entry for the data stream. After receiving the real-time first data to be processed, the real-time processing layer 10 copies and diverts the data. One copy of the data (i.e., the second data to be processed) is used for the subsequent batch processing layer 20 to perform offline calculations, and the other copy of the data (i.e., the first data to be processed) is directly calculated in real time by the real-time processing layer 10 to obtain corresponding real-time indicators. In addition, the real-time processing layer 10 can generate event notification messages that trigger the batch processing layer 20 to execute the current batch processing task, so that the real-time processing layer 10 and the batch processing layer 20 can interact with each other, and the application scenarios are relatively wide.

[0031] This embodiment provides a solution for triggering batch processing tasks by streaming tasks based on event notification. The provided solution mainly involves technical fields such as real-time computing, offline computing, event notification, message queues, streaming and batch integration or big data architecture, and especially involves the linkage technology of real-time computing and offline computing. The benefits of this embodiment are as follows: (1) In the existing design, real-time computing and offline computing are isolated from each other, and each performs its own tasks, which cannot meet the needs of some scenarios where real-time computing and offline computing interact with each other. The solution adopted in this embodiment enables the real-time processing layer and the batch processing layer to interact, and has a wider range of application scenarios; (2) The indicator correction of the traditional Lambda architecture is based on the timed triggering of offline tasks, which has a high delay. In this embodiment, since the real-time processing layer is used to trigger the tasks to be executed by the batch processing layer, there is no need to wait until the set time to trigger the current batch processing task, so the waiting time for executing the current batch processing task can be shortened, making the delay time of offline processing shorter; (3) In the existing design, real-time data and offline persistent data are two independent links, and it may happen that the data on one side is out but cannot be found on the other side, resulting in data inconsistency in the integration of stream and batch. In the solution adopted in this embodiment, since the real-time processing layer and the batch processing layer are connected, the data consistency is better.

[0032] See also Figure 4 , Figure 4 It is a structural diagram of another embodiment of the data processing system provided by the present application. The data processing system is provided with a data processing architecture, and the data processing architecture includes: a real-time processing layer 10 and a batch processing layer 20.

[0033] The real-time processing layer 10 is used to receive the first data to be processed in real time, and copy the first data to be processed to obtain the second data to be processed; specifically, the real-time processing layer 10 includes a data flow module 11 and an event trigger module 12 that are interconnected. The data flow module 11 is used to receive the first data to be processed from the data source 30 in real time, copy the first data to be processed to obtain the second data to be processed, and send the first data to be processed to the batch processing layer 20 and the event trigger module 12.

[0034] The real-time processing layer 10 is also used to determine whether all the second data to be processed corresponding to the current batch processing task have been received, and if all the second data to be processed corresponding to the current batch processing task have been received, an event notification message is generated. Specifically, the event triggering module 12 is used to determine whether all the second data to be processed corresponding to the current batch processing task have been received based on the first data to be processed.

[0035] The event trigger module 12 is also used to count the number of first data to be processed received within a preset time period, determine whether the number of first data to be processed is equal to the preset number, and generate an event notification message if the number of first data to be processed is equal to the preset number. Specifically, the preset time period includes a start time and an end time, the start time is the end time of data reception of the previous batch processing task of the current batch processing task, and the end time is the end time of data reception of the current batch processing task.

[0036] Furthermore, if Figure 4 As shown, the real-time processing layer 10 also includes a real-time indicator calculation module 13, which is used to perform real-time calculation on the first data to be processed, obtain a first calculation result, and persist the first calculation result in a real-time calculation result set.

[0037] like Figure 4 As shown, the data processing architecture also includes an event center 40, which is connected to the real-time processing layer 10. The event center 40 is used to push event notification messages to the batch processing layer 20 after receiving the event notification messages.

[0038] The batch processing layer 20 is connected to the real-time processing layer 10, and is used for processing all received second data to be processed after receiving the event notification message, so as to execute the current batch processing task.

[0039] In a specific embodiment, the batch processing layer 20 includes an event subscription module 21, a database 22 and a batch indicator calculation module 23. The event subscription module 21 is connected to the event center 40. The event subscription module 21 is used to send a subscription request to the event center 40 so that the event center 40 pushes event notification messages to the event subscription module 21.

[0040] The database 22 is connected to the batch index calculation module 23 and the data flow module 11 , and the database 22 is used to receive the second data to be processed sent by the data flow module 11 and store it.

[0041] The batch index calculation module 23 is used to perform real-time calculation on the second data to be processed received within a preset time, obtain a second calculation result, and persist the second calculation result in the batch calculation result set. Specifically, the batch index calculation module 23 can obtain the second data to be processed from the database 22. After the event trigger module 12 generates an event notification message, the batch index calculation module 23 actually enters the user-defined real-time calculation code and performs corresponding logical operations.

[0042] Continue reading Figure 4 The data processing architecture also includes a service layer 50, which is used to store batch calculation result sets and real-time calculation result sets, and after obtaining the second calculation result, correct the first calculation result corresponding to the second calculation result.

[0043] In the link of processing real-time streaming data (i.e., the first data to be processed), the first data to be processed enters the event trigger module 12. The event trigger module 12 determines whether all the data that the offline task depends on has been consumed. If all the dependent data has been consumed, an event notification message is generated; the event trigger module 12 sends the event notification message to the event center 40. When the batch indicator calculation module 23 receives the event notification message, it immediately triggers the corresponding batch processing task. After the calculation result is obtained, the real-time calculated indicator can be immediately corrected or pushed out for display.

[0044] In a specific embodiment, suppose that it is necessary to process real-time image streams for feature extraction and then clustering. The customer expects the delay to be as small as possible, and can tolerate a maximum clustering delay of 1 hour. The clustering algorithm works best when the data volume is 100,000.

[0045] 1) Real-time part: The real-time processing layer 10 receives the real-time image stream to extract features and obtain feature values, and then persists the feature values; at the same time, the event trigger module 12 tracks the processed data and records the data processing volume from the last event sending time (i.e., the last time the event notification message was sent) to the current moment. If the data processing volume reaches 10w, an event notification message is generated and sent to the batch processing layer 20 through the event center 40.

[0046] 2) Offline part: The batch processing layer 20 subscribes to the event notification message related to the offline clustering event, and sends out a clustering event after receiving the event notification message, clusters 100,000 data, and obtains the clustering result.

[0047] The advantage of this solution is that it minimizes the delay in the result generation through the event trigger mechanism while ensuring the optimal algorithm. If there is no event trigger mechanism, the batch processing task can only be triggered for detection every few minutes or every hour, resulting in high delays or empty runs.

[0048] In another specific embodiment, assuming that there is a real-time vehicle passing data, it is necessary to associate the basic vehicle information during real-time data processing, and then perform relevant calculations. The basic vehicle information as dimension table data is generally synchronized through scheduled updates; if the captured information of the passing vehicle has been obtained, but the scheduled update of the basic vehicle information has not been triggered, this data will be discarded as dirty data, that is, the basic vehicle information cannot be matched, but in fact it is just that the basic vehicle information has not been updated.

[0049] If the basic vehicle information is updated through an event trigger mechanism, the process is as follows:

[0050] 1) The vehicle data enters the link where real-time calculation is performed.

[0051] 2) The vehicle passing data is associated with the vehicle basic information dimension table, and it is found that the vehicle basic information dimension table does not have matching vehicle basic information.

[0052] 3) Cache the vehicle passing data and generate an update event for the basic vehicle information.

[0053] 4) The batch processing layer 20 receives the update event and updates the vehicle basic information dimension table.

[0054] 5) Cache the vehicle data, re-associate with the updated vehicle basic information table, and perform subsequent calculations.

[0055] Through the above steps, it is possible to effectively reduce the situation where normal data is misjudged as dirty data and discarded due to failure to synchronize data in a timely manner.

[0056] In the solution provided by this embodiment, a piece of data (i.e., the second data to be processed) output by the data flow module is directly placed in a persistent database for subsequent offline calculation; another piece of data (i.e., the first data to be processed) output by the data flow module directly enters the real-time indicator calculation module, thereby calculating the corresponding real-time indicators in real time. On the one hand, this processing method solves the problem that the Kappa architecture strongly relies on the persistence of the message middleware, and since in the construction of modern data warehouses, the data must eventually be persisted in the database, there is no problem of increasing storage; on the other hand, it can make the data that the results of real-time calculations depend on all be found in the database, solving the problem of data inconsistency in the Lambda architecture. The solution provided by this embodiment sets up an event notification mechanism to break the isolation of real-time calculations and offline calculations, so that real-time calculations can trigger offline calculations, so that real-time and offline calculations can be combined to adapt to more scenarios and solve more problems; and it reduces the delay of offline calculations in the traditional Lambda architecture, making offline calculations more real-time and accurate; in addition, real-time data and offline data are synchronized from the same data stream, and there will be no situation where data is not available on the other side after it comes out, solving the problem of data inconsistency in the stream-batch integration solution.

[0057] See also Figure 5 , Figure 5 1 is a flow chart of an embodiment of a method for triggering a batch processing task provided by the present application. The method is applied to a data processing system. The data processing system is arranged with a data processing architecture. The data processing architecture includes a real-time processing layer and a batch processing layer. The method includes:

[0058] Step 51: Use the real-time processing layer to receive the first data to be processed in real time, copy the first data to be processed to obtain the second data to be processed; determine whether all the second data to be processed corresponding to the current batch processing task have been received, and if all the second data to be processed corresponding to the current batch processing task have been received, generate an event notification message.

[0059] The data flow module is used to receive streaming data (i.e., the first data to be processed) from the data source, and then the streaming data is copied. One copy of the data (i.e., the second data to be processed) is directly persisted in the database of the batch processing layer, and the other copy of the data (i.e., the first data to be processed) enters the real-time indicator calculation module; the event triggering module is mainly responsible for determining whether all the data required for offline calculation is consumed and persistent. If the conditions are met, an event notification message is generated and sent to the event center. After receiving the event notification message, the event subscription module issues the corresponding batch processing task.

[0060] Step 52: After receiving the event notification message, the batch processing layer processes all received second data to be processed to execute the current batch processing task.

[0061] After receiving the current batch processing task sent by the event subscription module, the batch indicator calculation module in the batch processing layer processes all the second to-be-processed data of the current to-be-processed task, completes the processing of the current to-be-processed task, and obtains the execution result.

[0062] This embodiment breaks the isolation between real-time computing and offline computing through an event notification mechanism, ensures data consistency, and broadens the application scenarios of the architecture; moreover, the current batch processing task is triggered by an event trigger mechanism instead of a timing trigger mechanism. The event notification mechanism reduces the problem of high delay in correcting data in the traditional Lambda architecture, helps to shorten the delay time for processing offline tasks, and further shortens the time for correcting real-time results.

[0063] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 60 is used to store a computer program 61. When the computer program 61 is executed by a processor, it is used to implement the data processing system in the above embodiment.

[0064] The computer-readable storage medium 60 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which may store program codes.

[0065] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0066] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0067] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0068] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing system, characterized in that: The data processing system is provided with a data processing architecture, and the data processing architecture comprises: The real-time processing layer is used to receive the first data to be processed in real time, copy the first data to be processed to obtain the second data to be processed, and process the first data to be processed to obtain the first calculation result; determine whether all the second data to be processed corresponding to the current batch processing task have been received, and if so, generate an event notification message; A batch processing layer, connected to the real-time processing layer, for processing all the received second data to be processed after receiving the event notification message, to obtain a second calculation result, so as to execute the current batch processing task; The service layer is used to store the batch calculation result set and the real-time calculation result set, and after obtaining the second calculation result, to modify the first calculation result corresponding to the second calculation result.

2. The data processing system according to claim 1, characterized in that The real-time processing layer includes a data flow module and an event triggering module connected to each other, wherein the data flow module is used to receive the first data to be processed in real time, copy the first data to be processed to obtain second data to be processed, and send the first data to be processed to the batch processing layer and the event triggering module; The event triggering module is used to determine whether all the second data to be processed corresponding to the current batch processing task have been received based on the first data to be processed.

3. The data processing system according to claim 2, characterized in that: The event triggering module is further used to count the number of the first data to be processed received within a preset time period, determine whether the number of the first data to be processed is equal to a preset number, and if so, generate the event notification message.

4. The data processing system according to claim 3, characterized in that: The preset time period includes a start time and an end time, the start time is the end time of data reception of the previous batch processing task of the current batch processing task, and the end time is the end time of data reception of the current batch processing task.

5. The data processing system according to claim 1, characterized in that: The data processing architecture also includes an event center, which is connected to the real-time processing layer and is used to push the event notification message to the batch processing layer after receiving the event notification message.

6. The data processing system according to claim 5, characterized in that: The batch processing layer includes an event subscription module, which is connected to the event center and is used to send a subscription request to the event center so that the event center pushes the event notification message to the event subscription module.

7. The data processing system according to claim 1, characterized in that: The real-time processing layer also includes a real-time indicator calculation module, which is used to perform real-time calculation on the first data to be processed to obtain a first calculation result, and persist the first calculation result in a real-time calculation result set.

8. The data processing system according to claim 1, characterized in that: The batch processing layer also includes a batch index calculation module, which is used to perform real-time calculation on the second to-be-processed data received within a preset time to obtain a second calculation result, and persist the second calculation result in a batch calculation result set.

9. The data processing system according to claim 8, characterized in that: The batch processing layer also includes a database, which is connected to the batch index calculation module and the data flow module, and is used to receive and store the second data to be processed sent by the data flow module.

10. The data processing system according to claim 1, characterized in that: The service layer is also used to store batch calculation result sets and real-time calculation result sets.

11. A method for triggering a batch processing task, characterized in that: Applied to a data processing system, the data processing system is arranged with a data processing architecture, the data processing architecture includes a real-time processing layer, a batch processing layer and a service layer, the method includes: Using the real-time processing layer to receive first data to be processed in real time, copying the first data to be processed to obtain second data to be processed, and processing the first data to be processed to obtain a first calculation result; determining whether all second data to be processed corresponding to the current batch processing task have been received, and if so, generating an event notification message; After receiving the event notification message, the batch processing layer processes all the received second data to be processed to obtain a second calculation result, so as to execute the current batch processing task; After obtaining the second calculation result, the service layer modifies the first calculation result corresponding to the second calculation result.

12. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the batch processing task triggering method described in claim 11.

Images