A batch processing method and device of service data, electronic equipment and storage medium
By introducing a micro-batch processing paradigm and a queuing mechanism into the SQL engine, the low throughput problem caused by frequent interactions between UDF data processing logic and external services was solved, achieving more efficient data processing and performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2024-04-22
- Publication Date
- 2026-07-14
AI Technical Summary
In existing SQL engines, when the data processing logic of user-defined functions (UDFs) involves external interactions, each piece of business data needs to interact with the external service separately, resulting in low overall throughput.
By introducing a micro-batch processing paradigm into the SQL engine, the business data to be processed is batched using a queue mechanism, and external data interaction is performed when certain conditions are met, reducing the frequency of interaction with external services. The cache snapshot mechanism of the Flink SQL engine ensures that data is not lost.
It improves overall throughput, reduces development and maintenance difficulty, enhances data processing performance, and is suitable for both real-time and offline scenarios.
Smart Images
Figure CN118332003B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a batch processing method, apparatus, electronic device and storage medium for business data. Background Technology
[0002] Currently, the main open-source big data computing engines that support Structured Query Language (SQL) include Spark and Flink. Users often use User-Defined Functions (UDFs) when using SQL with the help of Spark and Flink.
[0003] Generally, regardless of whether it's Spark or Flink, the development specifications for UDFs provided by SQL engines are consistent: users must implement an `eval` method that takes a single row of data as input and outputs the computational result for that row. However, due to the API specifications of current SQL engine-provided UDFs, a UDF only accepts and returns one piece of business data. Therefore, when the UDF's data processing logic involves external interactions, each piece of business data to be processed by the UDF needs to interact with the external service separately, which can significantly impact overall throughput and lead to performance issues. Summary of the Invention
[0004] This application provides a batch processing method, apparatus, electronic device, and storage medium for business data to solve the problem in related technologies where, when the data processing logic of a UDF provided by an SQL engine involves external interaction, each piece of business data to be processed by the UDF needs to interact with an external service separately, resulting in low overall throughput.
[0005] In a first aspect, embodiments of this application provide a batch processing method for business data, applied to an SQL engine, including:
[0006] If it is determined that the user-defined function UDF contained in the obtained structured query language SQL statement is a micro-batch function, then the batch processing information of the UDF is obtained, and the batch processing information includes at least the batch size;
[0007] Create a queue based on the batch size;
[0008] The acquired business data to be processed by the UDF is written into the queue one by one until the batch processing condition is met. Then, the external data required by the UDF to process each business data in the queue is obtained from the external service in batches.
[0009] The UDF is used to process the business data in the queue and the external data to obtain the batch processing results of the business data in the queue.
[0010] In some embodiments, when each business data is offline data, the batch processing condition is that the queue is full.
[0011] In some embodiments, when the business data is real-time data, the batch processing information further includes a latency threshold, and the batch processing condition is:
[0012] The processing time of the business data at the head of the queue has reached the latency threshold, or...
[0013] The processing time of the business data at the head of the queue did not reach the delay threshold, but the queue was full.
[0014] In some embodiments, it also includes:
[0015] Take snapshots of the acquired business data to be processed by the UDF at a set frequency;
[0016] If the data recovery conditions are met, the business data in the queue is recovered based on the snapshot results;
[0017] The UDF is reused to process the business data in the queue.
[0018] In some embodiments, after obtaining the batch processing results of each business data in the queue, the method further includes:
[0019] If it is determined that the number of data entries in the batch processing result is equal to the actual number of data entries in the queue, then the batch processing result is pushed to the downstream.
[0020] If it is determined that the number of data entries in the batch processing result is not equal to the actual number of data entries in the queue, an exception message is sent.
[0021] Secondly, embodiments of this application provide a batch processing apparatus for business data, applied to an SQL engine, comprising:
[0022] The acquisition module is used to acquire batch processing information of a user-defined function (UDF) contained in an acquired structured query language (SQL) statement if it is determined that the UDF is a micro-batch function. The batch processing information includes at least the batch size.
[0023] A module is created to create a queue based on the batch size;
[0024] The batch processing module is used to write the acquired business data to be processed by the UDF into the queue one by one until the batch processing conditions are met, and then retrieve the external data required by the UDF to process each business data in the queue from the external service in batches.
[0025] The business processing module is used to process the business data in the queue and the external data using the UDF to obtain the batch processing results of the business data in the queue.
[0026] In some embodiments, when each business data is offline data, the batch processing condition is that the queue is full.
[0027] In some embodiments, when the business data is real-time data, the batch processing information further includes a latency threshold, and the batch processing condition is:
[0028] The processing time of the business data at the head of the queue has reached the latency threshold, or...
[0029] The processing time of the business data at the head of the queue did not reach the delay threshold, but the queue was full.
[0030] In some embodiments, it also includes:
[0031] The snapshot module is used to take snapshots of the acquired business data to be processed by the UDF at a set frequency.
[0032] The business processing module is further configured to, if the data recovery conditions are met, restore the business data in the queue based on the snapshot results; and reuse the UDF to process the business data in the queue.
[0033] In some embodiments, the result determination module is further configured to:
[0034] After obtaining the batch processing results of each business data in the queue, if it is determined that the number of data entries contained in the batch processing results is equal to the actual number of data entries contained in the queue, then the batch processing results are pushed to the downstream.
[0035] If it is determined that the number of data entries in the batch processing result is not equal to the actual number of data entries in the queue, an exception message is sent.
[0036] Thirdly, embodiments of this application provide an electronic device, including: at least one processor, and a memory communicatively connected to the at least one processor, wherein:
[0037] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to perform the batch processing method for the aforementioned business data.
[0038] Fourthly, embodiments of this application provide a storage medium in which, when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device is able to execute the aforementioned batch processing method for business data.
[0039] In this embodiment, if the SQL engine determines that the UDF contained in the acquired SQL statement is a micro-batch function, it acquires the batch processing information of the UDF. The batch processing information includes at least the batch size. A queue is created based on the batch size. Then, the acquired business data to be processed by the UDF is written into the queue one by one until the batch processing conditions are met. At this point, the external data required by each business data in the UDF processing queue is acquired in batches from an external service. Then, the UDF is used to process each business data and external data in the queue to obtain the batch processing results of each business data in the queue. In this way, batch processing is performed using a queue, and the batch processing does not need to be implemented by the user. When developing the UDF, the user only needs to specify the batch call logic, which can effectively reduce the interaction frequency with external services, improve the overall throughput and data processing performance, and reduce the development and maintenance difficulty. Attached Figure Description
[0040] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0041] Figure 1 A data interaction diagram provided for an embodiment of this application;
[0042] Figure 2 This is yet another data interaction diagram provided in an embodiment of this application;
[0043] Figure 3 This is yet another data interaction diagram provided in an embodiment of this application;
[0044] Figure 4 A flowchart illustrating a batch processing method for business data provided in this application embodiment;
[0045] Figure 5 A flowchart of yet another batch processing method for business data provided in an embodiment of this application;
[0046] Figure 6 A schematic diagram of the structure of a batch processing device for business data provided in an embodiment of this application;
[0047] Figure 7 This is a schematic diagram of the hardware structure of an electronic device for implementing a batch processing method for business data, provided in an embodiment of this application. Detailed Implementation
[0048] To address the issue in related technologies where each piece of business data to be processed by the UDF needs to interact with an external service individually when the data processing logic of the UDF provided by the SQL engine involves external data interaction, resulting in low overall throughput, this application provides a batch processing method, apparatus, electronic device, and storage medium for business data.
[0049] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.
[0050] Currently, the main open-source big data computing engines that support SQL include Spark and Flink. Both Flink and Spark have Application Programming Interfaces (APIs) for DataFrames / DataStreams and Table / Structured Query Language (SQL) APIs. Users frequently utilize User-Defined Functions (UDFs), User-Defined Aggregate Functions (UDAFs), and User-Defined Table-Generating Functions (UDTFs) when using SQL with Spark and Flink.
[0051] Whether using Spark or Flink, the UDF development specifications provided by the SQL engine are consistent: users must implement an `eval` method that takes a single row of data as input and outputs the computational result of that row. See also... Figure 1 , Figure 1 This diagram illustrates a data interaction in an embodiment of this application. The input arrow represents business data, the rectangle represents the UDF processing procedure, the cylinder represents external services, and the output arrow represents the processing result of the business data. Due to the API specifications of current SQL engines' UDFs, a UDF only receives and returns one piece of business data. Therefore, when the processing logic of business data in a UDF involves external data interaction, each piece of business data needs to undergo separate external interaction. This significantly impacts the overall throughput of the SQL engine, resulting in performance degradation.
[0052] Taking the Flink SQL engine as an example, in scenarios where SQL is used while maintaining performance, UDAF or UDTF can be considered to solve the above problems. Since the current operation is batch processing, although UDAF will aggregate, it requires specifying the aggregation logic. SQL cannot express batch processing for a certain amount of business data, so it cannot be used. Therefore, UDTF must be considered. The logic of UDTF is to generate multiple rows of output for a single input. For details, see [link to relevant documentation]. Figure 2 You can set up data caching in the UDTF method. If the specified number of data is not reached, the data will be cached in memory. If the specified number of data is reached, the batch processing logic will be called to process a batch of data, and then the batch processing results of all data will be pushed downstream.
[0053] However, requiring users to implement the approval / praise actions and maintain the data programmatically presents another problem. For example, if an external service interaction is scheduled for 32 business data entries, the business data is cached in memory until these 32 entries are reached. If a restart exception occurs, the business data in memory will be lost. In real-time scenarios, this manifests as data loss, which is unacceptable. Storing the business data in an external reliable cache also suffers from the same performance issues associated with external interactions. While Flink has an internal checkpoint snapshot mechanism for stateful events, it is not suitable for UDTFs because UDTFs are designed to process individual business data entries and do not have a data caching concept. Furthermore, implementing state persistence logic during UDTF development is not in line with the specifications for users.
[0054] At this point, users have no choice but to abandon the SQL API and use the underlying DataStream API instead. The underlying API is more flexible and can meet the performance requirements, but it will increase the development threshold. In addition, the delivered package is an engineering package, which will also increase the difficulty of operation and maintenance.
[0055] Similarly, the Spark SQL engine also suffers from the same problems mentioned above, which will not be elaborated upon here.
[0056] To address this, this application proposes a UDF-based micro-batch paradigm and provides an implementation scheme in a FlinkSQL scenario. In this scheme, for data processing tasks triggered by SQL statements, the SQL engine handles the exception handling for batch data calls. Users only need to specify the batch call logic when developing UDFs, without needing to worry about exception handling. At the SQL level, the normal UDF usage remains; users write their original development logic when writing SQL. However, the underlying SQL engine can control the batch size and latency through configuration, thereby controlling the overall throughput. This paradigm is applicable to various SQL engines and is suitable for both real-time and offline scenarios.
[0057] The following section uses the Flink SQL engine as an example to introduce the solution of this application embodiment.
[0058] In this proposal, the UDF paradigm API design for supporting micro-batch is as follows:
[0059] public class udf{
[0060] List <row>eval(List <row>);
[0061] }
[0062] In other words, in addition to the original single-item input data, the `eval` method now supports array-based input parameters. At this point, the UDF definition must ensure that the length of the input array and the length of the output array are consistent. For scenarios where they are inconsistent, the Flink SQL engine can handle them directly as exceptions.
[0063] For UDF scenarios involving offline tasks (non-real-time data processing tasks triggered by SQL statements), task recovery or a complete task restart can be performed based on the minimum recovery capability of the corresponding Flink SQL engine in case of task exceptions. For UDF scenarios involving real-time tasks (real-time data processing tasks triggered by SQL statements), the Flink SQL engine can implement a snapshot caching mechanism to ensure data is not lost.
[0064] Furthermore, this paradigm API proposes a tuning method based on configuration-controlled micro-batches, with the following configuration style:
[0065] udf.minibatch.size,
[0066] udf.minibatch.latcy,
[0067] Among them, udf.minibatch.size can control the micro-batch size of UDF, that is, how many business data are in a batch, which is the array size in the API above. udf.minibatch.latcy can control the maximum delay processing time of a batch of business data. This configuration item is only effective in real-time scenarios.
[0068] The implementation of the Flink SQL engine includes the design of the micro-batch API provided in the UDF, as well as how the Flink SQL engine determines whether to call the micro-batch processing logic, how to persist cached data, and how to control the size of the cache or trigger timeouts through configuration.
[0069] Taking the SQL statement `select miniBatch(a) from tableA` as an example, `miniBatch` is the UDF function to be implemented.
[0070] In practice, a base class `MiniBatchScalarFunction` can be designed. The code of the class itself is consistent with the original UDF base class `ScalarFunction`, except that when the user implements the `eval` method, both the input and output parameters need to be `List`. <row>.
[0071] The APIs are as follows:
[0072] public abstract class MiniBatchScalarFunction extends UserDefinedFunction{
[0073] List <row>eval(List <row>);
[0074] }
[0075] The difference between this API and the original API is whether the eval method is for batch business data; users do not need to worry about caching or other logic.
[0076] During the Flink SQL engine parsing process, the function type is determined in the expression parsing. If the function type is a MiniBatchScalarFunction, then the operator designed in this method can be generated when converting the logical plan to the physical plan. The operator structure is as follows: Figure 3 As shown, an internal queue of a specified size is declared. When business data arrives, if the queue is not full, the business data can be written to the queue. If the queue is full, the actual eval calculation will be triggered. Due to the characteristics of UDFs, the number of input and output business data entries must be consistent; that is, the number of business data entries pushed downstream must be the same as the number of input business data entries.
[0077] The above process can be followed Figure 4 The process shown includes the following steps.
[0078] In step 401, new business data arrives.
[0079] In step 402, it is determined whether the queue is full. If not, proceed to step 403; if so, proceed to step 404.
[0080] In step 403, the business data is written to the queue.
[0081] In step 404, the business data in the queue is retrieved and converted into a List. <row>Format.
[0082] In step 405, the user's eval method is executed to obtain an eval result array.
[0083] The user's `eval` method is the method implemented by the micro-batch UDF.
[0084] In step 406, it is determined whether the number of business data entries in the eval result array is equal to the number of business data entries in the queue. If yes, proceed to step 407; otherwise, proceed to step 408.
[0085] In step 407, the eval result array is pushed downstream.
[0086] In step 408, an exception is thrown.
[0087] In real-time scenarios, data stream interruptions may occur, meaning there may be no business data. This can prevent the business data in the queue from triggering calculations. To address this, a built-in timeout mechanism can be implemented, such as controlling a latency threshold through configuration items. The latency threshold represents the maximum data latency time that the user can accept. Generally, configuration items are designed as task-level configurations and are unbound from the UDF class. This avoids modifying the code during UDF usage.
[0088] The above process has two configuration items: udf.miniBatch.size and udf.miniBatch.latcy. udf.miniBatch.size configures the batch size, and udf.miniBatch.latcy configures the latency threshold. These two configuration parameters are the batch processing information for the UDF, and udf.miniBatch.latcy is typically used in real-time scenarios.
[0089] The queue of a MiniBatchOperator can request state persistence through Flink's state center. Upon detecting a queue request, the Flink SQL engine persists the queue's state (i.e., takes a snapshot of the business data in the queue). Only after a snapshot is triggered is the input business data marked as consumed. This way, when data recovery conditions are met, such as the occurrence of a specified exception or an exception frequency reaching a set frequency, the business data in the queue can be recovered based on the snapshot results, avoiding data loss in real-time scenarios.
[0090] The following section describes the anomaly recovery process in real-time scenarios.
[0091] In real-time scenarios, the Flink SQL engine enables checkpointing for real-time data processing tasks triggered by SQL statements, performing a snapshot action on the task. Generally, snapshots are triggered by the task's source operator (the SQL statement is parsed into multiple operators in a tree structure, with the root of the tree being the source operator; the MiniBatchOperator is one of the operators in this tree structure). After the source operator is triggered, a data barrier is generated (the barrier flows between operators according to the tree structure). The barrier flows to the MiniBatchOperator along with the business data, and the MiniBatchOperator also needs to implement the snapshot action. Specifically, the snapshot process of the MiniBatchOperator is as follows: upon receiving a barrier, it sends the barrier to its downstream operators, copies the business data from the queue, serializes the copied business data, and writes the serialization result to a file. Subsequently, when the task resumes, the MiniBatchOperator reads the corresponding snapshot content and restores it to the queue.
[0092] It should be noted that in offline scenarios, when the data recovery conditions are met, the data processing task triggered by the SQL statement can be re-executed to obtain the final result, so the Flink SQL engine does not need to trigger a snapshot.
[0093] The solution in this application embodiment satisfies data processing in micro-batch mode at the SQL level. Compared to Flink's DataStream API or other engines' imperative APIs, which require code development and engineering-level maintenance, this significantly reduces development costs. Furthermore, compared to traditional UDFs that can only process single data entries, this solution greatly improves interaction efficiency and maximizes performance when exchanging batch data with external services. Unlike UDTFs (which themselves do not have stateful processing logic), which require users to implement both data processing logic (i.e., corresponding methods) and be aware of data caching and snapshot logic (i.e., maintaining data state), this solution is independent of the UDF. Users only need to implement the data processing logic without needing to be aware of the Flink SQL engine's state maintenance logic; the Flink SQL engine maintains the data state. Simultaneously, this solution can propose configuration items corresponding to the UDF within the engine, allowing users to balance latency and throughput in micro-batch scenarios.
[0094] It should be noted that the above introduction uses the Flink SQL engine as an example. In practical applications, other SQL engines such as Spark SQL engine are also feasible.
[0095] In this embodiment, the approval action is implemented through a queue. The size of the queue and the latency threshold of the business data in the queue are controlled by configuration. The state of the queue (i.e., the change of business data in the queue) is maintained by the SQL engine. When developing UDFs, users only need to specify the batch call logic without worrying about exception handling. Therefore, it can improve the overall throughput of the SQL engine and reduce development difficulty and maintenance costs.
[0096] Figure 5 A flowchart of a batch processing method for business data provided in an embodiment of this application is shown. The method is applied in an SQL engine and includes the following steps.
[0097] In step 501, if it is determined that the UDF contained in the obtained SQL statement is a micro-batch function, then the batch processing information of the UDF is obtained, and the batch processing information includes at least the batch size.
[0098] Taking the Flink SQL engine as an example, the API design for the UDF paradigm that supports micro-batch processing is as follows:
[0099] public class udf{
[0100] List <row>eval(List <row>);
[0101] }
[0102] In other words, in addition to the original single-entry business data, the `eval` method now supports array-based input parameters. At this point, the UDF definition must ensure that the length of the input array and the length of the output array are consistent. For scenarios where they are inconsistent, the SQL engine can directly handle them as exceptions.
[0103] In practice, batch processing information can be configured in the SQL engine beforehand, and the scope of application of the batch processing information can be configured, such as applying it to micro-batch UDFs in all SQL statements within a session, or applying it only to micro-batch UDFs in a specific SQL statement. However, regardless of the method used to configure batch processing information, the batch processing information corresponding to a micro-batch UDF can always be found.
[0104] In step 502, a queue is created based on the batch size.
[0105] Assuming the batch size is 10, a queue with 10 elements can be created.
[0106] In step 503, the acquired business data to be processed by UDF is written into the queue one by one until the batch processing conditions are met. Then, the external data required for batch processing of each business data in the UDF processing queue is obtained from the external service.
[0107] External services refer to services that are independent of the SQL engine, such as databases and servers. Furthermore, external services and the SQL engine can be deployed on the same device or on different devices.
[0108] In practical applications, the business data to be processed by UDF may be either offline or real-time data. However, regardless of whether it is offline or real-time data, the business data to be processed by UDF is generally obtained one by one. Whenever a piece of business data to be processed by UDF is obtained, it can be determined whether the batch processing conditions are met. If not, the business data can be written into the queue. If it is met, the external data required for each piece of business data in the UDF processing queue can be obtained in batch from the external service.
[0109] In some embodiments, each business data is offline data (corresponding to an offline scenario). In this case, each business data has been reliably stored and can be retrieved again when the data recovery conditions are met. There is no need to record the status of the queue (i.e., the changes in the data in the queue) and there is no problem of data delay processing. Therefore, the batch processing condition can be that the queue is full.
[0110] In some embodiments, the business data is real-time data (corresponding to a real-time scenario). In this case, the business data is a data stream generated over time, and the data processing latency in a real-time scenario cannot be too long. Therefore, the batch processing information may also include a latency threshold. The batch processing condition may be: the latency processing time of the business data at the head of the queue has reached the latency threshold, or the latency processing time of the business data at the head of the queue has not reached the latency threshold but the queue is full. That is, the UDF logic can be executed when the queue is not full but the latency processing time has reached the latency threshold, or when the latency processing time has not reached the latency threshold but the queue is full.
[0111] In addition, in real-time scenarios, in order to ensure that business data can still be obtained when the data recovery conditions are met, the SQL engine can be used to record the status of the queue (i.e., the changes in the data in the queue). Specifically, snapshots of the business data to be processed by the UDF are taken at a set frequency. Subsequently, when the data recovery conditions are met, the business data in the queue can be recovered based on the snapshot results, and then the UDF can be reused to process the business data in the queue.
[0112] In step 504, UDFs are used to process the business data and external data in the queue to obtain the batch processing results of the business data in the queue.
[0113] That is, to execute the UDF processing logic.
[0114] In step 505, it is determined whether the number of data entries contained in the batch processing result is equal to the actual number of data entries contained in the queue. If yes, proceed to step 506; otherwise, proceed to step 507.
[0115] Here, the micro-batch UDF must ensure that the length of the input array and the length of the output array are consistent. Therefore, by comparing whether the number of data items contained in the batch processing result (the length of the output array) and the number of data items actually contained in the queue (the length of the input array) are consistent, it can be determined whether there is an anomaly.
[0116] In step 506, the batch processing results are pushed to the downstream.
[0117] In practical applications, SQL statements can be nested. If there is another SQL statement outside the current SQL statement, then pushing the batch processing result to the downstream is equivalent to pushing the batch processing result to the outer SQL statement and using it as the input of the outer SQL statement. If there is no SQL statement outside the current SQL statement, then pushing the batch processing result to the downstream is equivalent to outputting the batch processing result as part of the result.
[0118] At this point, the business data in the queue can be cleared, and the process can return to step 503 until the data processing stop condition is met, at which point the acquisition of business data is stopped and the queue is released.
[0119] In step 507, an exception message is sent.
[0120] Based on the same technical concept, this application also provides a batch processing device for business data. The principle of the batch processing device for business data is similar to that of the batch processing method for business data described above. Therefore, the implementation of the batch processing device for business data can refer to the implementation of the batch processing method for business data, and the repeated parts will not be described again.
[0121] Figure 6 A schematic diagram of a batch processing device for business data provided in an embodiment of this application includes...
[0122] The acquisition module 601 is used to acquire batch processing information of the UDF if it is determined that the user-defined function UDF contained in the acquired structured query language SQL statement is a micro-batch function. The batch processing information includes at least the batch size.
[0123] Module 602 is used to create a queue based on the batch size;
[0124] Batch processing module 603 is used to write the acquired business data to be processed by the UDF into the queue one by one until the batch processing conditions are met, and then retrieve the external data required by the UDF to process each business data in the queue from the external service in batches.
[0125] The business processing module 604 is used to process the business data in the queue and the external data using the UDF to obtain the batch processing results of the business data in the queue.
[0126] In some embodiments, when each business data is offline data, the batch processing condition is that the queue is full.
[0127] In some embodiments, when the business data is real-time data, the batch processing information further includes a latency threshold, and the batch processing condition is:
[0128] The processing time of the business data at the head of the queue has reached the latency threshold, or...
[0129] The processing time of the business data at the head of the queue did not reach the delay threshold, but the queue was full.
[0130] In some embodiments, it also includes:
[0131] Snapshot module 605 is used to take snapshots of the acquired business data to be processed by the UDF at a set frequency;
[0132] The business processing module 604 is further configured to, if the data recovery conditions are met, restore the business data in the queue based on the snapshot results; and reuse the UDF to process the business data in the queue.
[0133] In some embodiments, the result determination module 606 is further configured to:
[0134] After obtaining the batch processing results of each business data in the queue, if it is determined that the number of data entries contained in the batch processing results is equal to the actual number of data entries contained in the queue, then the batch processing results are pushed to the downstream.
[0135] If it is determined that the number of data entries in the batch processing result is not equal to the actual number of data entries in the queue, an exception message is sent.
[0136] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. Coupling between modules can be achieved through interfaces, typically electrical communication interfaces, but mechanical interfaces or other types of interfaces are also possible. Therefore, modules described as separate components may or may not be physically separate; they can be located in one place or distributed across different locations on the same or different devices. The integrated modules described above can be implemented in hardware or as software functional modules.
[0137] After introducing the batch processing method and apparatus for business data according to exemplary embodiments of this application, an electronic device according to another exemplary embodiment of this application will be introduced next.
[0138] The following reference Figure 7 To describe an electronic device 130 implemented according to this embodiment of the present application. Figure 7 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0139] like Figure 7 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).
[0140] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.
[0141] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.
[0142] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0143] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0144] In an exemplary embodiment, a storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the batch processing method for the aforementioned business data. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.
[0145] In an exemplary embodiment, the electronic device of this application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, it enables the at least one processor to perform the steps of any batch processing method for business data provided in the embodiments of this application.
[0146] In an exemplary embodiment, a computer program product is also provided, which, when executed by an electronic device, enables the electronic device to implement any of the exemplary methods provided in this application.
[0147] Furthermore, computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0148] The program product for batch processing of business data in the embodiments of this application may be a CD-ROM and include program code, and may run on a computing device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program, which may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0149] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0150] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency (RF), or any suitable combination thereof.
[0151] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0152] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.
[0153] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0155] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0158] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0159] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, then this application also includes such modifications and variations.< / row> < / row> < / row> < / row> < / row> < / row> < / row> < / row>
Claims
1. A batch processing method for business data, characterized in that, Applied to the SQL engine, the method includes: If it is determined that the user-defined function UDF contained in the obtained structured query language SQL statement is a micro-batch function, then the batch processing information of the UDF is obtained, and the batch processing information includes at least the batch size; Create a queue based on the batch size; The acquired business data to be processed by the UDF is written into the queue one by one until the batch processing condition is met. Then, the external data required by the UDF to process each business data in the queue is obtained from the external service in batches. The UDF is used to process the business data in the queue and the external data to obtain the batch processing results of the business data in the queue.
2. The method as described in claim 1, characterized in that, When the data for each business is offline, the batch processing condition is that the queue is full.
3. The method as described in claim 1, characterized in that, When the business data is real-time data, the batch processing information also includes a latency threshold, and the batch processing condition is: The processing time of the business data at the head of the queue has reached the latency threshold, or... The processing time of the business data at the head of the queue did not reach the delay threshold, but the queue was full.
4. The method as described in claim 3, characterized in that, Also includes: Take snapshots of the acquired business data to be processed by the UDF at a set frequency; If the data recovery conditions are met, the business data in the queue is recovered based on the snapshot results; The UDF is reused to process the business data in the queue.
5. The method according to any one of claims 1-4, characterized in that, After obtaining the batch processing results of each business data in the queue, the process also includes: If it is determined that the number of data entries in the batch processing result is equal to the actual number of data entries in the queue, then the batch processing result is pushed to the downstream. If it is determined that the number of data entries in the batch processing result is not equal to the actual number of data entries in the queue, an exception message is sent.
6. A batch processing device for business data, characterized in that, Applied to the SQL engine, including: The acquisition module is used to acquire batch processing information of a user-defined function (UDF) contained in an acquired structured query language (SQL) statement if it is determined that the UDF is a micro-batch function. The batch processing information includes at least the batch size. A module is created to create a queue based on the batch size; The batch processing module is used to write the acquired business data to be processed by the UDF into the queue one by one until the batch processing conditions are met, and then retrieve the external data required by the UDF to process each business data in the queue from the external service in batches. The business processing module is used to process the business data in the queue and the external data using the UDF to obtain the batch processing results of the business data in the queue.
7. The apparatus as claimed in claim 6, characterized in that, When the data for each business is offline, the batch processing condition is that the queue is full.
8. The apparatus as claimed in claim 6, characterized in that, When the business data is real-time data, the batch processing information also includes a latency threshold, and the batch processing condition is: The processing time of the business data at the head of the queue has reached the latency threshold, or... The processing time of the business data at the head of the queue did not reach the delay threshold, but the queue was full.
9. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to said at least one processor, wherein: The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-5.
10. A storage medium, characterized in that, When the computer program in the storage medium is executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-5.