Data processing method and device, electronic equipment and computer readable medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer readable medium, and relates to the technical field of data processing.The method comprises the steps that in response to the fact that service logic execution is completed, service data of a current batch is obtained, and a multi-level cache memory table is called to obtain auxiliary data corresponding to the service data; determining a to-be-submitted transaction and a transaction type based on the business data and the auxiliary data; determining a data source corresponding to the transaction, and starting a task thread to start and submit the transaction based on the data source and the transaction type; in response to the exception of the task instance, acquiring a transaction submission progress to determine a breakpoint transaction, and executing transaction submission automatic retry based on the breakpoint transaction, a data source corresponding to the breakpoint transaction, the transaction submission progress and a preset data consistency mode; and in response to the failure of the transaction submission automatic retry, generating and outputting abnormal information, and exiting the task thread. The complexity of development, operation and maintenance is reduced, the data query efficiency is improved, and the consistency of data processing is ensured.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method, apparatus, electronic device, and computer-readable medium. Background Art

[0002] In applications such as data processing and data migration that require processing a large amount of data, streaming data processing and batch data processing play crucial roles. With the continuous changes in the data volume and business requirements, a single streaming processing or batch processing has become difficult to meet the actual needs. However, the implementation methods of these two types of data processing are different, which increases the complexity of development and operation and maintenance. Moreover, during the data processing process, it is often necessary to query data from different data storage systems, resulting in low data query efficiency and inability to ensure the consistency of data processing after interruption recovery. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data processing method, apparatus, electronic device, and computer-readable medium, which can solve the problems that the implementation methods of existing streaming processing and batch processing are different, increasing the complexity of development and operation and maintenance, and during the data processing process, it is often necessary to query data from different data storage systems, resulting in low data query efficiency and inability to ensure the consistency of data processing after interruption recovery.

[0004] To achieve the above object, according to one aspect of the embodiments of this application, a data processing method is provided, including: upon completion of the execution of the business logic, obtaining the business data of the current batch, and invoking a multi-level cache memory table to obtain auxiliary data corresponding to the business data; based on the business data and the auxiliary data, determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction; determining the data source corresponding to the transaction, and starting a task thread to open and submit the transaction based on the data source and the transaction type; upon an exception occurring in the task instance corresponding to the task thread, obtaining the corresponding transaction submission progress, determining a breakpoint transaction according to the transaction submission progress, and performing automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode; upon failure of the automatic retry of transaction submission, generating an exception message and outputting it, and exiting the task thread.

[0005] Optionally, based on the business data and auxiliary data, determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction, including: determining the business data type corresponding to the business data and the auxiliary data type corresponding to the auxiliary data; according to the business data type and the auxiliary data type, determining the corresponding data table and message middleware, classifying the business data and the auxiliary data according to the data table and the message middleware to respectively obtain the input data corresponding to the data table and the input data corresponding to the message middleware; based on the data table, the message middleware and the respective corresponding input data, determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction.

[0006] Optionally, determine the breakpoint transaction according to the transaction submission progress, including: determining the transaction identifier at the breakpoint position according to the transaction submission progress, and determining the breakpoint transaction according to the transaction identifier; and performing automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and a preset data consistency mode, including: determining the serial number of the input data corresponding to the breakpoint transaction; according to the serial number, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode, determining the processing method for the input data corresponding to each transaction corresponding to the task instance; performing automatic retry of transaction submission based on the processing method.

[0007] Optionally, according to the serial number, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode, determine the processing method for the input data corresponding to each transaction of the task instance, including: determining the data source corresponding to the serial number according to the data source corresponding to the breakpoint transaction and the preset transaction submission priority; if the preset data consistency mode is repeatable execution, determining the processing method for the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, clearing all the input data corresponding to each transaction of the task instance that was historically written in the data source corresponding to the serial number, and then re-executing the task instance; determining the first serial number corresponding to the data table and the second serial number corresponding to the message middleware in the serial number, where the first serial number is greater than the second serial number; and if the preset data consistency mode is processing from the breakpoint, determining the processing method for the input data corresponding to each transaction of the task instance as: reading the input data for the data table corresponding to the first serial number but not writing it to the corresponding data table, reading the input data for the message middleware corresponding to the first serial number and writing it to the message middleware corresponding to the second serial number, and in response to successful writing, updating the second serial number to the first serial number, and then updating the transaction submission progress, and continuing to execute the task instance based on the updated transaction submission progress; if the preset data consistency mode is processing from the beginning, determining the processing method for the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, reading the input data for the message middleware corresponding to the second serial number and the serial numbers before the second serial number but not writing it to the corresponding message middleware, reading the input data for the data table corresponding to the first serial number and the serial numbers before the first serial number but not writing it to the corresponding data table, reading the input data for the message middleware corresponding to the first serial number and writing it to the message middleware corresponding to the second serial number, and in response to successful writing, updating the second serial number to the first serial number, and then updating the transaction submission progress, and continuing to execute the task instance based on the updated transaction submission progress.

[0008] Optionally, the data processing method further includes: in response to the task instance corresponding to the task thread not having an exception and the current batch being the last batch, obtaining the corresponding transaction submission result data and outputting it, and exiting the task thread.

[0009] Optionally, the data processing method further includes: in response to the task instance corresponding to the task thread not having an exception and the current batch not being the last batch, if no control instruction is received, continue to obtain the business data of the next batch as the business data of the current batch and continue with the data processing.

[0010] Optionally, the data processing method further includes: in response to the task instance corresponding to the task thread not having an exception, exiting the task thread when a control instruction is received.

[0011] In addition, the present application further provides a data processing device, including: an acquisition unit configured to acquire the service data of the current batch and call a multi-level cache memory table to acquire auxiliary data corresponding to the service data in response to the completion of the execution of the service logic; a transaction determination unit configured to determine the transaction to be committed corresponding to the service data and the transaction type of the transaction based on the service data and the auxiliary data; a transaction submission unit configured to determine the data source corresponding to the transaction, start a task thread to open and commit the transaction based on the data source and the transaction type; an exception handling unit configured to acquire the corresponding transaction submission progress in response to an exception occurring in the task instance corresponding to the task thread, determine the breakpoint transaction according to the transaction submission progress, and execute automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode; an output unit configured to generate and output an exception message and exit the task thread in response to the failure of the automatic retry of transaction submission.

[0012] Optionally, the transaction determination unit is further configured to: determine the service data type corresponding to the service data and the auxiliary data type corresponding to the auxiliary data; according to the service data type and the auxiliary data type, determine the corresponding data table and message middleware, classify the service data and the auxiliary data according to the data table and the message middleware to respectively obtain the input data corresponding to the data table and the input data corresponding to the message middleware; based on the data table, the message middleware, and the respective corresponding input data, determine the transaction to be committed corresponding to the service data and the transaction type of the transaction.

[0013] Optionally, the exception handling unit is further configured to: determine the transaction identifier at the breakpoint position according to the transaction submission progress, and determine the breakpoint transaction according to the transaction identifier; and the exception handling unit is further configured to: determine the serial number of the input data corresponding to the breakpoint transaction; according to the serial number, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode, determine the processing method for the input data corresponding to each transaction corresponding to the task instance; execute automatic retry of transaction submission based on the processing method.

[0014] Optionally, the exception handling unit is further configured to: determine the data source corresponding to the serial number according to the data source corresponding to the breakpoint transaction and the preset transaction submission priority; if the preset data consistency mode is repeatable execution, determine the processing method for the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, clear all the input data corresponding to each transaction of the task instance that has been historically written in the data source corresponding to the serial number, and then re-execute the task instance; determine the first serial number corresponding to the data table and the second serial number corresponding to the message middleware in the serial number, where the first serial number is greater than the second serial number; and if the preset data consistency mode is processing from the breakpoint, determine the processing method for the input data corresponding to each transaction of the task instance as: read the input data for the data table corresponding to the first serial number but do not write it to the corresponding data table, read the input data for the message middleware corresponding to the first serial number and write it to the message middleware corresponding to the second serial number, and in response to successful writing, update the second serial number to the first serial number, and then update the transaction submission progress, and continue to execute the task instance based on the updated transaction submission progress; if the preset data consistency mode is processing from the beginning, determine the processing method for the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, read the input data for the message middleware corresponding to the second serial number and the serial numbers before the second serial number but do not write it to the corresponding message middleware, read the input data for the data table corresponding to the first serial number and the serial numbers before the first serial number but do not write it to the corresponding data table, read the input data for the message middleware corresponding to the first serial number and write it to the message middleware corresponding to the second serial number, and in response to successful writing, update the second serial number to the first serial number, and then update the transaction submission progress, and continue to execute the task instance based on the updated transaction submission progress.

[0015] Optionally, the output unit is further configured to: in response to the task instance corresponding to the task thread not having an exception and the current batch being the last batch, obtain the corresponding transaction submission result data and output it, and exit the task thread.

[0016] Optionally, the acquisition unit is further configured to: in response to the task instance corresponding to the task thread not having an exception and the current batch not being the last batch, if no control instruction is received, continue to obtain the business data of the next batch as the business data of the current batch and continue data processing.

[0017] Optionally, the data processing device further includes a control instruction receiving unit, configured to: in response to the task instance corresponding to the task thread not having an exception, exit the task thread when a control instruction is received.

[0018] In addition, the present application also provides a data processing electronic device, including: one or more processors; a storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the data processing method as described above.

[0019] In addition, the present application also provides a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, it implements the data processing method as described above.

[0020] To achieve the above object, according to another aspect of the embodiments of the present application, a computer program product is provided.

[0021] A computer program product according to an embodiment of the present application includes a computer program, and when the program is executed by a processor, it implements the data processing method provided by the embodiment of the present application.

[0022] One embodiment of the above invention has the following advantages or beneficial effects: In the present application, after the business logic is executed, the business data of the current batch is obtained, and a multi-level cache memory table is called to obtain the auxiliary data corresponding to the business data; based on the business data and the auxiliary data, the transaction to be submitted corresponding to the business data and the transaction type of the transaction are determined; the data source corresponding to the transaction is determined, and a task thread is started to open and submit the transaction based on the data source and the transaction type; in response to an exception occurring in the task instance corresponding to the task thread, the corresponding transaction submission progress is obtained, the breakpoint transaction is determined according to the transaction submission progress, and the transaction submission automatic retry is executed based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode; in response to the failure of the transaction submission automatic retry, an exception message is generated and output, and the task thread is exited. Thus, by determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction to be submitted based on the business data and the auxiliary data, the complexity of development and operation and maintenance is reduced. By introducing a multi-level cache memory table, the data query efficiency is improved during the data processing process. By executing the transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode when an exception occurs in the task instance, the consistency of data processing after interruption recovery is ensured.

[0023] The further effects of the above non-conventional optional manners will be described in combination with specific embodiments below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings are used to better understand the present application and do not constitute an improper limitation to the present application. Among them: Figure 1 is a schematic diagram of the main process of a data processing method according to an embodiment of the present application; Figure 2It is a schematic diagram of the main process of the data processing method according to an embodiment of the present application; Figure 3 It is a schematic diagram of stream-batch integrated data processing of the data processing method according to an embodiment of the present application; Figure 4 It is a schematic diagram of the memory representation of the data processing method according to an embodiment of the present application; Figure 5 It is a schematic diagram of breakpoint handling of the data processing method according to an embodiment of the present application; Figure 6 It is a schematic diagram of head processing of the data processing method according to an embodiment of the present application; Figure 7 It is a schematic diagram of the main units of the data processing device according to an embodiment of the present application; Figure 8 It is an exemplary system architecture diagram to which the embodiments of the present application can be applied; Figure 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing the embodiments of the present application. Detailed implementation manners

[0025] The following describes exemplary embodiments of the present application with reference to the accompanying drawings. Various details of the embodiments of the present application are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted below. It should be noted that in the technical solution of the present application, the acquisition, transmission, storage, use, processing, etc. of data all comply with the relevant regulations of national laws and regulations. It should be noted that in the embodiments of the present application, some industry-existing solutions such as certain software, components, models, etc. may be mentioned, and they should be considered exemplary. The purpose is only to illustrate the feasibility in the implementation of the technical solution of the present application, but it does not mean that the applicant has already or necessarily used this solution. In the technical solution of the present application, in terms of the collection, acquisition, update, analysis, processing, use, transmission, storage, etc. of the user's personal information, it complies with the provisions of relevant laws and regulations, is used for legal and reasonable purposes, does not violate public order and good customs, is not shared, leaked, or sold outside these legal uses, and is subject to the supervision and management of the regulatory authorities. Necessary measures are taken for the user's personal information to prevent illegal access to such user personal information data, safeguard the security of the user's personal information, network security, and national security, ensure that the personnel with the right to access personal information data comply with the provisions of relevant laws and regulations, and ensure the security of the user's personal information. Once these user personal information data are no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0026] When in use, including in certain related application programs, user privacy is protected by de-identifying the data. For example, when in use, the specific identifier is removed, the amount or specificity of the stored data is controlled, how the data is stored is controlled, and / or other methods are used to de-identify.

[0027] Figure 1 is a schematic diagram of the main process of the data processing method according to an embodiment of the present application, as Figure 1 shown, the data processing method mainly includes the following steps S101 - step S105.

[0028] Step S101, in response to the completion of the execution of the business logic, obtain the business data of the current batch, and call the multi-level cache memory table to obtain the auxiliary data corresponding to the business data.

[0029] In this embodiment, the execution subject of the data processing method (for example, it can be a server) can detect in real time whether the business logic has been executed through a wired connection or a wireless connection. The business logic can be, for example, the business logic of a business application (such as the business application bound to devices such as sensors and scanners). When an end flag is obtained, it can be determined that the business logic corresponding to the end flag has been executed. When the business logic is executed, the business data output by the corresponding business application can be processed in batches. Specifically, the business data of the current batch can be obtained, and the corresponding auxiliary data can be queried and obtained from the multi-level cache memory table based on the business data of the current batch. The auxiliary data can include data that needs to be processed in real time and data that does not need to be processed in real time. For example, the business data of the current batch can be sensor data that needs to be processed in real time, and the auxiliary data corresponding to the business data of the current batch obtained can include sensor operation log data that does not need to be processed in real time and real-time decision-making data that needs to be processed in real time.

[0030] Step S102: Based on the business data and the auxiliary data, determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction.

[0031] Label the business data of the current batch and the auxiliary data corresponding to the business data of the current batch according to whether the data needs to be processed in real time, and cluster the business data of the current batch and the auxiliary data corresponding to the business data of the current batch according to the labeled tags to obtain each clustering cluster.

[0032] For example, each clustering cluster includes clustering cluster 1 and clustering cluster 2. Among them, clustering cluster 1 (label: needs to be processed in real time): sensor data + real-time decision-making data; clustering cluster 2 (label: does not need to be processed in real time): sensor operation log data.

[0033] Furthermore, the transaction 1 to be submitted is a transaction that writes the sensor data + real-time decision-making data in clustering cluster 1 (label: needs to be processed in real time) into the message middleware kafka, and the transaction type is a kafka transaction.

[0034] The transaction 2 to be submitted is a transaction that writes the sensor operation log data in clustering cluster 2 (label: does not need to be processed in real time) into the data table, and the transaction type is a database transaction.

[0035] Step S103: Determine the data source corresponding to the transaction, and start a task thread to open and submit the transaction based on the data source and the transaction type.

[0036] If the transaction type of a transaction is a database transaction, the data source corresponding to this transaction (e.g., transaction 2) is the target data table in the target relational database. The target relational database (e.g., relational database 2) and the target data table in this target relational database (e.g., data table 2 in relational database 2) can be determined based on the data in the clustering cluster corresponding to this transaction (e.g., clustering cluster 2 in step S102) (e.g., the operation log data of sensors in clustering cluster 2 (label: does not require real-time processing)). Start a task thread to start and commit a transaction based on the data source and transaction type, that is, start a task thread to accurately write the data in the clustering cluster corresponding to this transaction (e.g., the operation log data of sensors in clustering cluster 2 (label: does not require real-time processing)) into the target data table in the target relational database (e.g., relational database 2) (e.g., data table 2 in relational database 2).

[0037] If the transaction type is a kafka transaction, the data source corresponding to this transaction (e.g., transaction 1) is the target message middleware. The target message middleware (e.g., message middleware 1) can be determined based on the data in the clustering cluster corresponding to this transaction (e.g., clustering cluster 1 in step S102) (e.g., sensor data + real-time decision data in clustering cluster 1 (label: requires real-time processing)). Start a task thread to start and commit a transaction based on the data source and transaction type, that is, start a task thread to accurately write the data in the clustering cluster corresponding to this transaction (e.g., sensor data + real-time decision data in clustering cluster 1 (label: requires real-time processing)) into the target message middleware (e.g., message middleware 1).

[0038] Step S104, in response to an exception occurring in the task instance corresponding to the task thread, obtain the corresponding transaction submission progress, determine the breakpoint transaction based on the transaction submission progress, and perform automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode.

[0039] An exception occurring in the task instance corresponding to the task thread can be an exception in starting or submitting a transaction in this task instance. When an exception occurs in the task instance corresponding to the started task thread, the transaction submission progress (i.e., task progress) of the transaction in this task instance can be obtained. For example, which transactions have been submitted and which transactions have not been submitted, so as to determine the breakpoint transaction (i.e., the last submitted transaction corresponding to the transaction submission progress, e.g., Figure 5 the nth submitted transaction shown in

[0040] The preset data consistency mode can be repeatable execution, breakpoint processing, or processing from the beginning. After determining the breakpoint transaction, based on the breakpoint transaction (for example, based on the sequence number of the input data corresponding to the nth committed transaction), the data source corresponding to the breakpoint transaction (for example, the target data table in the target relational database, or the target message middleware), and the transaction submission progress, execute the automatic retry logic for transaction submission corresponding to the preset data consistency mode (for example, repeatable execution) (for example, clear all previous outputs or filter the processed inputs, and re-execute the task instance).

[0041] Step S105, in response to the failure of automatic retry of transaction submission, generate an exception message, output it, and exit the task thread.

[0042] If an exception still occurs in the transaction submitted through automatic retry, it is determined that the automatic retry of transaction submission fails, an exception message is generated and output, and the task thread is exited.

[0043] In some embodiments, the data processing method further includes: in response to the success of automatic retry of transaction submission, determine whether the current batch is the last batch. If the current batch is the last batch, exit the task thread; if the current batch is not the last batch, continue to determine whether a control instruction is received. If a control instruction (which can be used to control the exit of the task thread) is received, exit the task thread; if no control instruction is received, continue to obtain the business data of the next batch as the business data of the current batch and continue to perform data processing (according to the processing logic of the business data of the current batch) until the current batch is the last batch, and then exit the task thread.

[0044] In some embodiments, the data processing method further includes: in response to the task instance corresponding to the task thread not having an exception and the current batch being the last batch, obtain the corresponding transaction submission result data (for example, obtain the data in the target data table in the target relational database after successfully writing the data that does not need to be processed in real time corresponding to the transaction, or obtain the data in the target message middleware after successfully writing the data that needs to be processed in real time corresponding to the transaction) and output it, and exit the task thread; in response to the task instance corresponding to the task thread not having an exception and the current batch not being the last batch, if no control instruction is received, continue to obtain the business data of the next batch as the business data of the current batch and continue to perform data processing (according to the processing logic of the business data of the current batch); in response to the task instance corresponding to the task thread not having an exception, exit the task thread when a control instruction (which can be used to control the exit of the task thread) is received.

[0045] In this embodiment, upon completion of the execution of the business logic, the business data of the current batch is obtained, and a multi-level cache memory table is called to obtain the auxiliary data corresponding to the business data; based on the business data and the auxiliary data, the transaction to be committed corresponding to the business data and the transaction type of the transaction are determined; the data source corresponding to the transaction is determined, and a task thread is started to open and commit the transaction based on the data source and the transaction type; in response to an exception occurring in the task instance corresponding to the task thread, the corresponding transaction commit progress is obtained, the breakpoint transaction is determined according to the transaction commit progress, and the transaction commit automatic retry is executed based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction commit progress, and the preset data consistency mode; in response to the failure of the transaction commit automatic retry, an exception message is generated and output, and the task thread is exited. Thus, by determining the transaction to be committed corresponding to the business data and the transaction type of the transaction to be committed based on the business data and the auxiliary data, the complexity of development and operation and maintenance is reduced. By introducing a multi-level cache memory table, the data query efficiency is improved during the data processing process. By executing the transaction commit automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction commit progress, and the preset data consistency mode when an exception occurs in the task instance, the consistency of data processing after interruption recovery is ensured.

[0046] Figure 2 is a schematic diagram of the main process of the data processing method according to an embodiment of the present application. As Figure 2 shown, the data processing method mainly includes the following steps S201-step S207.

[0047] Step S201, upon completion of the execution of the business logic, obtain the business data of the current batch, and call a multi-level cache memory table to obtain the auxiliary data corresponding to the business data.

[0048] It is possible to detect in real time whether the business logic has been executed. The business logic can be, for example, the business logic of various business applications. When the end flag is obtained, it can be determined that the business logic corresponding to the end flag has been executed. When the business logic is executed, the business data output by the corresponding business application can be processed in batches. Specifically, the business data of the current batch can be obtained, and the corresponding auxiliary data can be queried and obtained from the multi-level cache memory table based on the business data of the current batch. The auxiliary data can include data that needs to be processed in real time and can also include data that does not need to be processed in real time.

[0049] Step S202, determine the business data type corresponding to the business data and the auxiliary data type corresponding to the auxiliary data.

[0050] For example, if the business data of the current batch is sensor data that needs to be processed in real time, the business data type corresponding to the business data of the current batch is data that needs to be processed in real time; if the auxiliary data corresponding to the business data of the current batch includes sensor operation log data that does not need to be processed in real time and real-time decision-making data that needs to be processed in real time, the business data type corresponding to the auxiliary data is data that does not need to be processed in real time + data that needs to be processed in real time.

[0051] Step S203: Determine the corresponding data table and message middleware according to the business data type and the auxiliary data type, and classify the business data and the auxiliary data according to the data table and the message middleware, so as to obtain the input data corresponding to the data table and the input data corresponding to the message middleware respectively.

[0052] For example, according to the business data type (for example, data that needs to be processed in real time) and the auxiliary data type (for example, data that does not need to be processed in real time + data that needs to be processed in real time), determine the corresponding data table (for example, the target data table in the target relational database corresponding to the sensor operation log data that does not need to be processed in the auxiliary data) and the message middleware (for example, the target message middleware corresponding to the sensor data + real-time decision-making data that need to be processed in the business data and the auxiliary data), and classify the business data and the auxiliary data according to the data table (corresponding to the data that does not need to be processed in real time) and the message middleware (corresponding to the data that needs to be processed in real time), that is, divide the business data and the auxiliary data into data corresponding to the data table (corresponding to the data that does not need to be processed in real time) and the message middleware (corresponding to the data that needs to be processed in real time) respectively, so as to obtain the input data corresponding to the data table and the input data corresponding to the message middleware respectively.

[0053] Step S204: Based on the data table, the message middleware, and their respective corresponding input data, determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction.

[0054] For example, the message middleware, such as Message Middleware 1; the data table, such as Data Table 2 in Relational Database 2.

[0055] Input data corresponding to Message Middleware 1: For example, it can be sensor data + real-time decision-making data.

[0056] Input data corresponding to Data Table 2 in Relational Database 2: For example, it can be the operation log data of the sensor.

[0057] Transaction to be submitted 1: For example, it can be a transaction to write the sensor data + real-time decision-making data as input data into Message Middleware 1, and the transaction type is a Kafka transaction.

[0058] Transaction 2 to be submitted: For example, it can be a transaction of writing the operation log data of a sensor as input data into a data table in a relational database 2, and the transaction type is a database transaction.

[0059] Step S205: Determine the data source corresponding to the transaction, and start a task thread to open and submit the transaction based on the data source and the transaction type.

[0060] If the transaction type is a Kafka transaction, the data source corresponding to this transaction (e.g., transaction 1) is the target message middleware. The target message middleware (e.g., message middleware 1) can be determined according to the input data (e.g., sensor data + real-time decision data) corresponding to this transaction (e.g., transaction 1). Start a task thread to open and submit the transaction based on the data source and the transaction type, that is, start a task thread to accurately write the input data (e.g., sensor data + real-time decision data) corresponding to this transaction (e.g., transaction 1) into the target message middleware (e.g., message middleware 1).

[0061] If the transaction type of the transaction is a database transaction, the data source corresponding to this transaction (e.g., transaction 2) is the target data table in the target relational database. The target relational database (e.g., relational database 2) and the target data table in the target relational database (e.g., data table 2 in relational database 2) can be determined according to the input data (e.g., operation log data of the sensor) corresponding to this transaction (e.g., transaction 2). Start a task thread to open and submit the transaction based on the data source and the transaction type, that is, start a task thread to accurately write the input data (e.g., operation log data of the sensor) corresponding to this transaction (e.g., transaction 2) into the target data table (e.g., data table 2 in relational database 2) in the target relational database (e.g., relational database 2).

[0062] Step S206: In response to an exception occurring in the task instance corresponding to the task thread, obtain the corresponding transaction submission progress, determine the breakpoint transaction according to the transaction submission progress, and perform automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode.

[0063] Specifically, determining the breakpoint transaction according to the transaction submission progress includes: determining the transaction identifier at the breakpoint position according to the transaction submission progress, and determining the breakpoint transaction according to the transaction identifier; and performing automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode includes: determining the serial number of the input data corresponding to the breakpoint transaction; according to the serial number, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode, determining the processing method for the input data corresponding to each transaction corresponding to the task instance; and performing automatic retry of transaction submission based on the processing method.

[0064] Exemplarily, determine the transaction identifier (such as n) at the breakpoint position according to the transaction submission progress, and determine the breakpoint transaction according to the transaction identifier (for example, the nth committed transaction); determine the serial numbers of the input data corresponding to the breakpoint transaction (for example, 103, 104); according to the serial numbers (such as 103, 104), the data source corresponding to the breakpoint transaction (for example, the data table 2 in message middleware 1 and relational database 2), the transaction submission progress (for example, the last committed transaction is the nth committed transaction), and the preset data consistency mode (such as repeatable execution, processing from the breakpoint, or processing from the beginning), determine the processing method for the input data corresponding to each transaction of the task instance (for example, clear all previous outputs or filter the processed inputs and re-execute the task instance; or, there is no need to clear the previous output, and when high-availability switching occurs, continue processing from the previous interruption position; or, there is no need to clear the previous output, and when high-availability switching occurs, the task instance starts processing from the first input data, but will intercept the outputs corresponding to the position before the previous interruption); accurately execute the automatic retry of transaction submission based on the determined processing method.

[0065] Specifically, according to the serial number, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode, determine the processing method for the input data corresponding to each transaction of the task instance, including: determining the data source corresponding to the serial number according to the data source corresponding to the breakpoint transaction and the preset transaction submission priority; if the preset data consistency mode is repeatable execution, determine the processing method for the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, clear all the input data corresponding to each transaction of the task instance that was historically written in the data source corresponding to the serial number, and then re-execute the task instance; determining the first serial number corresponding to the data table and the second serial number corresponding to the message middleware in the serial number, where the first serial number is greater than the second serial number; and if the preset data consistency mode is processing from the breakpoint, determine the processing method for the input data corresponding to each transaction of the task instance as: read the input data for the data table corresponding to the first serial number but do not write it to the corresponding data table, read the input data for the message middleware corresponding to the first serial number and write it to the message middleware corresponding to the second serial number, and in response to successful writing, update the second serial number to the first serial number, and then update the transaction submission progress, and continue to execute the task instance based on the updated transaction submission progress; if the preset data consistency mode is processing from the beginning, determine the processing method for the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, read the input data for the message middleware corresponding to the second serial number and the serial numbers before the second serial number but do not write it to the corresponding message middleware, read the input data for the data table corresponding to the first serial number and the serial numbers before the first serial number but do not write it to the corresponding data table, read the input data for the message middleware corresponding to the first serial number and write it to the message middleware corresponding to the second serial number, and in response to successful writing, update the second serial number to the first serial number, and then update the transaction submission progress, and continue to execute the task instance based on the updated transaction submission progress.

[0066] For example, according to the data source corresponding to the breakpoint transaction (for example, the target data table in the target message middleware and the target relational database, for example, it can be the data table 2 in message middleware 1 and relational database 2) and the preset transaction submission priority (for example, when writing data, the priority of the target data table in the target relational database is higher than that of the target message middleware), determine the data source corresponding to the serial number (for example, 103, 104) (for example, the data source corresponding to serial number 103 is the target message middleware, for example, it can be message middleware 1, and the data source corresponding to serial number 104 is the target data table in the target relational database, for example, it can be the data table 2 in relational database 2).

[0067] If the preset data consistency mode is repeatable execution, determine the processing method for the input data corresponding to each transaction of the task instance as follows: Based on the transaction submission progress (for example, the last committed transaction is the nth committed transaction), clear all the input data corresponding to each transaction of the task instance that was historically written in the data source corresponding to the sequence number (for example, the sequence numbers 103 and 104 corresponding to the input data of the nth committed transaction), and then re-execute the task instance. For example, when an exception occurs in the task instance (that is, an exception occurs in the transaction corresponding to the task instance), clear all the outputs of the previous data source or filter the processed inputs, and re-execute the task instance. This mode has the following characteristics: breakpoint transactions are not enabled during the execution of the task instance; an alarm is sent when the task instance exits abnormally; when manually retrying to start the task instance, first clean the output table or filter the processed inputs; after the task instance is restarted, start processing from the first input data.

[0068] Determine the first sequence number (e.g., 104) corresponding to the data table and the second sequence number (e.g., 103) corresponding to the message middleware in the sequence numbers of the input data corresponding to the breakpoint transaction according to the preset transaction submission priority (e.g., when writing data, the priority of the target data table in the target relational database is higher than that of the target message middleware), where the first sequence number (e.g., 104) is greater than the second sequence number (e.g., 103); and if the preset data consistency mode is breakpoint processing, determine the processing method for the input data corresponding to each transaction of the task instance as: read the input data (e.g., the operation log data of the sensor) for the data table corresponding to the first sequence number (e.g., 104) but do not write it to the corresponding data table (e.g., the data table 2 in the relational database 2), read the input data (e.g., sensor data + real-time decision data) for the message middleware corresponding to the first sequence number (e.g., 104) and write it to the message middleware (e.g., message middleware 1) corresponding to the second sequence number (e.g., 103), in response to the successful writing, update the second sequence number (e.g., 103) corresponding to the corresponding message middleware (e.g., message middleware 1) to the first sequence number (e.g., 104), indicating that the input data (e.g., sensor data + real-time decision data) corresponding to the first sequence number (e.g., 104) for the message middleware has been successfully written in the message middleware (e.g., message middleware 1), and then update the transaction submission progress to the successful submission of the breakpoint transaction (e.g., the submission of the nth submitted transaction is successful), and continue to execute the task instance based on the updated transaction submission progress, that is, continue to normally execute the submission of the next transaction. For example, when an exception occurs in the task instance (i.e., an exception occurs in the transaction corresponding to the task instance), there is no need to clear the previous output. After high-availability switching, continue to process from the position where the previous interruption occurred (e.g., Figure 5 The interception point corresponding to the nth submitted transaction before the task interruption shown: 104), and ensure that the processed data and results are complete and non-redundant through the Distributed Consensus Framework (DCF framework). This mode has the following characteristics: enable breakpoint transactions during the execution of the task instance and record the breakpoint transactions; send an alarm when the task instance exits abnormally; when manually retrying to start the task instance, read the position of the breakpoint transaction (e.g., interception point: 104); after the task instance is restarted, start processing from the breakpoint transaction position (e.g., business processing: 104). As Figure 5As shown, 1, 2, 3, 4, 5, …, 103, 104, 105, 106, 107, 108, 109 represent 109 pairs of input data. Among them, each pair of input data includes an input data that does not require real-time processing for the data table and an input data that requires real-time processing for the message middleware kafka. If an input data that does not require real-time processing for the data table in each pair of input data is successfully written into the corresponding data table and an input data that requires real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are merged and displayed as one number. For example, if an input data (i.e., the 103rd) that does not require real-time processing for the data table in the 103rd pair of input data is successfully written into the corresponding data table and an input data (i.e., the 103rd) that requires real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are merged and displayed as one number (i.e., 103); if an input data that does not require real-time processing for the data table in each pair of input data is successfully written into the corresponding data table, but an input data that requires real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately as numbers. For example, if an input data (i.e., the 104th) that does not require real-time processing for the data table in the 104th pair of input data is successfully written into the corresponding data table, but an input data (i.e., the 104th) that requires real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately as numbers (e.g., 103, 104). When writing data, the priority of the data table is higher than that of the message middleware kafka. Before the task is interrupted, when an exception occurs in the nth committed transaction, the positions of the serial numbers of the input data corresponding to the breakpoint transaction are 103 (representing the 103rd input data that has been written into kafka) and 104 (representing the 104th input data that has been written into the data table), meaning that the 104th input data that does not require real-time processing (such as the operation log data of the sensor) has been successfully written into the data table, but an exception occurs when writing the corresponding 104th input data that requires real-time processing (such as sensor data and real-time decision data) of the 104th input data that does not require real-time processing (such as the operation log data of the sensor) into the message middleware kafka (i.e., it is not successfully written into the message middleware kafka).Based on the interception point: 104, for business processing: 104, execute automatic retry, that is, start processing from the 104th input data that does not require real-time processing (such as the operation log data of sensors). Repeatedly read the 104th input data that does not require real-time processing (such as the operation log data of sensors) and intercept the transaction, that is, intercept the output corresponding to the 104th input data that does not require real-time processing (such as the operation log data of sensors) (i.e., do not write to the corresponding data table). Furthermore, read the 104th input data that requires real-time processing (such as sensor data and real-time decision data) corresponding to the 104th input data that does not require real-time processing (such as the operation log data of sensors) and write it to the corresponding message middleware kafka. After successfully writing to the message middleware kafka, resume the task. After the task is resumed, start from the 105th input data, and the business program enters the normal execution mode (that is, no longer intercept transactions, and both reading and writing of input data are performed).

[0069] If the preset data consistency mode is processing from the beginning, the processing method for the input data corresponding to each transaction corresponding to the task instance is determined as follows: Based on the transaction submission progress, read the second sequence number (for example, 103) and the input data corresponding to the sequence numbers before the second sequence number (for example, 103) for the message middleware (for example, instant messaging data, streaming media data, etc., data that needs to be processed in real time), but do not write it to the corresponding message middleware. Read the first sequence number (for example, 104) and the input data corresponding to the sequence numbers before the first sequence number (for example, 104) for the data table (for example, historical transaction data, customer access data, etc., data that does not need to be processed in real time), but do not write it to the corresponding data table. Read the input data corresponding to the first sequence number (for example, 104) for the message middleware (for example, sensor data + real-time decision data) and write it to the message middleware corresponding to the second sequence number (for example, 103) (for example, Message Middleware 1). In response to successful writing, update the second sequence number (for example, 103) corresponding to the corresponding message middleware (for example, Message Middleware 1) to the first sequence number (for example, 104), indicating that the input data corresponding to the first sequence number (for example, 104) for the message middleware (for example, sensor data + real-time decision data) has been successfully written to the message middleware (for example, Message Middleware 1). Furthermore, update the transaction submission progress to the breakpoint transaction submission success (for example, the nth committed transaction submission is successful), and continue to execute the task instance based on the updated transaction submission progress, that is, continue to normally submit the next transaction. For example, when an exception occurs in the task instance (that is, an exception occurs in the transaction corresponding to the task instance), there is no need to clean the previous output. After high-availability switching, the task instance starts processing from the first input data, but the Distributed Consensus Framework (DCF framework) will intercept the output corresponding to the position before the previous interruption (for example, Figure 6 the interception point corresponding to the nth committed transaction before the task interruption shown: 104). This mode has the following characteristics: Enable breakpoint transactions during the execution of the task instance, and record the sequence numbers of the input data corresponding to the breakpoint transactions (for example, the nth committed transaction) for the data table and the message middleware kafka respectively (for example, for message middleware kafka: 103, for data table: 104); Send an alarm when the task instance exits abnormally; When manually retrying to start the task instance, read the position of the breakpoint transaction (for example, the interception point 104); After the task instance is restarted, according to the business processing: 1, perform automatic retry, that is, start processing from the first input data and intercept the transaction, that is, intercept the output before the breakpoint transaction and normally submit the transaction after the breakpoint transaction. As Figure 6As shown, 1, 2, 3, 4, 5, …, 103, 104, 105, 106, 107, 108, 109 represent 109 pairs of input data. Among them, each pair of input data includes an input data that does not require real-time processing for the data table and an input data that requires real-time processing for the message middleware kafka. If an input data that does not require real-time processing for the data table in each pair of input data is successfully written into the corresponding data table and an input data that requires real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are combined and displayed as one number. For example, if an input data (i.e., the 103rd) that does not require real-time processing for the data table in the 103rd pair of input data is successfully written into the corresponding data table and an input data (i.e., the 103rd) that requires real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are combined and displayed as one number (i.e., 103); if an input data that does not require real-time processing for the data table in each pair of input data is successfully written into the corresponding data table, but an input data that requires real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately. For example, if an input data (i.e., the 104th) that does not require real-time processing for the data table in the 104th pair of input data is successfully written into the corresponding data table, but an input data (i.e., the 104th) that requires real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately (e.g., 103, 104). When writing data, the priority of the data table is higher than that of the message middleware kafka. Before the task is interrupted, the positions of the sequence numbers of the input data corresponding to the breakpoint transaction when the task fails are 103 (corresponding to kafka) and 104 (corresponding to the database). According to the interception point: 104, business processing: 1, execute automatic retry, that is, start processing from the 1st input data and intercept the transaction. That is, for the message middleware kafka, intercept the output before (including the 103rd input data) the 103rd input data that requires real-time processing, and for the data table, intercept the output before (including the 104th input data) the 104th input data that does not require real-time processing. Then read the 104th input data that requires real-time processing (such as sensor data and real-time decision data) corresponding to the 104th input data that does not require real-time processing (such as the operation log data of the sensor) and write it into the corresponding message middleware kafka. After successfully writing into the message middleware kafka, resume the task. After the task is resumed, start from the 105th input data, and the business program enters the normal execution mode (that is, no longer intercept the transaction, and both the reading and writing of input data are performed).

[0070] Step S207, in response to the automatic retry of the transaction submission failing, generate an exception message and output it, and exit the task thread.

[0071] If an exception still occurs in the transaction submitted by the automatic retry, it is determined that the automatic retry of the transaction submission fails, an exception message is generated and output, and the task thread is exited.

[0072] Figure 3 It is a schematic diagram of stream-batch integrated data processing of the data processing method according to an embodiment of the present application. As Figure 3 shown, when the business logic is executed, obtain the business data of the current batch, and call the multi-level cache memory table as Figure 4 shown to obtain the auxiliary data corresponding to the business data. Based on the business data and the auxiliary data, determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction; determine the data source corresponding to the transaction, start a task thread to start and submit the transaction based on the data source and the transaction type; in response to an exception occurring in the task instance corresponding to the task thread, obtain the corresponding transaction submission progress, determine the breakpoint transaction according to the transaction submission progress, and execute the automatic retry of the transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode; in response to the automatic retry of the transaction submission failing, generate an exception message and output it, and exit the task thread. In response to the successful automatic retry of the transaction submission, determine whether the current batch is the last batch. If the current batch is the last batch, exit the task thread. If the current batch is not the last batch, continue to determine whether a control instruction is received. If a control instruction is received, exit the task thread. If no control instruction is received, continue to obtain the business data of the next batch as the business data of the current batch to continue data processing until the current batch is the last batch, and then exit the task thread. Thus, by determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction to be submitted based on the business data and the auxiliary data, the complexity of development and operation and maintenance is reduced. By introducing a multi-level cache memory table, the data query efficiency is improved during the data processing process. By executing the automatic retry of the transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode when an exception occurs in the task instance, the consistency of data processing after interruption recovery is ensured.

[0073] Figure 4 It is a schematic diagram of the memory of the data processing method according to an embodiment of the present application. In the embodiment of the present application, the memory table is the multi-level cache memory table. As Figure 4As shown in the figure, the multi-level cache memory table may include an interface layer, a first-level cache, and a second-level cache. Among them, the interface layer is used to provide an interface externally for data query. The hot data is stored in the first-level cache. The first-level cache is implemented by Guava and Caffeine. Among them, Guava is an open-source set of Java libraries developed by Google, which includes various data structures commonly used by Google, caches, collections that support primitive operations, concurrent utility classes, common annotations, event buses, tools for supporting internationalization, etc. Caffeine is a high-performance, near-optimal cache library based on Java 8. Caffeine provides an in-memory cache using an API inspired by Google Guava. The second-level cache stores all the data in the data tables in the relational database and all the data in the message middleware Kafka. The second-level cache is implemented by SQLite and IMTable. Among them, SQLite is a lightweight, self-contained, and serverless relational database management system (RDBMS), which is easy to create and requires less configuration. SQLite is an open-source, lightweight, fast, independent, and highly reliable SQL database engine written in the C language, and it provides a fully functional database solution. IMTable is the abbreviation of "In-Memory Table", which is a table technology that stores data in memory. By storing data in a table in memory, it enables fast access and processing. By introducing the multi-level cache memory table, the query of read-only data can be greatly optimized.

[0074] In the embodiment of the present application, when the task thread starts, the task thread context can be initialized, including breakpoint transactions, sharding information, business data sources, transaction managers, write component instances, etc. The read component is initialized by the task thread to read a batch of business data (that is, obtain the business data of the current batch), and process the business data. Specifically, the business data can be processed according to the process of pre-processing (beforeWindow) - processing (execute) - post-processing (afterWindow) to obtain the processed result data. For example, the processed result data may include business data (such as sensor data) and auxiliary data corresponding to the business data (such as the operation log data and real-time decision data recorded corresponding to the sensor data). After obtaining the processed result data, the task thread can call the task initialization method, and the Distributed Consensus Framework (DCF framework) can start and commit the transaction, and record the transaction commit progress (that is, the task progress). Then, the task thread determines whether the task is processed. If not, it continues to obtain the next batch of business data and continues to process it when no control instruction is received. If so, it calls the pre-exit processing logic (exit), and then exits the task thread.

[0075] In the embodiments of the present application, the management of transactions is as follows: Transaction takeover: All business data output by the business application is uniformly taken over by the Distributed Consensus Framework (DCF framework), including the start and commit of transactions, and supports custom data source configurations. The data source in the embodiments of the present application can be a data table in a relational database or the message middleware kafka. A delayed start mechanism can be adopted, that is, each transaction can be marked as started before the business logic is executed, but the data source connection is not actually obtained to start the transaction or the consumer starts the transaction. After the business logic is executed, the data source connection is obtained to start the transaction or the consumer starts the transaction and commits, and transaction timeouts caused by the business logic will not occur. Transaction commit interception: When the application enables the high availability of the framework, the transaction commit progress is recorded during transaction commit. At the same time, when repeating the execution, the committed data can be intercepted according to the transaction commit progress. Distributed transaction: The Distributed Consensus Framework (DCF framework) realizes cross-media transaction commit, and supports the joint commit of database transactions (that is, transactions corresponding to batch data, and batch data refers to data that does not need to be processed in real time) and kafka transactions (that is, transactions corresponding to streaming data, and streaming data refers to data that needs to be processed in real time). The commit process is to first commit the database transaction, and then commit the kafka transaction after success.

[0076] In the embodiments of the present application, data consistency means that the data in the data table and the message middleware kafka should correspond one by one. For example, if there are 109 pieces of data in the data table, there should also be 109 pieces of data in the message middleware kafka and they should all correspond one by one to the data in the data table. In terms of data consistency, the task progress is recorded according to the respective transaction commit situations in the data table and the message middleware kafka. It is necessary to persistently store status data such as the task progress (i.e., the transaction commit progress) during the data processing to ensure the consistency of data processing after interruption recovery. When the application has distributed transaction output, there will be two task progress, one representing the database transaction commit situation and the other representing the kafka transaction commit situation. The progress of both will not exceed 1 window.

[0077] Transaction exception handling: It refers to the handling of situations such as network jitter, transaction timeout, and data writing exceptions that occur during the start and commit of transactions. For these exception situations, the transaction manager can perform three automatic retries, namely: repeatable execution, resume from breakpoint, and restart from the beginning. Based on these three automatic retries, most of the exception situations that occur can be handled. If the automatic retries still cannot solve the exception situations that occur, the exception information can be output to the log and the current task thread can be exited.

[0078] The goal of data consistency is to ensure that when the task is restored, the data for business processing is complete and non-redundant by the Distributed Consensus Framework (DCF framework). The Distributed Consensus Framework (DCF framework) provides three data consistency solutions: repeatable execution, resume from breakpoint, and restart from scratch.

[0079] Specifically, for repeatable execution: when an exception occurs in the task instance (i.e., an exception occurs in the transaction corresponding to the task instance), all previous outputs can be cleared or the processed inputs can be filtered, and the task instance can be re-executed. This mode has the following characteristics: the breakpoint transaction is not enabled during the execution of the task instance; an alarm is sent when the task instance exits abnormally; when manually retrying to start the task instance, the output table is first cleared or the processed inputs are filtered; after the task instance is restarted, processing starts from the first input data.

[0080] Resume from breakpoint: when an exception occurs in the task instance (i.e., an exception occurs in the transaction corresponding to the task instance), there is no need to clear the previous output. After high-availability switching, continue processing from the previous interruption point (for example, Figure 5 the interception point corresponding to the nth committed transaction before the task interruption shown: 104), and the Distributed Consensus Framework (DCF framework) ensures that the processed data and results are complete and non-redundant. This mode has the following characteristics: the breakpoint transaction is enabled during the execution of the task instance, and the breakpoint transaction is recorded; an alarm is sent when the task instance exits abnormally; when manually retrying to start the task instance, the position of the breakpoint transaction is read (for example, the interception point: 104); after the task instance is restarted, processing starts from the breakpoint transaction position (for example, business processing: 104).

[0081] Figure 5 is a schematic diagram of resume from breakpoint processing of the data processing method according to an embodiment of the present application. As Figure 5As shown, 1, 2, 3, 4, 5, …, 103, 104, 105, 106, 107, 108, 109 represent 109 pairs of input data. Among them, each pair of input data includes an input data that does not require real-time processing for the data table and an input data that requires real-time processing for the message middleware kafka. If an input data that does not require real-time processing for the data table in each pair of input data is successfully written into the corresponding data table and an input data that requires real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are merged and displayed as one number. For example, if an input data (i.e., the 103rd) that does not require real-time processing for the data table in the 103rd pair of input data is successfully written into the corresponding data table and an input data (i.e., the 103rd) that requires real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are merged and displayed as one number (i.e., 103); if an input data that does not require real-time processing for the data table in each pair of input data is successfully written into the corresponding data table, but an input data that requires real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately as numbers. For example, if an input data (i.e., the 104th) that does not require real-time processing for the data table in the 104th pair of input data is successfully written into the corresponding data table, but an input data (i.e., the 104th) that requires real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately as numbers (e.g., 103, 104). When writing data, the priority of the data table is higher than that of the message middleware kafka. Before the task is interrupted, when an exception occurs in the nth committed transaction, the positions of the sequence numbers of the input data corresponding to the breakpoint transaction are 103 (representing the 103rd input data that has been written into kafka) and 104 (representing the 104th input data that has been written into the data table), which means that the 104th input data that does not require real-time processing (such as the operation log data of the sensor) has been successfully written into the data table, but an exception occurs (i.e., not successfully written into the message middleware kafka) when writing the corresponding 104th input data that requires real-time processing (such as sensor data and real-time decision data) of the 104th input data that does not require real-time processing (such as the operation log data of the sensor) into the message middleware kafka.

[0082] Based on interception point: 104, business processing: 104, automatic retry is performed, that is, processing starts from the 104th input data that does not require real-time processing (such as sensor operation log data), repeatedly reads the 104th input data that does not require real-time processing (such as sensor operation log data) and intercepts the transaction, that is, intercepts the output corresponding to the 104th input data that does not require real-time processing (such as sensor operation log data) (that is, does not write to the corresponding data table), and then reads the 104th input data that requires real-time processing (such as sensor data and real-time decision data) corresponding to the 104th input data that does not require real-time processing (such as sensor operation log data) and writes it to the corresponding message middleware Kafka. After successfully writing to the message middleware Kafka, the task is resumed. After the task is resumed, starting from the 105th input data, the business program enters normal execution mode (that is, no longer intercepts transactions, and both reading and writing input data are performed).

[0083] Processing from the beginning: When an exception occurs in a task instance (i.e., an exception occurs in the transaction corresponding to the task instance), there is no need to clean up the previous output. After the high availability switch, the task instance starts processing from the first input data, but the Distributed Consensus Framework (DCF framework) will intercept the last interruption position (for example, Figure 6 The output before the interception point (104) corresponding to the nth committed transaction before the task was interrupted is shown. This mode has the following features: breakpoint transactions are enabled during task instance execution, and the sequence number of the input data corresponding to the breakpoint transaction (e.g., the nth committed transaction) is recorded for the data table and message middleware Kafka (e.g., 103 for message middleware Kafka, 104 for data table). An alert is sent when the task instance exits abnormally. When manually retrying the task instance, the location of the breakpoint transaction (e.g., interception point 104) is read. After the task instance is restarted, automatic retry is performed based on business processing: 1, starting with the first input data and intercepting the transaction. This means that the output before the breakpoint transaction is intercepted, and the transaction after the breakpoint transaction is committed normally.

[0084] Figure 6 FIG. 1 is a schematic diagram of a data processing method according to an embodiment of the present application. Figure 6As shown, 1, 2, 3, 4, 5, …, 103, 104, 105, 106, 107, 108, 109 represent 109 pairs of input data. Among them, each pair of input data includes an input data that does not require real-time processing for the data table and an input data that requires real-time processing for the message middleware kafka. If an input data that does not require real-time processing for the data table in each pair of input data is successfully written into the corresponding data table and an input data that requires real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are combined into one number for display. For example, if an input data (i.e., the 103rd) that does not require real-time processing for the data table in the 103rd pair of input data is successfully written into the corresponding data table and an input data (i.e., the 103rd) that requires real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are combined into one number (i.e., 103) for display; if an input data that does not require real-time processing for the data table in each pair of input data is successfully written into the corresponding data table, but an input data that requires real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately. For example, if an input data (i.e., the 104th) that does not require real-time processing for the data table in the 104th pair of input data is successfully written into the corresponding data table, but an input data (i.e., the 104th) that requires real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately (e.g., 103, 104). When writing data, the priority of the data table is higher than that of the message middleware kafka. Before the task is interrupted, the positions of the serial numbers of the input data corresponding to the breakpoint transaction when the task fails are 103 (corresponding to kafka) and 104 (corresponding to the database).

[0085] According to the interception point: 104, business processing: 1, execute automatic retry, that is, start processing from the 1st input data and intercept the transaction. That is, for the message middleware kafka, intercept the output before (including the 103rd input data) the 103rd input data that requires real-time processing, and for the data table, intercept the output before (including the 104th input data) the 104th input data that does not require real-time processing. Then read the 104th input data that requires real-time processing (such as sensor data and real-time decision data) corresponding to the 104th input data that does not require real-time processing (such as the operation log data of the sensor) and write it into the corresponding message middleware kafka. After successfully writing into the message middleware kafka, resume the task. After the task is resumed, start from the 105th input data, and the business program enters the normal execution mode (that is, no longer intercept the transaction, and both the reading and writing of input data are carried out).

[0086] In applications such as data processing and data migration that require processing large amounts of data, stream data processing and batch data processing play crucial roles. Stream data processing is a technology that processes data in real time while the data is flowing in. It processes data by continuously receiving continuous data streams, and can achieve low-latency and high-throughput processing. Batch data processing is a technology that processes data in batches at regular intervals, usually executing data processing tasks in batches at predetermined time intervals. The embodiments of the present application provide a stream-batch integrated data processing method, which can achieve efficient stream-batch integrated data processing and unify the stream data processing method and the batch data processing method. A multi-level cache mechanism is provided to achieve the encapsulation of different data storage systems. A data consistency scheme based on data tables in a relational database is provided to reduce the complexity of the system.

[0087] Figure 7 is a schematic diagram of the main units of a data processing device according to an embodiment of the present application. As Figure 7 shown, the data processing device 700 includes an acquisition unit 701, a transaction determination unit 702, a transaction submission unit 703, an exception handling unit 704, and an output unit 705.

[0088] The acquisition unit 701 is configured to, in response to the completion of the execution of the business logic, acquire the business data of the current batch and call a multi-level cache memory table to acquire auxiliary data corresponding to the business data.

[0089] The transaction determination unit 702 is configured to determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction based on the business data and the auxiliary data.

[0090] The transaction submission unit 703 is configured to determine the data source corresponding to the transaction and start a task thread to open and submit the transaction based on the data source and the transaction type.

[0091] The exception handling unit 704 is configured to, in response to an exception occurring in the task instance corresponding to the task thread, acquire the corresponding transaction submission progress, determine a breakpoint transaction according to the transaction submission progress, and execute automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode.

[0092] The output unit 705 is configured to, in response to the failure of the automatic retry of transaction submission, generate and output an exception message and exit the task thread.

[0093] In some embodiments, the transaction determination unit 702 is further configured to: determine the business data type corresponding to the business data and the auxiliary data type corresponding to the auxiliary data; according to the business data type and the auxiliary data type, determine the corresponding data table and message middleware, and classify the business data and the auxiliary data according to the data table and the message middleware to respectively obtain the input data corresponding to the data table and the input data corresponding to the message middleware; based on the data table, the message middleware, and the respective corresponding input data, determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction.

[0094] In some embodiments, the exception handling unit 704 is further configured to: determine the transaction identifier at the breakpoint position according to the transaction submission progress, and determine the breakpoint transaction according to the transaction identifier; and the exception handling unit 704 is further configured to: determine the sequence number of the input data corresponding to the breakpoint transaction; according to the sequence number, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode, determine the processing method for the input data corresponding to each transaction corresponding to the task instance; perform an automatic retry of the transaction submission based on the processing method.

[0095] In some embodiments, the exception handling unit 704 is further configured to: determine the data source corresponding to the serial number according to the data source corresponding to the breakpoint transaction and the preset transaction submission priority; if the preset data consistency mode is repeatable execution, determine the processing method for the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, clear all the input data corresponding to each transaction of the task instance that has been historically written in the data source corresponding to the serial number, and then re-execute the task instance; determine the first serial number corresponding to the data table and the second serial number corresponding to the message middleware in the serial number, where the first serial number is greater than the second serial number; and if the preset data consistency mode is processing from the breakpoint, determine the processing method for the input data corresponding to each transaction of the task instance as: read the input data for the data table corresponding to the first serial number but do not write it to the corresponding data table, read the input data for the message middleware corresponding to the first serial number and write it to the message middleware corresponding to the second serial number, and in response to successful writing, update the second serial number to the first serial number, and then update the transaction submission progress, and continue to execute the task instance based on the updated transaction submission progress; if the preset data consistency mode is processing from the beginning, determine the processing method for the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, read the input data for the message middleware corresponding to the second serial number and the serial numbers before the second serial number but do not write it to the corresponding message middleware, read the input data for the data table corresponding to the first serial number and the serial numbers before the first serial number but do not write it to the corresponding data table, read the input data for the message middleware corresponding to the first serial number and write it to the message middleware corresponding to the second serial number, and in response to successful writing, update the second serial number to the first serial number, and then update the transaction submission progress, and continue to execute the task instance based on the updated transaction submission progress.

[0096] In some embodiments, the output unit 705 is further configured to: in response to the task instance corresponding to the task thread not having an exception and the current batch being the last batch, obtain the corresponding transaction submission result data and output it, and exit the task thread.

[0097] In some embodiments, the acquisition unit 701 is further configured to: in response to the task instance corresponding to the task thread not having an exception and the current batch not being the last batch, if no control instruction is received, continue to obtain the service data of the next batch as the service data of the current batch to continue data processing.

[0098] In some embodiments, the data processing device further includes Figure 7 a control instruction receiving unit not shown in the figure, which is configured to: in response to the task instance corresponding to the task thread not having an exception, exit the task thread when a control instruction is received.

[0099] It should be noted that the data processing method and data processing device of the present application have corresponding relationships in specific implementation contents, so repeated contents will not be described again.

[0100] Figure 8 An exemplary system architecture 800 is shown to which the data processing method or data processing device according to the embodiments of the present application can be applied.

[0101] As Figure 8 shown, the system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. The network 804 is used as a medium to provide a communication link between the terminal devices 801, 802, 803 and the server 805. The network 804 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0102] Users can use the terminal devices 801, 802, 803 to interact with the server 805 through the network 804 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 801, 802, 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0103] The terminal devices 801, 802, 803 may be various electronic devices having a data processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0104] The server 805 may be a server that provides various services, such as a background management server (only for example) that supports the service data output by the user terminal devices 801, 802, and 803. The background management server may, in response to the completion of the execution of the service logic, obtain the service data of the current batch, call the multi-level cache memory table to obtain the auxiliary data corresponding to the service data; based on the service data and the auxiliary data, determine the transaction to be submitted corresponding to the service data and the transaction type of the transaction; determine the data source corresponding to the transaction, start a task thread to open and submit the transaction based on the data source and the transaction type; in response to an exception occurring in the task instance corresponding to the task thread, obtain the corresponding transaction submission progress, determine the breakpoint transaction according to the transaction submission progress, and execute automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode; in response to the failure of the automatic retry of transaction submission, generate an exception message and output it, and exit the task thread. Thus, by determining the transaction to be submitted corresponding to the service data and the transaction type of the transaction to be submitted based on the service data and the auxiliary data, the complexity of development and operation and maintenance is reduced. By introducing a multi-level cache memory table, the data query efficiency is improved during the data processing process. By executing automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode when an exception occurs in the task instance, the consistency of data processing after interruption recovery is ensured.

[0105] It should be noted that the data processing method provided by the embodiments of the present application is generally executed by the server 805. Correspondingly, the data processing device is generally set in the server 805.

[0106] It should be understood that Figure 8 the numbers of the terminal devices, networks, and servers in

[0107] are merely illustrative. According to the implementation requirements, there may be any number of terminal devices, networks, and servers. Figure 9 is a schematic structural diagram of a computer system 900 of a terminal device suitable for implementing the embodiments of the present application. Figure 9 The terminal device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0108] As Figure 9As shown, computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or programs loaded into a random access memory (RAM) 903 from a storage section 908. In the RAM 903, various programs and data required for the operation of the computer system 900 are also stored. The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0109] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read therefrom can be installed into the storage section 908 as needed.

[0110] Specifically, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above functions defined in the system of the present application are executed.

[0111] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the above two. A computer-readable storage medium can, for example, include but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media can include but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, device, or component. In this application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, which can depend on the functions involved. It should also be noted that each block in a block diagram or flowchart, as well as the combination of blocks in a block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0113] The units involved in the embodiments of the present application can be implemented in software or in hardware. The described units can also be provided in a processor. For example, it can be described as: a processor includes an acquisition unit, a transaction determination unit, a transaction submission unit, an exception handling unit, and an output unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases.

[0114] As another aspect, the present application also provides a computer-readable medium. The computer-readable medium can be included in the device described in the above embodiments; it can also exist alone without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the device, the device, in response to the completion of the execution of the business logic, acquires the business data of the current batch, calls a multi-level cache memory table to acquire the auxiliary data corresponding to the business data; based on the business data and the auxiliary data, determines the transaction to be submitted corresponding to the business data and the transaction type of the transaction; determines the data source corresponding to the transaction, starts a task thread to open and submit the transaction based on the data source and the transaction type; in response to an exception occurring in the task instance corresponding to the task thread, acquires the corresponding transaction submission progress, determines the breakpoint transaction according to the transaction submission progress, and executes automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode; in response to the failure of the automatic retry of transaction submission, generates an exception message and outputs it, and exits the task thread.

[0115] The computer program product of the present application includes a computer program which, when executed by a processor, implements the data processing method in the embodiments of the present application.

[0116] According to the technical solution of the embodiments of the present application, by determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction to be submitted based on the business data and the auxiliary data, the complexity of development and operation and maintenance is reduced. By introducing a multi-level cache memory table, the data query efficiency is improved during the data processing process. By executing automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode when an exception occurs in the task instance, the consistency of data processing after interruption recovery is ensured.

[0117] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A data processing method, characterized in that, Including: Upon completion of the execution of the business logic, obtain the business data of the current batch, and call the multi-level cache memory table to obtain the auxiliary data corresponding to the business data; Based on the business data and the auxiliary data, determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction; Determine the data source corresponding to the transaction, and start a task thread to open and submit the transaction based on the data source and the transaction type; Upon an exception occurring in the task instance corresponding to the task thread, obtain the corresponding transaction submission progress, determine the breakpoint transaction based on the transaction submission progress, and perform automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode; Upon failure of the automatic retry of transaction submission, generate and output an exception message, and exit the task thread.

2. The method according to claim 1, wherein The determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction based on the business data and the auxiliary data includes: Determine the business data type corresponding to the business data and the auxiliary data type corresponding to the auxiliary data; According to the business data type and the auxiliary data type, determine the corresponding data table and message middleware, and classify the business data and the auxiliary data according to the data table and the message middleware to respectively obtain the input data corresponding to the data table and the input data corresponding to the message middleware; Based on the data table, the message middleware, and their respective corresponding input data, determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction.

3. The method according to claim 2, characterized in that The determining the breakpoint transaction according to the transaction submission progress includes: Determine the transaction identifier at the breakpoint position according to the transaction submission progress, and determine the breakpoint transaction according to the transaction identifier; and The performing automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode includes: Determine the sequence number of the input data corresponding to the breakpoint transaction; According to the sequence number, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode, determine the processing method for the input data corresponding to each transaction corresponding to the task instance; Perform automatic retry of transaction submission based on the processing method.

4. The method according to claim 3, wherein The determining the processing method for the input data corresponding to each transaction corresponding to the task instance according to the sequence number, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode includes: According to the data source corresponding to the breakpoint transaction and a preset transaction submission priority, determine the data source corresponding to the sequence number; If the preset data consistency mode is repeatable execution, determine the processing method for the input data corresponding to each transaction corresponding to the task instance as: based on the transaction submission progress, clear all the input data corresponding to each transaction corresponding to the task instance that was historically written in the data source corresponding to the sequence number, and then re-execute the task instance; Determine a first serial number corresponding to the data table and a second serial number corresponding to the message middleware among the serial numbers, where the first serial number is greater than the second serial number; and If the preset data consistency mode is to process from the breakpoint, determine the processing method for the input data corresponding to each transaction corresponding to the task instance as follows: read the input data for the data table corresponding to the first serial number but do not write it to the corresponding data table, read the input data for the message middleware corresponding to the first serial number and write it to the message middleware corresponding to the second serial number, and in response to successful writing, update the second serial number to the first serial number, and then update the transaction submission progress, and continue to execute the task instance based on the updated transaction submission progress; If the preset data consistency mode is to process from the beginning, determine the processing method for the input data corresponding to each transaction corresponding to the task instance as follows: based on the transaction submission progress, read the input data for the message middleware corresponding to the second serial number and the serial numbers before the second serial number but do not write it to the corresponding message middleware, read the input data for the data table corresponding to the first serial number and the serial numbers before the first serial number but do not write it to the corresponding data table, read the input data for the message middleware corresponding to the first serial number and write it to the message middleware corresponding to the second serial number, and in response to successful writing, update the second serial number to the first serial number, and then update the transaction submission progress, and continue to execute the task instance based on the updated transaction submission progress.

5. The method according to claim 1, wherein The method further includes: In response to the task instance corresponding to the task thread not having an exception and the current batch being the last batch, obtain the corresponding transaction submission result data and output it, and exit the task thread.

6. The method according to claim 1, wherein The method further includes: In response to the task instance corresponding to the task thread not having an exception and the current batch not being the last batch, if no control instruction is received, continue to obtain the business data of the next batch as the business data of the current batch to continue data processing.

7. The method according to claim 1, characterized in that, The method further includes: In response to the task instance corresponding to the task thread not having an exception, exit the task thread when a control instruction is received.

8. A data processing device, characterized in that, It includes: An acquisition unit configured to, in response to the completion of the execution of the business logic, acquire the business data of the current batch and call a multi-level cache memory table to acquire the auxiliary data corresponding to the business data; A transaction determination unit configured to determine the transactions to be submitted corresponding to the business data and the transaction types of the transactions based on the business data and the auxiliary data; A transaction submission unit configured to determine the data source corresponding to the transaction, start a task thread to open and submit the transaction based on the data source and the transaction type; An exception handling unit configured to, in response to the task instance corresponding to the task thread having an exception, obtain the corresponding transaction submission progress, determine the breakpoint transaction according to the transaction submission progress, and execute automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode; An output unit, configured to automatically retry a failure in response to a transaction commit, generate exception information and output it, and exit the task thread.

9. A data processing electronic device, characterized in that, Comprising: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method according to any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1-7.

Images