A data processing method and device, electronic equipment and computer readable medium

Abstract

The application discloses a data processing method and device, electronic equipment and computer readable medium, relates to the technical field of data processing, and the method comprises the following steps: in response to the completion of business logic execution, acquiring the current batch of business data, calling a multi-level cache memory table to acquire the auxiliary data corresponding to the business data; based on the business data and the auxiliary data, determining the transaction to be submitted and the transaction type; determining the data source corresponding to the transaction, starting the task thread to start and submit the transaction based on the data source and the transaction type; in response to an exception of the task instance, acquiring the transaction submission progress to determine the breakpoint transaction, and performing transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode; in response to the failure of the transaction submission automatic retry, generating and outputting exception information, and exiting the task thread. The complexity of development and operation and maintenance is reduced, the data query efficiency is improved, and the consistency of data processing is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data processing method and device, electronic equipment and computer readable medium. BACKGROUND

[0002] In applications such as data processing and data migration that need to process a large amount of data, stream data processing and batch data processing play a crucial role. With the continuous change of data volume and business requirements, a single stream processing or batch processing has been difficult to meet actual requirements. However, the implementation methods of the two kinds of data processing are different, which increases the complexity of development and operation. Moreover, in the process of data processing, data often needs to be queried from different data storage systems, the data query efficiency is low, and the consistency of data processing after interruption recovery cannot be guaranteed. SUMMARY

[0003] Therefore, the embodiments of the present application provide a data processing method and device, electronic equipment and computer readable medium, which can solve the problem that the implementation methods of the existing stream processing and batch processing are different, which increases the complexity of development and operation, and in the process of data processing, data often needs to be queried from different data storage systems, the data query efficiency is low, and the consistency of data processing after interruption recovery cannot be guaranteed.

[0004] To achieve the above object, according to an aspect of the embodiments of the present application, a data processing method is provided, comprising: in response to the completion of business logic execution, acquiring current batch business data, and calling a multi-level cache memory table to acquire auxiliary data corresponding to the business data; based on the business data and the auxiliary data, determining a transaction to be submitted corresponding to the business data and a transaction type of the transaction; determining a data source corresponding to the transaction, and starting a task thread to start and submit the transaction based on the data source and the transaction type; in response to an exception of a task instance corresponding to the task thread, acquiring a transaction submission progress corresponding thereto, determining a breakpoint transaction according to the transaction submission progress, and performing transaction submission automatic retry based on the breakpoint transaction, a data source corresponding to the breakpoint transaction, the transaction submission progress and a preset data consistency mode; in response to the failure of the transaction submission automatic retry, generating exception information and outputting, and exiting the task thread.

[0005] Optionally, based on the business data and the auxiliary data, the to-be-submitted transaction corresponding to the business data and the transaction type of the transaction are determined, including: determining a business data type corresponding to the business data and an auxiliary data type corresponding to the auxiliary data; determining a corresponding data table and a message middleware according to the business data type and the auxiliary data type, and classifying the business data and the auxiliary data according to the data table and the message middleware to obtain input data corresponding to the data table and input data corresponding to the message middleware respectively; and determining the to-be-submitted transaction corresponding to the business data and the transaction type of the transaction based on the data table, the message middleware and the input data corresponding thereto.

[0006] Optionally, the breakpoint transaction is determined according to the transaction submission progress, including: determining a transaction identifier at the breakpoint position according to the transaction submission progress, and determining the breakpoint transaction according to the transaction identifier; and performing transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode, including: determining a serial number of input data corresponding to the breakpoint transaction; determining a processing mode of input data corresponding to each transaction of the task instance according to the serial number, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode; and performing transaction submission automatic retry based on the processing mode.

[0007] Optionally, the processing manner of the input data corresponding to each transaction of the task instance is determined according to the sequence number, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode, including: determining the data source corresponding to the sequence number according to the data source corresponding to the breakpoint transaction and the preset transaction submission priority; if the preset data consistency mode is repeatable execution, determining the processing manner of the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, all the input data corresponding to each transaction of the task instance written in the data source corresponding to the sequence number are cleared, and then the task instance is executed again; determining the first sequence number corresponding to the data table and the second sequence number corresponding to the message middleware in the sequence number, wherein the first sequence number is greater than the second sequence number; and if the preset data consistency mode is breakpoint processing, determining the processing manner of the input data corresponding to each transaction of the task instance as: reading the input data for the data table corresponding to the first sequence number but not writing into the corresponding data table, reading the input data for the message middleware corresponding to the first sequence number and writing into the message middleware corresponding to the second sequence number, updating the second sequence number to the first sequence number in response to successful writing, and then updating the transaction submission progress, and continuing to execute the task instance based on the updated transaction submission progress; if the preset data consistency mode is head processing, determining the processing manner of the input data corresponding to each transaction of the task instance as: based on the transaction submission progress, reading the input data for the message middleware corresponding to the second sequence number and the sequence numbers before the second sequence number but not writing into the corresponding message middleware, reading the input data for the data table corresponding to the first sequence number and the sequence numbers before the first sequence number but not writing into the corresponding data table, reading the input data for the message middleware corresponding to the first sequence number and writing into the message middleware corresponding to the second sequence number, updating the second sequence number to the first sequence number in response to successful writing, and then updating the transaction submission progress, and continuing to execute the task instance based on the updated transaction submission progress.

[0008] Optionally, the data processing method further includes: in response to the task instance corresponding to the task thread being normal and the current batch being the last batch, obtaining the transaction submission result data corresponding thereto and outputting, and exiting the task thread.

[0009] Optionally, the data processing method further includes: in response to the task instance corresponding to the task thread being normal and the current batch not being the last batch, if no control instruction is received, continuing to obtain the business data of the next batch as the business data of the current batch to continue data processing.

[0010] Optionally, the data processing method further includes: in response to the task instance corresponding to the task thread being normal, exiting the task thread when the control instruction is received.

[0011] In addition, the application further provides a data processing apparatus, comprising: an acquisition unit configured to acquire the business data of the current batch in response to the completion of the execution of the business logic, and call a multi-level cache memory table to acquire the auxiliary data corresponding to the business data; a transaction determination unit configured to determine the transaction to be committed corresponding to the business data and the transaction type of the transaction based on the business data and the auxiliary data; a transaction submission unit configured to determine the data source corresponding to the transaction, start a task thread to start and submit the transaction based on the data source and the transaction type; an exception processing unit configured to acquire the transaction submission progress corresponding to the task instance in response to the occurrence of an exception in the task instance, determine the breakpoint transaction according to the transaction submission progress, and perform transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode; and an output unit configured to generate and output exception information and exit the task thread in response to the failure of the transaction submission automatic retry.

[0012] Optionally, the transaction determination unit is further configured to: determine the business data type corresponding to the business data and the auxiliary data type corresponding to the auxiliary data; determine the corresponding data table and message middleware according to the business data type and the auxiliary data type, and classify the business data and the auxiliary data according to the data table and the message middleware to obtain the input data corresponding to the data table and the input data corresponding to the message middleware respectively; and determine the transaction to be committed corresponding to the business data and the transaction type of the transaction based on the data table, the message middleware and the input data corresponding thereto.

[0013] Optionally, the exception processing unit is further configured to: determine the transaction identifier at the breakpoint position according to the transaction submission progress, and determine the breakpoint transaction according to the transaction identifier; and the exception processing unit is further configured to: determine the serial number of the input data corresponding to the breakpoint transaction; determine the processing mode of the input data corresponding to each transaction of the task instance according to the serial number, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode; and perform transaction submission automatic retry based on the processing mode.

[0014] Optionally, the abnormality processing unit is further configured to: determine the data source corresponding to the serial number according to the data source corresponding to the breakpoint transaction and the preset transaction commit priority; if the preset data consistency mode is repeatable execution, determine the processing manner of the input data corresponding to each transaction of the task instance as: based on the transaction commit progress, clear all the input data corresponding to each transaction of the task instance that is historically written in the data source corresponding to the serial number, and then re-execute the task instance; determine the first serial number corresponding to the data table and the second serial number corresponding to the message middleware in the serial number, wherein the first serial number is greater than the second serial number; and if the preset data consistency mode is breakpoint processing, determine the processing manner of the input data corresponding to each transaction of the task instance as: reading the input data for the data table corresponding to the first serial number but not writing into the corresponding data table, reading the input data for the message middleware corresponding to the first serial number and writing into the message middleware corresponding to the second serial number, in response to successful writing, updating the second serial number to the first serial number, and then updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress; if the preset data consistency mode is head processing, determine the processing manner of the input data corresponding to each transaction of the task instance as: based on the transaction commit progress, reading the input data for the message middleware corresponding to the second serial number and the serial numbers before the second serial number but not writing into the corresponding message middleware, reading the input data for the data table corresponding to the first serial number and the serial numbers before the first serial number but not writing into the corresponding data table, reading the input data for the message middleware corresponding to the first serial number and writing into the message middleware corresponding to the second serial number, in response to successful writing, updating the second serial number to the first serial number, and then updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress.

[0015] Optionally, the output unit is further configured to: in response to the task instance corresponding to the task thread not being abnormal and the current batch being the last batch, obtain the corresponding transaction commit result data and output, and exit the task thread.

[0016] Optionally, the obtaining unit is further configured to: in response to the task instance corresponding to the task thread not being abnormal and the current batch not being the last batch, if the control instruction is not received, continue to obtain the business data of the next batch as the business data of the current batch to continue data processing.

[0017] Optionally, the data processing apparatus further comprises a control instruction receiving unit configured to: in response to the task instance corresponding to the task thread not being abnormal, exit the task thread when the control instruction is received.

[0018] In addition, the application further provides a data processing electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described above.

[0019] In addition, the application further provides a computer readable medium, which stores a computer program, when the program is executed by a processor, the data processing method as described above is implemented.

[0020] To achieve the above object, according to another aspect of the embodiments of the application, a computer program product is provided.

[0021] The computer program product of the embodiments of the application comprises a computer program, when the program is executed by a processor, the data processing method provided by the embodiments of the application is implemented.

[0022] The embodiment of the above application has the following advantages or beneficial effects: the application obtains the business data of the current batch, calls the multi-level cache memory table to obtain the auxiliary data corresponding to the business data, determines the transaction to be submitted corresponding to the business data and the transaction type of the transaction based on the business data and the auxiliary data, determines the data source corresponding to the transaction, starts the task thread to start and submit the transaction based on the data source and the transaction type, obtains the transaction submission progress corresponding to the task instance in response to the exception of the task instance, determines the breakpoint transaction according to the transaction submission progress, and performs transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode; in response to the failure of the transaction submission automatic retry, generates exception information and outputs, and exits the task thread. Thus, by determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction to be submitted based on the business data and the auxiliary data, the complexity of development and operation is reduced, by introducing the multi-level cache memory table, the data query efficiency is improved during data processing, by performing transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode when the task instance is abnormal, the consistency of data processing after interruption is ensured.

[0023] The further effects of the above-mentioned non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are used to better understand the application, and do not constitute an improper limitation on the application. Among them:

[0025] Figure 1 is a schematic diagram of the main process of the data processing method according to an embodiment of the application;

[0026] Figure 2 is a schematic diagram of the main flow of the data processing method according to an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of the flow-batch integrated data processing of the data processing method according to an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of the memory representation of the data processing method according to an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of the breakpoint processing of the data processing method according to an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of the head processing of the data processing method according to an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of the main unit of the data processing apparatus according to an embodiment of the present application;

[0032] Figure 8 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0033] Figure 9 is a structural schematic diagram of a computer system suitable for realizing a terminal device or a server of an embodiment of the present application. DETAILED DESCRIPTION

[0034] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are cited by way of example only. The various details of the embodiments of the present application are described herein in order to provide what is considered to be a thorough description of the present application. It should be understood to those of ordinary skill in the art that the details given herein are not intended to limit the scope of the present application. It should also be understood that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the purpose of clarity and a concise description, descriptions of well-known functions and constructions can be omitted. It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present application comply with relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present application, some industry existing solutions, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solutions. In the technical solutions of the present application, the collection, collection, updating, analysis, processing, use, transmission, storage, etc. of user personal information comply with relevant laws and regulations, are used for legal and reasonable purposes, do not violate public order and good customs, are not shared, disclosed or sold outside these legal uses, and are subject to supervision and management by regulatory authorities. Necessary measures are taken for user personal information to prevent illegal access to such user personal information data, to maintain user personal information security, network security and national security, to ensure that personnel with access to personal information data comply with relevant laws and regulations, and to ensure user personal information security. Once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0035] When used, including in certain related applications, user privacy is protected by de-identifying data, for example by removing specific identifiers, controlling the amount or specificity of data stored, controlling how data is stored, and / or other methods of de-identifying.

[0036] Figure 1 is a schematic diagram of the main process of the data processing method according to an embodiment of the present application, as shown in Figure 1 The data processing method mainly includes the following steps S101-S105.

[0037] Step S101, in response to the completion of the business logic execution, the current batch of business data is acquired, and a multi-level cache memory table is called to acquire the auxiliary data corresponding to the business data.

[0038] In this embodiment, the execution subject of the data processing method (for example, a server) can detect in real time whether the business logic is executed, for example, the business logic of a business application (for example, a business application bound with a sensor, a scanner, and the like) through wired connection or wireless connection. When the end identifier is acquired, it is determined that the business logic corresponding to the end identifier is executed. When the business logic is executed, the data processing can be performed on the business data output by the corresponding business application in batches. Specifically, the current batch of business data can be acquired, and the corresponding auxiliary data can be queried and acquired from the multi-level cache memory table based on the current batch of business data. The auxiliary data can include data that needs to be processed in real time, or data that does not need to be processed in real time. For example, the current batch of business data can be sensor data that needs to be processed in real time, and the acquired auxiliary data corresponding to the current batch of business data can include sensor operation log data that does not need to be processed in real time and real-time decision data that needs to be processed in real time.

[0039] In step S102, based on the business data and the auxiliary data, the transaction to be submitted corresponding to the business data and the transaction type of the transaction are determined.

[0040] The current batch of business data and the auxiliary data corresponding to the current batch of business data are labeled according to whether the data needs to be processed in real time, and the current batch of business data and the auxiliary data corresponding to the current batch of business data are clustered according to the labeled tags to obtain each cluster.

[0041] For example, each cluster includes cluster 1 and cluster 2, wherein cluster 1 (tag: needs to be processed in real time): sensor data + real-time decision data; and cluster 2 (tag: does not need to be processed in real time): sensor operation log data.

[0042] Further, the transaction to be submitted 1 is a transaction of writing the sensor data + real-time decision data in the cluster 1 (tag: needs to be processed in real time) into the message middleware kafka, and the transaction type is a kafka transaction.

[0043] The transaction to be submitted 2 is a transaction of writing the sensor operation log data in the cluster 2 (tag: does not need to be processed in real time) into a data table, and the transaction type is a database transaction.

[0044] In step S103, the data source corresponding to the transaction is determined, and a task thread is started to start and submit the transaction based on the data source and the transaction type.

[0045] If the transaction type of the transaction is a database transaction, the data source corresponding to the transaction (e.g., transaction 2) is a target data table in a target relational database, and the target relational database (e.g., relational database 2) and the target data table (e.g., data table 2 in relational database 2) in the target relational database can be determined according to the data (e.g., operation log data of the sensor in the cluster 2 (label: no real-time processing required)) in the cluster (e.g., cluster 2 in step S102) corresponding to the transaction (e.g., transaction 2). A task thread is started to start and commit the transaction based on the data source and the transaction type, that is, the task thread is started to accurately write the data (e.g., operation log data of the sensor in the cluster 2 (label: no real-time processing required)) in the cluster corresponding to the transaction (e.g., transaction 2) into the target data table (e.g., data table 2 in relational database 2) in the target relational database (e.g., relational database 2).

[0046] If the type of the transaction is a kafka transaction, the data source corresponding to the transaction (e.g., transaction 1) is a target message middleware, and the target message middleware (e.g., message middleware 1) can be determined according to the data (e.g., sensor data + real-time decision data in cluster 1 (label: real-time processing required)) in the cluster (e.g., cluster 1 in step S102) corresponding to the transaction (e.g., transaction 1). A task thread is started to start and commit the transaction based on the data source and the transaction type, that is, the task thread is started to accurately write the data (e.g., sensor data + real-time decision data in cluster 1 (label: real-time processing required)) in the cluster corresponding to the transaction (e.g., transaction 1) into the target message middleware (e.g., message middleware 1).

[0047] In step S104, in response to an exception of the task instance corresponding to the task thread, a transaction commit progress corresponding to the task thread is obtained, a breakpoint transaction is determined according to the transaction commit progress, and a transaction commit automatic retry is performed based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction commit progress, and a preset data consistency mode.

[0048] The exception of the task instance corresponding to the task thread can be that there is a transaction start or commit exception in the task instance. When the task instance corresponding to the started task thread has an exception, the transaction commit progress (i.e., the task progress) of the transaction in the task instance can be obtained, for example, which transactions have been committed and which transactions have not been committed, so as to determine the breakpoint transaction (i.e., the last committed transaction corresponding to the transaction commit progress, for example, the nth committed transaction shown in FIG. 6), that is, the transaction with the exception. Figure 5

[0049] ​The preset data consistency mode can be repeatable execution, processing from a breakpoint, or processing from the beginning. After the breakpoint transaction is determined, based on the breakpoint transaction (for example, based on the sequence number of the input data corresponding to the nth committed transaction), the data source corresponding to the breakpoint transaction (for example, a target data table in a target relational database, and for example, a target message middleware), and the transaction submission progress, the transaction submission automatic retry logic (for example, all the outputs of the last time are cleared or the processed input is filtered, and the task instance is re-executed) corresponding to the preset data consistency mode (for example, repeatable execution) is executed.

[0050] In step S105, in response to the failure of the transaction submission automatic retry, an exception information is generated and output, and the task thread is exited.

[0051] If the transaction submitted through the automatic retry still has an exception, it is determined that the transaction submission automatic retry fails, an exception information is generated and output, and the task thread is exited.

[0052] In some embodiments, the data processing method further comprises: in response to the success of the transaction submission automatic retry, determining whether the current batch is the last batch, if the current batch is the last batch, exiting the task thread, if the current batch is not the last batch, continuing to determine whether a control instruction is received, if the control instruction (which can be used to control the exit of the task thread) is received, exiting the task thread, and if the control instruction is not received, continuing to acquire the business data of the next batch as the business data of the current batch to continue (according to the processing logic of the business data of the current batch) data processing until the current batch is the last batch, and the task thread is exited.

[0053] In some embodiments, the data processing method further comprises: in response to the task instance corresponding to the task thread not having an exception and the current batch being the last batch, acquiring and outputting the corresponding transaction submission result data (for example, acquiring the data successfully written into the target data table in the target relational database after the data corresponding to the transaction that does not need real-time processing, and for example, acquiring the data successfully written into the target message middleware after the data corresponding to the transaction that needs real-time processing), and exiting the task thread; in response to the task instance corresponding to the task thread not having an exception and the current batch not being the last batch, if the control instruction is not received, continuing to acquire the business data of the next batch as the business data of the current batch to continue (according to the processing logic of the business data of the current batch) data processing; and in response to the task instance corresponding to the task thread not having an exception, when the control instruction (which can be used to control the exit of the task thread) is received, exiting the task thread.

[0054] The embodiment obtains the business data of the current batch in response to the completion of the business logic, calls the multi-level cache memory table to obtain the auxiliary data corresponding to the business data, determines the transaction to be submitted corresponding to the business data and the transaction type of the transaction based on the business data and the auxiliary data, determines the data source corresponding to the transaction, starts a task thread to start and submit the transaction based on the data source and the transaction type, obtains the transaction submission progress corresponding to the task instance in response to an exception of the task instance, determines the breakpoint transaction according to the transaction submission progress, and performs transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode. In response to the failure of the transaction submission automatic retry, an exception information is generated and output, and the task thread is exited. Thus, by determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction to be submitted based on the business data and the auxiliary data, the complexity of development and operation is reduced, by introducing the multi-level cache memory table, the data query efficiency is improved during data processing, and by performing transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode when the task instance is abnormal, the consistency of data processing after interruption is ensured.

[0055] Figure 2 is the main flow diagram of the data processing method according to an embodiment of the present application, as shown in Figure 2 the data processing method mainly includes the following steps S201-S207.

[0056] Step S201, in response to the completion of the business logic, the business data of the current batch is obtained, and the multi-level cache memory table is called to obtain the auxiliary data corresponding to the business data.

[0057] The business logic can be detected in real time, which can be the business logic of various business applications. When the end identifier is obtained, it can be determined that the business logic corresponding to the end identifier is executed. When the business logic is executed, the business data output by the corresponding business application is processed in batches, specifically, the business data of the current batch is obtained, and the corresponding auxiliary data is queried and obtained from the multi-level cache memory table based on the business data of the current batch. The auxiliary data can include data that needs to be processed in real time, or data that does not need to be processed in real time.

[0058] Step S202, determine the business data type corresponding to the business data and the auxiliary data type corresponding to the auxiliary data.

[0059] For example, if the current batch of business data is sensor data that needs to be processed in real time, the business data type corresponding to the current batch of business data is data that needs to be processed in real time; if the auxiliary data corresponding to the current batch of business data includes sensor operation log data that does not need to be processed in real time and real-time decision data that needs to be processed in real time, the business data type corresponding to the auxiliary data is data that does not need to be processed in real time + data that needs to be processed in real time.

[0060] In step S203, according to the business data type and the auxiliary data type, the corresponding data table and the message middleware are determined, and the business data and the auxiliary data are classified according to the data table and the message middleware to obtain the input data corresponding to the data table and the input data corresponding to the message middleware, respectively.

[0061] For example, according to the business data type (for example, data that needs to be processed in real time) and the auxiliary data type (for example, data that does not need to be processed in real time + data that needs to be processed in real time), the corresponding data table (for example, a target data table in a target relational database corresponding to the sensor operation log data in the auxiliary data that does not need to be processed in real time) and the message middleware (for example, a target message middleware corresponding to the sensor data and the real-time decision data in the business data and the auxiliary data that need to be processed in real time) are determined, and the business data and the auxiliary data are classified according to the data table (corresponding to data that does not need to be processed in real time) and the message middleware (corresponding to data that needs to be processed in real time), that is, the business data and the auxiliary data are divided into data corresponding to the data table (corresponding to data that does not need to be processed in real time) and the message middleware (corresponding to data that needs to be processed in real time) respectively, to obtain the input data corresponding to the data table and the input data corresponding to the message middleware, respectively.

[0062] In step S204, based on the data table, the message middleware, and the input data corresponding to each of them, the transaction to be submitted corresponding to the business data and the transaction type of the transaction are determined.

[0063] For example, the message middleware is, for example, message middleware 1; the data table is, for example, data table 2 in a relational database 2.

[0064] The input data corresponding to the message middleware 1: for example, can be sensor data + real-time decision data.

[0065] The input data corresponding to the data table 2 in the relational database 2: for example, can be sensor operation log data.

[0066] The transaction to be submitted 1: for example, can be a transaction of writing the sensor data + real-time decision data as input data into the message middleware 1, and the transaction type is a kafka transaction.

[0067] Transaction 2 to be submitted: for example, a transaction of writing the operation log data of the sensor as input data into the data table 2 in the relational database 2, and the transaction type is a database transaction.

[0068] Step S205: determining the data source corresponding to the transaction, and starting a task thread to start and submit the transaction based on the data source and the transaction type.

[0069] If the type of the transaction is a kafka transaction, the data source corresponding to the transaction (for example, transaction 1) is a target message middleware, and the target message middleware (for example, message middleware 1) can be determined according to the input data (for example, sensor data + real-time decision data) corresponding to the transaction (for example, transaction 1). Start a task thread to start and submit the transaction based on the data source and the transaction type, that is, start a task thread to accurately write the input data (for example, sensor data + real-time decision data) corresponding to the transaction (for example, transaction 1) into the target message middleware (for example, message middleware 1).

[0070] If the type of the transaction is a database transaction, the data source corresponding to the transaction (for example, transaction 2) is a target data table in a target relational database, and the target relational database (for example, relational database 2) and the target data table (for example, data table 2 in the relational database 2) in the target relational database can be determined according to the input data (for example, operation log data of the sensor) corresponding to the transaction (for example, transaction 2). Start a task thread to start and submit the transaction based on the data source and the transaction type, that is, start a task thread to accurately write the input data (for example, operation log data of the sensor) corresponding to the transaction (for example, transaction 2) into the target data table (for example, data table 2 in the relational database 2) in the target relational database (for example, relational database 2).

[0071] Step S206: in response to an exception of the task instance corresponding to the task thread, obtaining a transaction submission progress corresponding thereto, determining a breakpoint transaction according to the transaction submission progress, and performing transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode.

[0072] Specifically, determining the breakpoint transaction according to the transaction submission progress includes: determining a transaction identifier at a breakpoint position according to the transaction submission progress, and determining the breakpoint transaction according to the transaction identifier; and performing transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode includes: determining a serial number of the input data corresponding to the breakpoint transaction; determining a processing mode of the input data corresponding to each transaction of the task instance according to the serial number, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode; and performing transaction submission automatic retry based on the processing mode.

[0073] According to an example, the transaction identifier (e.g., n) at the breakpoint position is determined according to the transaction commit progress, the breakpoint transaction (e.g., the nth committed transaction) is determined according to the transaction identifier, the sequence number (e.g., 103, 104) of the input data corresponding to the breakpoint transaction (e.g., the nth committed transaction) is determined, the processing mode (e.g., clear all the outputs of the last time or filter the processed input, and re-execute the task instance; for example, do not clear the outputs of the last time, continue processing from the position of the last interruption after the high availability switching; for example, do not clear the outputs of the last time, the task instance starts processing from the first input data after the high availability switching, but intercepts the outputs corresponding to the position before the last interruption) of the input data corresponding to each transaction of the task instance is determined according to the sequence number (e.g., 103, 104), the data source (e.g., the data table 2 in the message middleware 1 and the relational database 2) corresponding to the breakpoint transaction, the transaction commit progress (e.g., the last committed transaction is the nth committed transaction), and the preset data consistency mode (e.g., repeatable execution, processing from the breakpoint, or processing from the beginning), and the transaction commit automatic retry is accurately executed based on the determined processing mode.

[0074] Specifically, the processing mode of the input data corresponding to each transaction of the task instance is determined according to the sequence number, the data source corresponding to the breakpoint transaction, the transaction commit progress and the preset data consistency mode, including: determining the data source corresponding to the sequence number according to the data source corresponding to the breakpoint transaction and the preset transaction commit priority; if the preset data consistency mode is repeatable execution, determining the processing mode of the input data corresponding to each transaction of the task instance as: based on the transaction commit progress, all the input data corresponding to each transaction of the task instance written in the data source corresponding to the sequence number is cleared, and then the task instance is re-executed; determining the first sequence number corresponding to the data table and the second sequence number corresponding to the message middleware in the sequence number, wherein the first sequence number is greater than the second sequence number; and if the preset data consistency mode is breakpoint processing, determining the processing mode of the input data corresponding to each transaction of the task instance as: reading the input data corresponding to the data table at the first sequence number but not writing into the corresponding data table, reading the input data corresponding to the message middleware at the first sequence number and writing into the message middleware at the second sequence number, updating the second sequence number to the first sequence number in response to successful writing, and then updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress; if the preset data consistency mode is head processing, determining the processing mode of the input data corresponding to each transaction of the task instance as: based on the transaction commit progress, reading the input data corresponding to the message middleware at the second sequence number and the sequence number before the second sequence number but not writing into the corresponding message middleware, reading the input data corresponding to the data table at the first sequence number and the sequence number before the first sequence number but not writing into the corresponding data table, reading the input data corresponding to the message middleware at the first sequence number and writing into the message middleware at the second sequence number, updating the second sequence number to the first sequence number in response to successful writing, and then updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress.

[0075] For example, according to the data source (for example, the target data table in the target message middleware and the target relational database, for example, the data table 2 in the message middleware 1 and the relational database 2) corresponding to the breakpoint transaction and the preset transaction commit priority (for example, when writing data, the priority of the target data table in the target relational database is higher than that of the target message middleware), the data source (for example, the sequence number 103 corresponds to the target message middleware, for example, it can be the message middleware 1, and the sequence number 104 corresponds to the target data table in the target relational database, for example, it can be the data table 2 in the relational database 2) corresponding to the sequence number (for example, 103, 104) is determined.

[0076] If the preset data consistency mode is repeatable execution, the processing manner of the input data corresponding to each transaction of the task instance is determined as follows: based on the transaction submission progress (for example, the last committed transaction is the nth committed transaction), the input data corresponding to the nth committed transaction is cleared from the data source (for example, the data source corresponding to the sequence number 103 is a target message middleware, for example, it can be message middleware 1, and the data source corresponding to the sequence number 104 is a target data table in a target relational database, for example, it can be data table 2 in relational database 2) in which the input data corresponding to each transaction of the task instance is historically written, and then the task instance is re-executed. For example, when the task instance appears an exception (i.e., the transaction corresponding to the task instance appears an exception), the output of the last data source is cleared or the processed input is filtered, and the task instance is re-executed. This mode has the following characteristics: the breakpoint transaction is not enabled during the execution of the task instance; an alarm is sent when the task instance exits abnormally; when the task instance is started manually for retry, the output table is cleaned or the processed input is filtered; after the task instance is restarted, the first input data is processed.

[0077] According to the preset transaction commit priority (for example, the priority of the target data table in the target relational database is higher than that of the target message middleware when writing data), the first sequence number (for example, 104) corresponding to the data table and the second sequence number (for example, 103) corresponding to the message middleware in the sequence number of the input data corresponding to the breakpoint transaction are determined, wherein the first sequence number (for example, 104) is greater than the second sequence number (for example, 103); and if the preset data consistency mode is from the breakpoint processing, the processing mode of the input data corresponding to each transaction of the task instance is determined as follows: reading the input data (for example, the operation log data of the sensor) corresponding to the data table of the first sequence number (for example, 104) but not writing the corresponding data table (for example, data table 2 in relational database 2), reading the input data (for example, sensor data + real-time decision data) corresponding to the message middleware of the first sequence number (for example, 104) and writing the message middleware (for example, message middleware 1) corresponding to the second sequence number (for example, 103), in response to the writing success, updating the second sequence number (for example, 103) corresponding to the corresponding message middleware (for example, message middleware 1) to the first sequence number (for example, 104), indicating that the input data (for example, sensor data + real-time decision data) corresponding to the message middleware of the first sequence number (for example, 104) is successfully written in the message middleware (for example, message middleware 1), and then updating the transaction commit progress to be the breakpoint transaction commit success (for example, the nth committed transaction commit success), and continuing to execute the task instance based on the updated transaction commit progress, that is, continuing to normally execute the commit of the next transaction. For example, when the task instance appears an exception (that is, the transaction corresponding to the task instance appears an exception), there is no need to clear the last output, and after the high availability is switched, the position (for example, Figure 5 The interception point: 104 corresponding to the nth committed transaction before the task interruption is shown, and the distributed consistency consensus framework (DCF framework) is used to ensure that the processed data and results are not repeated or missed. This mode has the following characteristics: enabling the breakpoint transaction during the execution of the task instance, recording the breakpoint transaction; sending an alarm when the task instance exits abnormally; reading the position (for example, the interception point: 104) of the breakpoint transaction when the task instance is started manually; and processing (for example, business processing: 104) from the breakpoint transaction position after the task instance is restarted. For example, Figure 5As shown, 1, 2, 3, 4, 5, …, 103, 104, 105, 106, 107, 108, 109 represent 109 pairs of input data, wherein each pair of input data includes one piece of input data that does not need real-time processing for the data table and one piece of input data that needs real-time processing for the message middleware kafka. If one piece of input data that does not need real-time processing for the data table in each pair of input data is successfully written into the corresponding data table and one piece of input data that needs real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are combined and displayed as one number, for example, one piece of input data (i.e., the 103rd) that does not need real-time processing for the data table in the 103rd pair of input data is successfully written into the corresponding data table and one piece of input data (i.e., the 103rd) that needs real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, which are combined and displayed as one number (i.e., 103); if one piece of input data that does not need real-time processing for the data table in each pair of input data is successfully written into the corresponding data table, but one piece of input data that needs real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately, for example, one piece of input data (i.e., the 104th) that does not need real-time processing for the data table in the 104th pair of input data is successfully written into the corresponding data table, but one piece of input data (i.e., the 104th) that needs real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, which need to be displayed separately (e.g., 103, 104). When writing data, the priority of the data table is higher than that of the message middleware kafka. Before the task is interrupted, when the nth committed transaction occurs an exception, the positions of the serial numbers of the input data corresponding to the breakpoint transaction are 103 (representing the 103rd input data that has been written into kafka) and 104 (representing the 104th input data that has been written into the data table), respectively, which means that the 104th input data (e.g., sensor operation log data) that does not need real-time processing has been successfully written into the data table, but an exception (i.e., not successfully written into the message middleware kafka) occurs when writing the 104th input data (e.g., sensor data and real-time decision data) that needs real-time processing corresponding to the 104th input data (e.g., sensor operation log data) that does not need real-time processing into the message middleware kafka.According to the intercept point: 104, the business processing: 104, the execution automatic retry, that is, from the 104th input data (for example, the operation log data of the sensor) which does not need real-time processing, repeatedly read the 104th input data (for example, the operation log data of the sensor) which does not need real-time processing and intercept the transaction, that is, intercept the output corresponding to the 104th input data (for example, the operation log data of the sensor) which does not need real-time processing (that is, do not write into the corresponding data table), and then read the 104th input data (for example, sensor data and real-time decision data) which needs real-time processing corresponding to the 104th input data (for example, the operation log data of the sensor) which does not need real-time processing and write into the corresponding message middleware kafka, and after successfully writing into the message middleware kafka, the task is restored, and after the task is restored, the 105th input data is started, and the business program enters the normal execution mode (that is, no longer intercepts the transaction, both the reading of the input data and the writing of the input data).

[0078] If the preset data consistency mode is from head processing, the processing mode of the input data corresponding to each transaction of the task instance is determined as follows: based on the transaction submission progress, reading the input data (for example, instant messaging data, streaming media data, and other data that needs to be processed in real time) corresponding to the message middleware at the second sequence number (for example, 103) and the sequence numbers before the second sequence number (for example, 103) but not writing into the corresponding message middleware, reading the input data (for example, historical transaction data, customer access data, and other data that does not need to be processed in real time) corresponding to the data table at the first sequence number (for example, 104) and the sequence numbers before the first sequence number (for example, 104) but not writing into the corresponding data table, reading the input data (for example, sensor data + real-time decision data) corresponding to the message middleware at the first sequence number (for example, 104) and writing into the message middleware (for example, message middleware 1) corresponding to the second sequence number (for example, 103), in response to the successful writing, updating the second sequence number (for example, 103) corresponding to the corresponding message middleware (for example, message middleware 1) to the first sequence number (for example, 104), indicating that the input data (for example, sensor data + real-time decision data) corresponding to the message middleware at the first sequence number (for example, 104) is successfully written into the message middleware (for example, message middleware 1), and then updating the transaction submission progress to breakpoint transaction submission success (for example, the nth submitted transaction is successfully submitted), and continuing to execute the task instance based on the updated transaction submission progress, that is, continuing to normally execute the submission of the next transaction. For example, when the task instance appears an exception (that is, the transaction corresponding to the task instance appears an exception), there is no need to clean up the output of the last time. After the high availability switching, the task instance starts processing from the first input data, but the Distributed Consensus Framework (DCF framework) will intercept the output corresponding to the position (for example, Figure 6 before the interception point 104) of the nth submitted transaction before the task interruption shown in the figure. This mode has the following characteristics: enabling breakpoint transactions during the execution of the task instance, recording the sequence numbers (for example, message middleware kafka: 103, data table: 104) of the input data corresponding to the breakpoint transaction (for example, the nth submitted transaction) for the data table and the message middleware kafka respectively; sending an alarm when the task instance exits abnormally; reading the position (for example, the interception point 104) of the breakpoint transaction when manually retrying to start the task instance; after restarting the task instance, executing automatic retry according to business processing: 1, that is, starting processing from the first input data and intercepting transactions, that is, intercepting the output before the breakpoint transaction and normally submitting the transactions after the breakpoint transaction. For example, Figure 6As shown, 1, 2, 3, 4, 5, …, 103, 104, 105, 106, 107, 108, 109 represent 109 pairs of input data, wherein each pair of input data includes one piece of input data that does not need real-time processing for the data table and one piece of input data that needs real-time processing for the message middleware kafka. If one piece of input data that does not need real-time processing for the data table in each pair of input data is successfully written into the corresponding data table and one piece of input data that needs real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are combined and displayed as one number, for example, one piece of input data (i.e., the 103rd) that does not need real-time processing for the data table in the 103rd pair of input data is successfully written into the corresponding data table and one piece of input data (i.e., the 103rd) that needs real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, which are combined and displayed as one number (i.e., 103); if one piece of input data that does not need real-time processing for the data table in each pair of input data is successfully written into the corresponding data table, but one piece of input data that needs real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately, for example, one piece of input data (i.e., the 104th) that does not need real-time processing for the data table in the 104th pair of input data is successfully written into the corresponding data table, but one piece of input data (i.e., the 104th) that needs real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, which need to be displayed separately (e.g., 103, 104). When writing data, the priority of the data table is higher than that of the message middleware kafka. Before the task is interrupted, the serial numbers of the input data corresponding to the breakpoint transaction when the task fails are respectively located at 103 (corresponding to kafka) and 104 (corresponding to the database). According to the interception point: 104, the business processing: 1, the automatic retry is executed, that is, the processing starts from the first input data and the transaction is intercepted, that is, for the message middleware kafka, the output before (including the 103rd input data) the 103rd input data that needs real-time processing is intercepted, for the data table, the output before (including the 104th input data) the 104th input data that does not need real-time processing is intercepted, and then the 104th input data (e.g., sensor data and real-time decision data) that needs real-time processing corresponding to the 104th input data (e.g., operation log data of the sensor) that does not need real-time processing is read and written into the corresponding message middleware kafka, and after the writing into the message middleware kafka is successful, the task is resumed, and after the task is resumed, the processing starts from the 105th input data, and the business program enters the normal execution mode (i.e., the transaction is no longer intercepted, both the reading and the writing of the input data are performed).

[0079] Step S207, in response to the transaction commit automatic retry failure, generate exception information and output, exit the task thread.

[0080] If the automatic retry of the transaction that is submitted still appears an exception, it is determined that the transaction commit automatic retry fails, exception information is generated and output, and the task thread is exited.

[0081] Figure 3 is a flow batch integrated data processing schematic diagram of a data processing method according to an embodiment of the present application. As shown in Figure 3 When the business logic is executed, the current batch of business data is obtained, and a multi-level cache memory table as shown in Figure 4 is called to obtain the auxiliary data corresponding to the business data. Based on the business data and the auxiliary data, the transaction to be submitted corresponding to the business data and the transaction type of the transaction are determined; the data source corresponding to the transaction is determined, and a task thread is started to start and submit the transaction based on the data source and the transaction type; in response to an exception of a task instance corresponding to the task thread, the transaction submission progress corresponding to the transaction is obtained, the breakpoint transaction is determined according to the transaction submission progress, and the transaction commit automatic retry is performed based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode. In response to the transaction commit automatic retry failure, exception information is generated and output, and the task thread is exited. In response to the transaction commit automatic retry success, it is judged whether the current batch is the last batch, if the current batch is the last batch, the task thread is exited, if the current batch is not the last batch, it is continuously judged whether a control instruction is received, if the control instruction is received, the task thread is exited, if the control instruction is not received, the next batch of business data is continuously obtained as the business data of the current batch to continue data processing, until the current batch is the last batch, the task thread is exited. Thus, by determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction to be submitted based on the business data and the auxiliary data, the complexity of development and operation is reduced, by introducing the multi-level cache memory table, the data query efficiency is improved during data processing, by performing the transaction commit automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode when the task instance appears an exception, the consistency of data processing after interruption is guaranteed.

[0082] Figure 4 is a memory representation diagram of a data processing method according to an embodiment of the present application. In the embodiment of the present application, the memory table is a multi-level cache memory table. As shown in Figure 4As shown, the multi-level cache memory table can include an interface layer, a first-level cache, and a second-level cache, wherein the interface layer is configured to provide an interface for data query. The first-level cache stores hot data. The first-level cache is implemented by Guava and Caffeine, wherein Guava is a set of Java libraries open sourced by Google, which contains various data structures, caches, collections supporting primitive operations, concurrent tool classes, common annotations, event buses, internationalization support, etc. Caffeine is a high-performance cache library based on Java 8, close to the best cache library. Caffeine provides in-memory caching using Google Guava-inspired APIs. The second-level cache stores all data in the data table in the relational database and all data in the message middleware kafka. The second-level cache is implemented by SQLite and IMTable, wherein SQLite is a lightweight, self-contained and serverless relational database management system (RDBMS) that is easy to create and requires less configuration. SQLite is an open source, lightweight, fast, independent and highly reliable SQL database engine written in C language, which provides a complete database solution. IMTable is the abbreviation of “In-Memory Table”, which is a table technology that stores data in memory to facilitate fast access and processing. By introducing the multi-level cache memory table, the query of read-only data can be greatly optimized.

[0083] In the embodiment of the present application, when the task thread is started, the task thread context can be initialized, including breakpoint transaction, shard information, business data source, transaction manager, write component instance, etc. The read component is initialized by the task thread to read a batch of business data (i.e., to obtain the current batch of business data) and process the business data. Specifically, the processing of the business data can be performed according to the process of before processing (beforeWindow)-processing (execute)-after processing (afterWindow) to obtain the processing result data. For example, the processing result data can include business data (e.g., sensor data) and auxiliary data corresponding to the business data (e.g., operation log data and real-time decision data corresponding to the record of the sensor data). After obtaining the processing result data, the task initialization method can be called by the task thread, the transaction is started and committed by the Distributed Consensus Framework (DCF framework), and the transaction submission progress (i.e., task progress) is recorded. Then, the task thread judges whether the task is completed, if not, the next batch of business data is obtained and the processing is continued until the control instruction is received, if yes, the exit before processing logic (exit) is called, and then the task thread is exited.

[0084] In the embodiments of the present application, the management of the transaction is as follows: transaction takeover: all the business data output by the business application is taken over by the distributed consensus framework (DCF framework), including the opening and submission of the transaction, and supporting custom data source configuration. The data source in the embodiments of the present application can be a data table in a relational database or a message middleware kafka. A delayed opening mechanism can be used, that is, each transaction can be marked as opened before the business logic is executed, but not actually acquiring the data source connection to open the transaction or the consumer to open the transaction. After the business logic is executed, the data source connection is acquired to open the transaction or the consumer to open the transaction and is submitted, and transaction timeout caused by business logic does not occur. Transaction submission interception: when the application enables the high availability of the framework, the transaction submission progress is recorded when the transaction is submitted, and when the transaction is repeatedly executed, the submitted data can be intercepted according to the transaction submission progress. Distributed transaction: the distributed consensus framework (DCF framework) realizes cross-medium transaction submission, supports the submission of database transactions (that is, the transaction corresponding to batch data, and the batch data refers to data that does not need to be processed in real time) and kafka transactions (that is, the transaction corresponding to streaming data, and the streaming data refers to data that needs to be processed in real time) together. The submission process is to submit the database transaction first, and then submit the kafka transaction after success.

[0085] In the embodiments of the present application, data consistency refers to the data in the data table and the data in the message middleware kafka corresponding one by one. For example, there are 109 data in the data table, and there should also be 109 data in the message middleware kafka and all correspond to the data in the data table one by one. In terms of data consistency, the task progress is recorded according to the respective transaction submission in the data table and the message middleware kafka. The status data such as the task progress (that is, the transaction submission progress) needs to be persistently stored in the data processing process to ensure the consistency of the data processing after the interruption is recovered. When the application has distributed transaction output, there will be two task progresses, one representing the database transaction submission and one representing the kafka transaction submission. The progress of the two will not exceed 1 window.

[0086] Transaction exception handling: refers to the handling of network jitter, transaction timeout and data writing exception and the like occurring when the transaction is opened and submitted. For these abnormal situations, the transaction manager can perform three automatic retries, respectively: repeatable execution, breakpoint processing and head processing, based on the three automatic retries, most of the abnormal situations occurring can be handled, if the automatic retry still cannot solve the abnormal situation occurring, the abnormal information can be output to the log and the current task thread is exited.

[0087] The goal of data consistency is to ensure that there is no data duplication and no data loss when a task is resumed by the Distributed Consensus Framework (DCF framework). The Distributed Consensus Framework (DCF framework) provides three data consistency solutions: repeatable execution, resume from breakpoint, and start from scratch.

[0088] Specifically, the repeatable execution: when a task instance is abnormal (i.e., the transaction corresponding to the task instance is abnormal), the last output can be completely cleared or the processed input is filtered, and the task instance is re-executed. This mode has the following characteristics: breakpoint transactions are not enabled during task instance execution; an alarm is sent when the task instance exits abnormally; when the task instance is started manually for retry, the output table is cleaned up or the processed input is filtered; after the task instance is restarted, the first input data is processed.

[0089] The resume from breakpoint: when a task instance is abnormal (i.e., the transaction corresponding to the task instance is abnormal), the last output does not need to be cleared. After high availability switching, the processing continues from the last interrupted position (e.g., the interception point: 104) of the nth committed transaction before the task interruption, and the Distributed Consensus Framework (DCF framework) ensures that the processed data and results are not duplicated and not lost. This mode has the following characteristics: breakpoint transactions are enabled during task instance execution; an alarm is sent when the task instance exits abnormally; when the task instance is started manually for retry, the position of the breakpoint transaction (e.g., the interception point: 104) is read; after the task instance is restarted, the processing starts from the breakpoint transaction position (e.g., the business processing: 104). Figure 5 The start from scratch: when a task instance is abnormal (i.e., the transaction corresponding to the task instance is abnormal), the last output is cleared, and the task instance is re-executed from the first input data. This mode has the following characteristics: breakpoint transactions are not enabled during task instance execution; an alarm is sent when the task instance exits abnormally; when the task instance is started manually for retry, the output table is cleaned up or the processed input is filtered; after the task instance is restarted, the first input data is processed.

[0090] Figure 5 FIG. 1 is a schematic diagram of the resume from breakpoint of the data processing method according to an embodiment of the present application. As shown in FIG. 1, the task instance is interrupted at the interception point: 104, and the task instance is restarted from the interception point: 104. Figure 5As shown, 1, 2, 3, 4, 5, …, 103, 104, 105, 106, 107, 108, 109 represent 109 pairs of input data, wherein each pair of input data includes one piece of input data that does not need real-time processing for the data table and one piece of input data that needs real-time processing for the message middleware kafka. If one piece of input data that does not need real-time processing for the data table in each pair of input data is successfully written into the corresponding data table and one piece of input data that needs real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, they are combined and displayed as one number, for example, one piece of input data (i.e., the 103rd) that does not need real-time processing for the data table in the 103rd pair of input data is successfully written into the corresponding data table and one piece of input data (i.e., the 103rd) that needs real-time processing for the message middleware kafka is successfully written into the corresponding message middleware kafka, which are combined and displayed as one number (i.e., 103); if one piece of input data that does not need real-time processing for the data table in each pair of input data is successfully written into the corresponding data table, but one piece of input data that needs real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, they need to be displayed separately, for example, one piece of input data (i.e., the 104th) that does not need real-time processing for the data table in the 104th pair of input data is successfully written into the corresponding data table, but one piece of input data (i.e., the 104th) that needs real-time processing for the message middleware kafka is not successfully written into the corresponding message middleware kafka, which need to be displayed separately (e.g., 103, 104). When writing data, the priority of the data table is higher than that of the message middleware kafka. Before the task is interrupted, when the nth committed transaction is abnormal, the positions of the serial numbers of the input data corresponding to the breakpoint transaction are 103 (representing the 103rd input data that has been written into kafka) and 104 (representing the 104th input data that has been written into the data table), respectively, which means that the 104th input data (e.g., sensor operation log data) that does not need real-time processing has been successfully written into the data table, but an abnormality (i.e., not successfully written into the message middleware kafka) occurs when writing the 104th input data (e.g., sensor data and real-time decision data) that needs real-time processing corresponding to the 104th input data (e.g., sensor operation log data) that does not need real-time processing into the message middleware kafka.

[0091] Based on interception point: 104, business processing: 104, automatic retry is performed, that is, processing starts from the 104th input data that does not require real-time processing (for example, sensor operation log data), repeatedly reads the 104th input data that does not require real-time processing (for example, sensor operation log data) and intercepts the transaction, that is, intercepts the output corresponding to the 104th input data that does not require real-time processing (for example, sensor operation log data) (that is, does not write to the corresponding data table), and then reads the 104th input data that requires real-time processing (for example, sensor data and real-time decision data) corresponding to the 104th input data that does not require real-time processing (for example, sensor operation log data) and writes it to the corresponding message middleware Kafka. After successfully writing to the message middleware Kafka, the task is resumed. After the task is resumed, starting from the 105th input data, the business program enters normal execution mode (that is, no longer intercepts transactions, and both reading and writing input data are performed).

[0092] Processing from the beginning: When an exception occurs in a task instance (i.e., an exception occurs in the transaction corresponding to the task instance), there is no need to clean up the previous output. After the high availability switch, the task instance starts processing from the first input data, but the Distributed Consensus Framework (DCF framework) will intercept the last interruption position (for example, Figure 6 The output before the interception point (104) corresponding to the nth committed transaction before the task was interrupted is shown. This mode has the following features: breakpoint transactions are enabled during task instance execution, and the sequence number of the input data corresponding to the breakpoint transaction (e.g., the nth committed transaction) is recorded for the data table and message middleware Kafka (e.g., 103 for message middleware Kafka, 104 for data table). An alert is sent when a task instance exits abnormally. When manually retrying the task instance, the location of the breakpoint transaction (e.g., interception point 104) is read. After the task instance is restarted, an automatic retry is performed based on business processing: 1, starting with the first input data and intercepting the transaction. This means that the output before the breakpoint transaction is intercepted, and the transactions after the breakpoint transaction are committed normally.

[0093] Figure 6 FIG. 1 is a schematic diagram of a data processing method according to an embodiment of the present application. Figure 6As shown, 1, 2, 3, 4, 5, …, 103, 104, 105, 106, 107, 108, 109 represent 109 pairs of input data, wherein each pair of input data includes one piece of input data for the data table which does not need real-time processing and one piece of input data for the message middleware kafka which needs real-time processing. If one piece of input data for the data table which does not need real-time processing in each pair of input data is successfully written into the corresponding data table and one piece of input data for the message middleware kafka which needs real-time processing is successfully written into the corresponding message middleware kafka, they are combined and displayed as one number, for example, one piece of input data for the data table (i.e. the 103rd) in the 103rd pair of input data is successfully written into the corresponding data table and one piece of input data for the message middleware kafka (i.e. the 103rd) is successfully written into the corresponding message middleware kafka, which are combined and displayed as one number (i.e. 103); if one piece of input data for the data table which does not need real-time processing in each pair of input data is successfully written into the corresponding data table, but one piece of input data for the message middleware kafka which needs real-time processing is not successfully written into the corresponding message middleware kafka, they need to be displayed separately, for example, one piece of input data for the data table (i.e. the 104th) in the 104th pair of input data is successfully written into the corresponding data table, but one piece of input data for the message middleware kafka (i.e. the 104th) is not successfully written into the corresponding message middleware kafka, which need to be displayed separately (e.g. 103, 104). When writing data, the priority of the data table is higher than that of the message middleware kafka. Before the task is interrupted, the serial numbers of the input data corresponding to the breakpoint transaction when the task fails are respectively located at 103 (corresponding to kafka) and 104 (corresponding to the database).

[0094] According to the interception point: 104, the business processing: 1, the automatic retry is executed, that is, the processing starts from the first input data and the transaction is intercepted, that is, for the message middleware kafka, the output before (including) the 103rd input data which needs real-time processing is intercepted, for the data table, the output before (including) the 104th input data which does not need real-time processing is intercepted, and then the 104th input data which does not need real-time processing (e.g. the operation log data of the sensor) is read and the 104th input data which needs real-time processing (e.g. the sensor data and the real-time decision data) corresponding to the 104th input data which does not need real-time processing is written into the corresponding message middleware kafka, and after the writing into the message middleware kafka is successful, the task is resumed, and after the task is resumed, the processing starts from the 105th input data, and the business program enters the normal execution mode (i.e. the transaction is no longer intercepted, both the reading and the writing of the input data are performed).

[0095] In applications such as data processing, data migration, and other applications that need to process a large amount of data, stream data processing and batch data processing play a crucial role. Stream data processing is a technology that processes data in real time while data is flowing in. It can achieve low latency and high throughput processing by continuously receiving continuous data streams for data processing. Batch data processing is a technology that processes data in batches at a predetermined time interval. The embodiments of the present application provide a stream-batch integrated data processing method, which can realize efficient stream-batch integrated data processing and unify the stream data processing method and the batch data processing method. A multi-level cache mechanism is provided to encapsulate different data storage systems. A data consistency scheme based on data tables in a relational database is provided to reduce the complexity of the system.

[0096] Figure 7 is a schematic diagram of the main unit of the data processing apparatus according to the embodiments of the present application. As shown in Figure 7 , the data processing apparatus 700 includes an acquisition unit 701, a transaction determination unit 702, a transaction submission unit 703, an exception processing unit 704, and an output unit 705.

[0097] The acquisition unit 701 is configured to acquire the business data of the current batch in response to the completion of the business logic execution, and call the multi-level cache memory table to acquire the auxiliary data corresponding to the business data.

[0098] The transaction determination unit 702 is configured to determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction based on the business data and the auxiliary data.

[0099] The transaction submission unit 703 is configured to determine the data source corresponding to the transaction, start a task thread to start and submit the transaction based on the data source and the transaction type.

[0100] The exception processing unit 704 is configured to acquire the transaction submission progress corresponding to the task instance in response to an exception of the task instance corresponding to the task thread, determine a breakpoint transaction according to the transaction submission progress, and perform transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress, and the preset data consistency mode.

[0101] The output unit 705 is configured to generate exception information and output in response to the failure of the transaction submission automatic retry, and exit the task thread.

[0102] In some embodiments, the transaction determination unit 702 is further configured to determine a business data type corresponding to the business data and an auxiliary data type corresponding to the auxiliary data, determine a data table and a message middleware corresponding to the business data type and the auxiliary data type according to the business data type and the auxiliary data type, classify the business data and the auxiliary data according to the data table and the message middleware to obtain input data corresponding to the data table and input data corresponding to the message middleware respectively, and determine a transaction to be committed corresponding to the business data and a transaction type of the transaction based on the data table, the message middleware, and the input data corresponding thereto respectively.

[0103] In some embodiments, the exception handling unit 704 is further configured to determine a transaction identifier at the breakpoint position according to the transaction commit progress, and determine the breakpoint transaction according to the transaction identifier, and the exception handling unit 704 is further configured to determine a sequence number of the input data corresponding to the breakpoint transaction, determine a processing manner of the input data corresponding to each transaction corresponding to the task instance according to the sequence number, the data source corresponding to the breakpoint transaction, the transaction commit progress, and a preset data consistency mode, and perform automatic retry of the transaction commit based on the processing manner.

[0104] In some embodiments, the abnormality processing unit 704 is further configured to: determine the data source corresponding to the serial number according to the data source corresponding to the breakpoint transaction and the preset transaction commit priority; if the preset data consistency mode is repeatable execution, determine the processing manner of the input data corresponding to each transaction of the task instance as: based on the transaction commit progress, all of the input data corresponding to each transaction of the task instance that are historically written in the data source corresponding to the serial number are cleared, and then the task instance is re-executed; determine the first serial number corresponding to the data table and the second serial number corresponding to the message middleware in the serial number, wherein the first serial number is greater than the second serial number; and if the preset data consistency mode is breakpoint processing, determine the processing manner of the input data corresponding to each transaction of the task instance as: reading the input data for the data table corresponding to the first serial number but not writing into the corresponding data table, reading the input data for the message middleware corresponding to the first serial number and writing into the message middleware corresponding to the second serial number, in response to successful writing, updating the second serial number to the first serial number, and then updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress; if the preset data consistency mode is head processing, determine the processing manner of the input data corresponding to each transaction of the task instance as: based on the transaction commit progress, reading the input data for the message middleware corresponding to the second serial number and the serial numbers before the second serial number but not writing into the corresponding message middleware, reading the input data for the data table corresponding to the first serial number and the serial numbers before the first serial number but not writing into the corresponding data table, reading the input data for the message middleware corresponding to the first serial number and writing into the message middleware corresponding to the second serial number, in response to successful writing, updating the second serial number to the first serial number, and then updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress.

[0105] In some embodiments, the output unit 705 is further configured to: in response to the task instance corresponding to the task thread not being abnormal and the current batch being the last batch, acquire the corresponding transaction commit result data and output, and exit the task thread.

[0106] In some embodiments, the acquisition unit 701 is further configured to: in response to the task instance corresponding to the task thread not being abnormal and the current batch not being the last batch, if the control instruction is not received, continue to acquire the business data of the next batch as the business data of the current batch to continue data processing.

[0107] In some embodiments, the data processing apparatus further comprises Figure 7 a control instruction receiving unit not shown in the figure, configured to: in response to the task instance corresponding to the task thread not being abnormal, exit the task thread when the control instruction is received.

[0108] It should be noted that the data processing method and the data processing apparatus of the present application have a corresponding relationship in the specific implementation content, and therefore repeated content will not be described.

[0109] Figure 8 An exemplary system architecture 800 to which the data processing method or data processing apparatus of the embodiments of the present application can be applied is shown.

[0110] As shown in Figure 8 The system architecture 800 can include terminal devices 801, 802, 803, a network 804, and a server 805. The network 804 is a medium for providing a communication link between the terminal devices 801, 802, 803 and the server 805. The network 804 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0111] A user can use the terminal devices 801, 802, 803 to interact with the server 805 through the network 804 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 801, 802, 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0112] The terminal devices 801, 802, 803 can be various electronic devices with data processing screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc.

[0113] The server 805 can be a server providing various services, for example, a background management server providing support for service data output by the user terminal devices 801, 802, and 803 (only as an example). The background management server can acquire the current batch of service data in response to the completion of service logic execution, call a multi-level cache memory table to acquire auxiliary data corresponding to the service data, determine a to-be-submitted transaction corresponding to the service data and a transaction type of the transaction based on the service data and the auxiliary data, determine a data source corresponding to the transaction, start a task thread to start and submit the transaction based on the data source and the transaction type, acquire a transaction submission progress corresponding to the task thread in response to an exception of a task instance corresponding to the task thread, determine a breakpoint transaction according to the transaction submission progress, and perform transaction submission automatic retry based on the breakpoint transaction, a data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode. In response to a failure of the transaction submission automatic retry, the background management server generates and outputs exception information and exits the task thread. Thus, by determining the to-be-submitted transaction corresponding to the service data and the transaction type of the to-be-submitted transaction based on the service data and the auxiliary data, the complexity of development and operation and maintenance is reduced. By introducing the multi-level cache memory table, data query efficiency is improved during data processing. By performing transaction submission automatic retry based on the breakpoint transaction, a data source corresponding to the breakpoint transaction, the transaction submission progress, and a preset data consistency mode when an exception occurs in a task instance, the consistency of data processing after interruption is ensured.

[0114] It should be noted that the data processing method provided in the embodiments of the present application is generally executed by the server 805, and accordingly, the data processing apparatus is generally arranged in the server 805.

[0115] It should be understood that Figure 8 The number of terminal devices, networks, and servers in the system is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks, and servers.

[0116] Reference is made below to Figure 9 which shows a structural schematic diagram of a computer system 900 of a terminal device suitable for use to implement the embodiments of the present application. Figure 9 The terminal device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0117] As Figure 9As shown, the computer system 900 includes a central processing unit (CPU) 901 which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 902 or loaded into a random access memory (RAM) 903 from a storage section 908. In the RAM 903, various programs and data required for the operation of the computer system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0118] Connected to the I / O interface 905 are an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display device such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as necessary. A removable recording medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read therefrom is installed into the storage section 908 as necessary.

[0119] In particular, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 909, and / or installed from the removable recording medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-described functions defined in the system of the present application are executed.

[0120] Note that the computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or a combination thereof. The computer-readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus, or device to function according to the program. In the present application, the computer-readable signal medium can include a computer-readable storage medium or any computer-readable medium that transmits, propagates, or transfers programs used by an instruction execution system, apparatus, or device to function according to the programs. The program code contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination of the foregoing.

[0121] The flow diagrams and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0122] The units described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described units can also be arranged in a processor, for example, can be described as: a processor includes an acquisition unit, a transaction determination unit, a transaction submission unit, an exception processing unit and an output unit. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0123] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to, in response to the completion of the business logic execution, acquire the current batch of business data, call the multi-level cache memory table to acquire the auxiliary data corresponding to the business data; based on the business data and the auxiliary data, determine the transaction to be submitted corresponding to the business data and the transaction type of the transaction; determine the data source corresponding to the transaction, start a task thread to start and submit the transaction based on the data source and the transaction type; in response to an exception of a task instance corresponding to the task thread, acquire the corresponding transaction submission progress, determine a breakpoint transaction according to the transaction submission progress, and perform transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode; in response to the failure of the transaction submission automatic retry, generate exception information and output, and exit the task thread.

[0124] The computer program product of the present application includes a computer program, which, when executed by a processor, implements the data processing method in the embodiments of the present application.

[0125] According to the technical scheme of the embodiments of the present application, by determining the transaction to be submitted corresponding to the business data and the transaction type of the transaction to be submitted based on the business data and the auxiliary data, the complexity of development and operation is reduced, by introducing the multi-level cache memory table, the data query efficiency is improved during data processing, and by performing transaction submission automatic retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode when the task instance is abnormal, the consistency of data processing after interruption is ensured.

[0126] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: include: In response to the completion of the business logic execution, the business data of the current batch is obtained, and the multi-level cache memory table is called to obtain auxiliary data corresponding to the business data; Determining, based on the business data and the auxiliary data, a transaction to be submitted corresponding to the business data and a transaction type of the transaction; Determine a data source corresponding to the transaction, start a task thread to open and submit the transaction based on the data source and the transaction type; In response to an exception in the task instance corresponding to the task thread, the corresponding transaction submission progress is obtained, a breakpoint transaction is determined according to the transaction submission progress, and automatic transaction submission retry is performed based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode; the automatic transaction submission retry based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode includes: determining the sequence number of the input data corresponding to the breakpoint transaction; determining the data source corresponding to the sequence number according to the data source corresponding to the breakpoint transaction and the preset transaction submission priority; if the preset data consistency mode is repeatable, determining the processing method for the input data corresponding to each transaction corresponding to the task instance as follows: based on the transaction submission progress, clearing all the input data corresponding to each transaction corresponding to the task instance that has been historically written in the data source corresponding to the sequence number, and then re-executing the task instance; determining a first sequence number corresponding to the data table and a second sequence number corresponding to the message middleware in the sequence number, wherein the first sequence number is greater than the second sequence number; and if the preset data consistency mode is processing from the breakpoint, determining the processing method for each transaction corresponding to the task instance The corresponding input data is processed as follows: reading the input data for the data table corresponding to the first sequence number but not writing it to the corresponding data table, reading the input data for the message middleware corresponding to the first sequence number and writing it to the message middleware corresponding to the second sequence number, in response to a successful write, updating the second sequence number to the first sequence number, further updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress; if the preset data consistency mode is de novo processing, determining the processing method for the input data corresponding to each transaction corresponding to the task instance is: based on the transaction commit progress, reading the input data for the message middleware corresponding to the second sequence number and the sequence number before the second sequence number but not writing it to the corresponding message middleware, reading the input data for the data table corresponding to the first sequence number and the sequence number before the first sequence number but not writing it to the corresponding data table, reading the input data for the message middleware corresponding to the first sequence number and writing it to the message middleware corresponding to the second sequence number, in response to a successful write, updating the second sequence number to the first sequence number, further updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress; performing automatic retry of transaction commit based on the processing method; In response to the failure of the automatic retry of transaction submission, abnormal information is generated and output, and the task thread is exited.

2. The method according to claim 1, characterized in that The determining, based on the business data and the auxiliary data, a transaction to be submitted corresponding to the business data and a transaction type of the transaction includes: Determining a service data type corresponding to the service data and an auxiliary data type corresponding to the auxiliary data; Determining corresponding data tables and message middleware according to the business data type and the auxiliary data type, and classifying the business data and the auxiliary data according to the data table and the message middleware to obtain input data corresponding to the data table and input data corresponding to the message middleware, respectively; Based on the data table, the message middleware, and the corresponding input data, a transaction to be submitted corresponding to the business data and a transaction type of the transaction are determined.

3. The method according to claim 2, characterized in that Determining the breakpoint transaction according to the transaction submission progress includes: The transaction identifier at the breakpoint position is determined according to the transaction commit progress, and the breakpoint transaction is determined according to the transaction identifier.

4. The method according to claim 1, wherein The method further comprises: In response to the fact that no exception occurs in the task instance corresponding to the task thread and the current batch is the last batch, the corresponding transaction submission result data is obtained and output, and the task thread is exited.

5. The method according to claim 1, characterized in that The method further comprises: In response to no exception in the task instance corresponding to the task thread and the current batch is not the last batch, if no control instruction is received, the business data of the next batch is obtained to continue data processing as the business data of the current batch.

6. The method according to claim 1, characterized in that The method further comprises: In response to no exception occurring in the task instance corresponding to the task thread, the task thread is exited upon receiving a control instruction.

7. A data processing device, characterized in that: include: an acquisition unit configured to, in response to completion of the execution of the business logic, acquire the business data of the current batch and call the multi-level cache memory table to acquire auxiliary data corresponding to the business data; a transaction determination unit, configured to determine, based on the business data and the auxiliary data, a transaction to be committed corresponding to the business data and a transaction type of the transaction; a transaction committing unit, configured to determine a data source corresponding to the transaction, start a task thread to start and commit the transaction based on the data source and the transaction type; The exception handling unit is configured to, in response to an exception in the task instance corresponding to the task thread, obtain the corresponding transaction submission progress, determine the breakpoint transaction according to the transaction submission progress, and perform automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode; the automatic retry of transaction submission based on the breakpoint transaction, the data source corresponding to the breakpoint transaction, the transaction submission progress and the preset data consistency mode includes: determining the sequence number of the input data corresponding to the breakpoint transaction; determining the data source corresponding to the sequence number according to the data source corresponding to the breakpoint transaction and the preset transaction submission priority; if the preset data consistency mode is repeatable, determining the processing method for the input data corresponding to each transaction corresponding to the task instance as follows: based on the transaction submission progress, clearing all the input data corresponding to each transaction corresponding to the task instance that has been historically written in the data source corresponding to the sequence number, and then re-executing the task instance; determining the first sequence number corresponding to the data table and the second sequence number corresponding to the message middleware in the sequence number, wherein the first sequence number is greater than the second sequence number; and if the preset data consistency mode is processing from the breakpoint, determining the processing method for the task instance to The processing method for the input data corresponding to each transaction is as follows: reading the input data for the data table corresponding to the first sequence number but not writing it to the corresponding data table, reading the input data for the message middleware corresponding to the first sequence number and writing it to the message middleware corresponding to the second sequence number, in response to a successful write, updating the second sequence number to the first sequence number, further updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress; if the preset data consistency mode is de novo processing, determining the processing method for the input data corresponding to each transaction corresponding to the task instance is as follows: based on the transaction commit progress, reading the input data for the message middleware corresponding to the second sequence number and the sequence number before the second sequence number but not writing it to the corresponding message middleware, reading the input data for the data table corresponding to the first sequence number and the sequence number before the first sequence number but not writing it to the corresponding data table, reading the input data for the message middleware corresponding to the first sequence number and writing it to the message middleware corresponding to the second sequence number, in response to a successful write, updating the second sequence number to the first sequence number, further updating the transaction commit progress, and continuing to execute the task instance based on the updated transaction commit progress; performing automatic retry of transaction commit based on the processing method; The output unit is configured to generate and output abnormal information in response to failure of automatic retry of transaction submission, and exit the task thread.

8. A data processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images