A data processing method and system
By setting data marker bits and a global file counter at the processing end, combined with a shared state storage area, the problem of missing data in the last batch in integrated stream and batch data processing is solved, and the reliability and accuracy of data transmission are achieved.
Patent Information
- Application Number
- CN202611637072.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-22
- Publication Date
- 2026-11-17
AI Technical Summary
In integrated batch processing of data, existing technologies cannot accurately determine the end of data transmission, which may result in the last batch of data ending prematurely before it is sent, leading to data transmission failures or duplicate transmissions.
By setting data marker bits and a global file counter on the processing end, the theoretical and actual total consumption is recorded. The production end temporarily stores the data after generating the batch and sends it after the theoretical total consumption is determined. The processing end ends the task when the actual total consumption reaches the theoretical total consumption and synchronizes the data status using the shared state storage area.
This reduces the probability of data transmission failure, prevents the omission and duplicate transmission of data in the last batch, and improves the reliability and accuracy of data processing.
Smart Images

Figure CN123115587A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing and task scheduling technology, specifically to a data processing method and system. Background Technology
[0002] With the continuous development of internet technology, many industries have accumulated massive amounts of business data that need to be processed. Due to real-time requirements, the concept of stream-batch processing has been introduced into the field of big data processing. Stream-batch processing refers to integrating real-time data stream processing and batch processing into a unified framework. For example, Apache Flink is a common open-source distributed processing framework that supports stream-batch processing.
[0003] In the integrated batch processing design, data processing tasks do not read all the data before sending it. Instead, the production end reads the input data while simultaneously forming batches by page, by number of records, or by size, and then hands them over to the subsequent processing end. For example, in a business scenario where raw business data in a database is transformed into analyzable report data through cleaning operations, the production end reads data page by page from the database, continuously receives data from message streams, or continuously reads records from file streams and sends them to the processing end. The processing end then processes the data according to various predefined logics to obtain usable clean report data. Typically, when the production end starts reading data, the system cannot know how many batches the object data of this processing task will ultimately form.
[0004] In existing technologies, data is typically sent immediately upon batch formation at the production end, while the processing end receives the data and performs logical processing. The production end can communicate with the data source to determine whether all data for the current task has been sent, while the processing end does not communicate directly with the data source and generally uses indirect methods to determine whether the current task has finished sending data. For example, when the processing end cannot retrieve new batch data at a certain point in time, it will initiate a task end judgment branch, reading a preset task end judgment condition: if the data queue is empty and has remained empty for XX time, then the data stream is judged to have ended; once the processing end determines that the current situation meets the preset condition, it will judge that the task has ended. However, in practical applications, it has been found that this sending and receiving method may lead to incorrect judgments. For example, if the last batch is still being generated and has not yet been sent at the production end, it may be delayed in sending due to some reason. If the processing end only judges the end based on "the queue is temporarily empty", it will miss the last batch of data, resulting in data sending errors. In some scenarios, repeated callbacks or retries may also cause data to be sent repeatedly, leading to data sending errors. Summary of the Invention
[0005] To address the problem in existing technologies where the total number of batches is unknown beforehand, leading to premature termination of subsequent processing and data transmission failures before the final batch is sent during streaming batch generation, this invention provides a data processing method that can reduce the probability of data transmission failure during streaming batch generation. A data processing system is also disclosed herein.
[0006] The technical solution of the present invention is as follows: a data processing method, characterized in that it includes the following steps: S1: Set the data flag bit for the processing end; The data marker bit is used to record the theoretical consumption of the processing end, and is initially empty; Establish a global file counter, which includes: theoretical total consumption and actual total consumption; The theoretical total consumption is initialized to sendfileCount=0 for the processing end's data consumption volume; the theoretical total consumption is calculated based on the total number of sending batches from the production end; Initialize the actual total consumption to 0 for the processing end; the actual total consumption is calculated based on the actual total processing volume of the processing end. S2: The production end continuously reads input data and generates the i-th batch of data; i≥1; S3: After each batch is generated on the production side, perform the following operations; Count the total number of batches that have been generated so far, calculate the theoretical total consumption, and assign the value to sendfileCount; S4: Determine if i is greater than 1; If so, proceed to step S5; If not, after placing the newly generated i-th batch of data in the temporary storage area, proceed to step S6; S5: Send the (i-1)th batch of data to the processing end; Place the newly generated i-th batch of data in the temporary storage area; S6: Determine whether all input data for this processing task has been completed; If the input data has been completed, proceed to step S7; If the input data has not yet ended, repeat steps S2 to S5. S7: After assigning the value of sendfileCount to the data flag bit, determine whether sendfileCount is 0; If sendfileCount is 0, no batch data will be sent, and the current processing task will end. If sendfileCount is not 0, send the batch data in the temporary storage area to the processing end; S8: The processing end confirms whether the data flag bit is empty; If the data flag is empty, wait to receive the batch data from the sending end; after each batch of data is received, perform the following operations: Perform predetermined logical processing on the received batch data; count the batch data that has been processed by the processing end currently, calculate to obtain the actual total consumption and assign it to processfileCount; If the data flag is not empty, proceed to step S9; S9: The processing end reads the value DM in the data flag bit; Determine whether the value DM in the data flag bit is 0; If it is 0, directly determine that this processing task is ended; If DM is not 0, proceed to step S10; S10: Compare the value DM in the data flag bit with processfileCount; If DM=processfileCount, and all logical processing tasks on the current processing end are completed, determine that the current processing task is ended; If DM<processfileCount, proceed to step S11; If DM>processfileCount, issue an abnormality alarm; S11: Wait to receive the batch data from the processing end, and time the waiting time WaitT; Compare the waiting time WaitT with the preset waiting time threshold WaitD; When WaitT>WaitD, issue an abnormality alarm; If batch data is received when WaitT≤WaitD, perform predetermined logical processing on the received batch data; count the total number of batches that have been processed by the processing end currently, calculate to obtain the actual total consumption and assign it to processfileCount, then proceed to step S10.
[0007] It is further characterized in that: A shared state storage area is provided between the production end and the processing end, and both the production end and the processing end can access the shared state storage area simultaneously; The data flag bit is stored in the shared state storage area; The calculation methods for the theoretical total consumption and the actual total consumption include the following cases: Scenario 1: The processing end includes only one consumption branch, and the generating end sends a complete batch of data to the branch for consumption each time; a data flag is set for the consumption branch of the processing end; Theoretical total consumption sendfileCount = total number of batches sent from the production end; Actual total consumption (processfileCount) = Total number of batches processed at the processing end; Scenario 2: The processing end has multiple consumption branches. The production end sends a complete batch of data to the processing end for consumption each time, and a data marker bit is established for each consumption branch. Theoretical total consumption sendfileCount = Total number of batches sent from the production side to each branch; Actual total consumption processfileCount = Total number of batches processed in each branch of the processing end; Scenario 3: The processing end has multiple consumption branches. After the production end reads a batch of data, it sends a single data item from the batch to the processing end for consumption. Each branch consumes a single data item at a time. A data marker is created for each consumption branch. Theoretical total consumption sendfileCount = m1 + ... + m j +…+m n n represents the total number of batches sent from the production end, and m represents the total number of batches sent from the production end. j Let j be the total amount of data included in the j-th batch; The actual total consumption at the processing end, processfileCount, equals the sum of the number of data items processed in each branch. Data updates in the shared state storage area are performed atomically. The shared state storage area also stores: sending state and processing state; The sending status is used to mark whether the data has been sent. The sending status includes: not sent and sent, with the default status being not sent. The production end generates the sending status of the data synchronously when generating each batch of data. After the data is successfully sent to the processing end, the sending status of the data is modified to sent. Each time the processing terminal receives data, it generates a processing status for the received data. The processing status includes: unprocessed, in progress, and processed, with the default status being unprocessed. After processing the data according to a preset logic, the processing terminal sets the data to in progress, and after processing is completed, it modifies the processing status of the data to processed. When the production end sends data, a unique identifier is constructed for this data transmission; the transmission status and the processing status in the shared status storage area are distinguished by the unique identifier. After receiving the data, the processing terminal first confirms whether the data has already been consumed. Data that has already been consumed will not be received again. In step S6, the methods for determining whether all input data for this processing task has been completed include: The execution is triggered when the end of the file is reached, the database query range has been exhausted and there is no next page of data, the message stream receives an end message, or all input sources within the current execution range have ended.
[0008] A data processing system, characterized in that it comprises: a production end, a processing end, and a shared state storage area; The production end connects to the data source to collect input data; the production end and the processing end communicate with each other; the production end and the processing end are respectively connected to the shared state storage area; The production end includes: a batch formation module, a temporary storage module, a sending module, and an end confirmation module; The batch generation module is used to communicate with the data source, continuously read input data, and generate a theoretical total consumption. The temporary storage module is used to temporarily store the latest batch of data. The sending module is used to send the previously stored batch when a new batch is formed, and to send the latest batch that is currently stored after the theoretical consumption total is written. The end confirmation module is used to confirm that the input data has ended, and after the input data ends, to calculate the actual theoretical consumption total and write it into the shared state storage area; The processing terminal includes: a receiving module, a logic processing module, and a completion judgment module; The receiving module is used to receive batch data from the production end, and at the same time to count the total actual consumption that the processing end has processed. The logic processing module is used to process each batch of data according to a preset logic; The completion judgment module is used to determine whether the theoretical total consumption has been determined, and whether the number of batches sent and the actual consumption that has been processed have reached the theoretical total consumption; and to confirm whether the task has been completed based on the judgment results.
[0009] This application provides a data processing method and system that sets a data flag bit for each processing task. Whether the data flag bit is empty indicates whether the theoretical total consumption of the current processing task has been generated. After each processing task begins execution, an empty data flag bit indicates that the theoretical total consumption is in an undetermined state. While the production end continuously reads data and forms batches, it constantly counts the number of batches generated and calculates the theoretical total consumption. During the period when the theoretical total consumption of each processing task is undetermined, the latest generated data only enters a temporary storage area and is not immediately sent. When a new batch of data is generated subsequently, the previously temporary batch of data is sent, and the new batch replaces the temporary batch. When the input data stream reaches the termination condition, the production end counts the actual theoretical total consumption, writes the theoretical total consumption to a readable status position on the processing end, and then sends the last temporarily stored batch of data. The processing end only terminates when the theoretical total consumption is determined, the number of sent data reaches the total batch size, and the actual consumption processed reaches the theoretical total consumption. When the theoretical total consumption is undetermined, even if there is no batch data to be processed in the sending channel, the current processing is not marked as terminated, ensuring that no last batch of data is missed. This method sets a sending status and a processing status for each data transmission, preventing duplicate data transmission or processing based on the content of these two status markers. The method is simple in logic, easy to implement, and solves the problem at a low cost, making it highly practical. Attached Figure Description
[0010] Figure 1 This is a flowchart of the data processing method in this application; Figure 2 This is a diagram of the data processing system architecture in this application; Figure 3 Multi-input task streaming reading example 1; Figure 4 This is an example of the configuration page in Example 1; Figure 5 This refers to the data processing flow in Example 1; Figure 6 This is an example of the parent process in the parent-child process of Example 2; Figure 7 This is an example of the page configuration for the parent process in Example 2; Figure 8 This is an example of a child process in the parent-child process of Example 2; Figure 9 This is an example of page configuration for a sub-process in Example 2; Figure 10 This is the data processing flow in Example 2. Detailed Implementation
[0011] This method can be applied to any scenario where input data needs to be delivered to subsequent processing units in batches, such as data cleaning, batch processing, and message processing. Figure 1 As shown, this application includes a data processing method, which includes the following steps.
[0012] S1: The execution of each processing task includes: the production end and the processing end; Set data flags for the processing end; the data flags are used to record the theoretical consumption of the processing end, and are initially empty; Establish a global file counter, which includes: theoretical total consumption and actual total consumption; To initialize the theoretical total consumption volume for the processing end, sendfileCount = 0; the theoretical total consumption volume is calculated based on the total number of sending batches from the production end. Initialize the actual consumption total to processfileCount=0; the actual consumption total is calculated based on the total number of batches processed.
[0013] In practical business applications, there may be multiple consumption branches in the processing end. The total amount of batch data generated by the production end is not necessarily the same as the actual consumption amount in the processing end. In order to facilitate statistics, this application establishes a global file counter for each consumption branch in the processing end.
[0014] When the processing end has only one consumption branch, the generating end sends a complete batch of data to the branch for consumption each time; each batch of data sent by the generating end to the processing end is consumed by the processing end in one go, so the total number of batches generated by the generating end is the amount consumed by the processing end in one go; therefore, it is sufficient to set a data flag bit for the processing end.
[0015] For example: The production end reads data from the database, reading 5 records at a time, for 3 batches. The production end sends one batch of data to the processing end each time, and the processing end processes one batch of 5 records at a time, for a total of 3 batches. In this scenario, the total number of batches processed by the production end can be used as the theoretical total consumption, and the total number of batches processed by the processing end can be used as the actual total consumption: Theoretical total consumption sendfileCount = Total number of batches sent by the production end.
[0016] However, when there is more than one branch in the processing end, or when the consumption method within the branch does not treat the entire batch of data as a single consumption, it is necessary to calculate the theoretical consumption volume of the processing end based on the batch quantity or the amount of data generated by the production end, and then compare it with the actual consumption volume of the processing end in order to accurately determine whether the processing end consumption has ended.
[0017] Scenario 2: There are multiple consumption branches on the processing end. The production end sends a complete batch of data to the processing end for consumption each time, and a data marker is established for each consumption branch.
[0018] For example, in Example 1, the production end reads data from the database, sending a complete batch of data to two branches each time. Each of the two branches then performs logical processing on a single complete batch of data. Therefore, for each consumer branch: Theoretical total consumption sendfileCount = Total number of batches sent from the production side to each branch; Actual total consumption processfileCount = Total number of batches processed by each branch on the processing end.
[0019] Scenario 3: The processing end has multiple consumption branches. After the production end reads a batch of data, it sends a single data item from the batch to the processing end for consumption. Each branch consumes a single data item at a time. A data marker is created for each consumption branch. For example, in Example 2, the production end reads data from the database, generating a total of n batches of data; the j-th batch of data includes m j Instead of sending each batch of data at once, the production end sends one piece of data at a time to each of the two processing branches, with each branch consuming one piece of data at a time. Therefore, for each processing branch: Theoretical total consumption sendfileCount = m1 + ... + m j +…+m n n represents the total number of batches sent from the production end, and m represents the total number of batches sent from the production end. j Let j be the total amount of data included in the j-th batch; The actual total consumption at the processing end, processfileCount, equals the sum of the number of data items processed in each branch.
[0020] When there are multiple consumption branches in the processing stage, the system monitors the execution status of each consumption branch by comparing the theoretical total consumption with the actual total consumption. If all consumption branches complete normally, it means that the current processing task has completed normally.
[0021] S2: The production end continuously reads input data and generates the i-th batch of data; i≥1.
[0022] In this method, batch data can be a data file, a memory object, or a message body. A file is merely a specific implementation and is not a necessary condition for this application. In specific implementations, batch data includes: data files, memory objects, or message bodies formed according to a fixed number of records, a fixed data volume, database pagination, message windows, or business rules.
[0023] S3: After each batch is generated on the production side, perform the following operations; Count the total number of batches that have been generated so far, calculate the theoretical total consumption, and assign the result to sendfileCount.
[0024] S4: Determine if i is greater than 1. If it is, proceed to step S5. If not, place the newly generated i-th batch of data in the temporary storage area and then proceed to step S6.
[0025] When input data is continuously read and batches are formed, the latest batch is saved in the temporary storage area and not sent immediately; when a new batch is formed, the previous batch saved in the temporary storage area is sent, and the newly formed batch is replaced with the latest batch in the temporary storage area.
[0026] S5: Send the (i-1)th batch of data to the processing end; Place the newly generated i-th batch of data in the temporary storage area.
[0027] S6: Determine whether all input data for this processing task has been completed; If the input data has been completed, proceed to step S7; If the input data has not yet finished, repeat steps S2 to S5.
[0028] Specific methods for determining whether all input data for this processing task has been completed include: The execution is triggered when the end of the file is reached, the database query range has been exhausted and there is no next page of data, the message stream receives an end message, or all input sources within the current execution range have ended.
[0029] S7: After assigning the value of sendfileCount to the data flag bit, check whether sendfileCount is 0; If sendfileCount is 0, no batch data will be sent, and the current processing task will end. If sendfileCount is not 0, then the batch data in the temporary storage area is sent to the processing end.
[0030] In practical applications, some tasks may have batch data of 0, which requires separate judgment and handling.
[0031] In some visualized cleaning processes, a single execution may involve multiple input plans. These input plans can be read sequentially or independently. Multiple inputs are just one common scenario in the aforementioned problem: regardless of whether the input source is one or multiple, as long as the theoretical total consumption is unknown before the data is read, the last batch may face the risk of premature termination.
[0032] Therefore, in this method, when a processing task includes a plurality of task branches, the theoretical total consumption recorded in the data flag bit is the sum of the theoretical total consumption of all task branches; meanwhile, a branch consumption quantity counter is respectively constructed for each task branch, for recording the total actual consumption quantity processed by each task branch.
[0033] The "end of input stream" of each processing task in the present application is not limited to the completion of reading a single file: in the case of single input, it may mean that the input source reaches the end of the stream; in the case of multiple inputs, it may mean that all input sources within the current execution range are completed.
[0034] S8: The processing end confirms whether the data flag bit is empty; If the data flag is empty, wait to receive batch data from the sending end; after receiving each batch of data, perform the following operations: Perform predetermined logical processing on the received batch data; count the total number of batches that the processing end has currently processed, calculate to obtain the actual total consumption, and assign it to processfileCount; If the data flag is not empty, perform step S9; S9: The processing end reads the value DM in the data flag bit; Determine whether the value DM in the data flag bit is 0; If it is 0, directly determine that the current processing task is ended; If DM is not 0, perform step S10; S10: Compare the value DM in the data flag bit with processfileCount; If DM=processfileCount, and all logical processing tasks at the current processing end are ended, it is determined that the current processing task is ended; If DM<processfileCount, perform step S11; If DM>processfileCount, issue an abnormality alarm; S11: Wait to receive batch data from the processing end, and time the waiting time as WaitT; Compare the waiting time WaitT with a preset waiting time threshold WaitD; When WaitT>WaitD, issue an abnormality alarm; If WaitT≤WaitD and batch data is received, perform predetermined logical processing on the received batch data; count the total number of batches that the processing end has currently processed, calculate to obtain the actual total consumption and assign it to processfileCount, then perform step S10.
[0035] A shared state storage area is set up between the production end and the processing end, and both the production end and the processing end can access the shared state storage area simultaneously; data flag bits are stored in the shared state storage area.
[0036] When it is confirmed that the input data has ended, the theoretical total consumption generated in this process is calculated and written into a status position that can be read by the subsequent processing end. Then, the latest batch in the temporary storage area is sent. The subsequent processing end determines that the current processing is over when the batch number is determined, the number of batches sent reaches the batch number, and the number of batches processed reaches the batch number.
[0037] In practical applications, repeated callbacks or retries may occur during data transmission for various reasons. For example, the same batch may re-enter the transmission logic due to network retries, task recovery, or duplicate notifications. Without transmission status checks, the same batch may be repeatedly placed in the queue, leading to subsequent duplicate processing. To address this issue, this method sets transmission and processing statuses for batch data in the transmission task.
[0038] Specifically, the sending status and processing status are stored in a shared status storage area; The sending status records whether the data has been sent. The sending status includes: not sent and sent, with the default status being not sent. The production end generates the sending status of the data synchronously each time a batch of data is generated. After the data is successfully sent to the processing end, the sending status of the corresponding data is changed to sent. Each time the processing end receives data, it generates a processing status for the received data. The processing status includes: unprocessed, in progress, and processed, with the default status being unprocessed. After the processing end processes the data according to the preset logic, it sets the status to in progress. After the processing is completed, it changes the processing status of the data to processed.
[0039] Each time the production end sends data, it constructs a unique identifier for the data being sent. Specifically, since the production end and the processing end communicate based on messages, and each message has a unique identifier, the message ID used when sending the data is used as the unique identifier for this data transmission.
[0040] The sending and processing states in the shared state storage area are distinguished by this unique identifier. When the processing end receives data, it first checks whether the data has already been consumed; data that has already been consumed will not be received again.
[0041] In actual business applications, the consumer side includes Flink and MQ (Message Queue). Regardless of the implementation method, the received data will be checked. For example, before sending data, the MQ producer will send a unique identifier (messageID) to the MQ. After the MQ polls and listens for this message, it will determine whether the data has already been consumed based on the messageID and other methods. If it has been consumed, it will not be received again. The specific judgment method is implemented based on existing technologies.
[0042] Before sending data, the Flink producer first sends a unique identifier, `arrowMessageItemKey`, to Redis. After retrieving the batch messages from the Redis queue, Flink determines whether the data has already been consumed based on `arrowMessageItemKey` and other methods. If it has been consumed, it does not receive any more data. The specific determination method is implemented based on existing technology.
[0043] In this method, data updates in the shared state storage area are performed atomically. That is, updates to the batch quantity, sending status, and processing status are all performed atomically to avoid duplicate sending, duplicate counting, or premature termination.
[0044] Based on this method, at the start of each execution, the theoretical total consumption is marked as undetermined. During the period when the theoretical total consumption is undetermined, the latest generated data only enters the temporary storage area and is not sent immediately. When a new batch of data is generated later, the previously temporary batch of data is sent, and the new batch replaces the temporary batch. When the input stream reaches the termination condition, the actual theoretical total consumption is calculated, written to a readable status position on the processing end, and then the last temporary batch of data is sent. The processing end only terminates when the theoretical total consumption is determined, the number of sent data reaches the total batch size, and the actual consumption processed reaches the theoretical total consumption. This method does not depend on the input quantity or limit the file carrier. It temporarily stores the latest batch while continuously forming batches and sends the previous batch when the next batch is received. After the input stream truly ends, the theoretical total consumption is written first, and then the last temporary batch of data is sent, so that the subsequent processing end can reliably determine completion based on this.
[0045] like Figure 2 As shown, in order to implement the above data processing method, this application constructs a data processing system, which includes: a production end, a processing end, and a shared state storage area; The production end connects to the data source to collect input data; there is a communication connection between the production end and the processing end; the production end and the processing end are each connected to a shared state storage area; The production side includes: batch formation module, temporary storage module, sending module, and end confirmation module; The batch generation module is used to connect to the data source and continuously read input data to generate batch data; The temporary storage module is used to temporarily store the latest batch of data. The sending module is used to send the previously stored batch when a new batch is formed, and to send the latest currently stored batch after the actual batch number is written. The end confirmation module is used to confirm that the input data has ended, and to calculate the actual theoretical consumption total after the input data ends and write it to the shared state storage area; The processing unit includes: a receiving module, a logic processing module, and a completion judgment module; The receiving module is used to receive batch data from the production end, and at the same time, to count the total actual consumption that has been processed by the processing end. The logic processing module is used to process each batch of data according to preset logic; In the completion judgment module, when the theoretical total consumption has been determined, and the number of batches sent and the actual consumption total processed have both reached the theoretical total consumption, the task processing is confirmed to be complete.
[0046] The temporary storage module temporarily stores batch data, while the shared state storage area stores the batch quantity, sending status, and processing status. In one specific implementation, the temporary storage module and the shared state storage area are based on Redis, message middleware state storage, or other shared state storage. The corresponding content in the logic and methods of each module is the same.
[0047] The specific process of this method is illustrated below based on two embodiments.
[0048] Example 1: Multi-input task streaming reading (multiple input nodes) business scenario.
[0049] Scenario Description: Multiple departments need to periodically cleanse their penalty data from their original penalty tables (input nodes) using cleaning rules, and then transfer it to their business penalty tables (output nodes). Each department's original penalty data is stored in original penalty tables, which include: an administrative penalty table for natural persons and an administrative penalty table for legal persons. While the table structures of these two tables may differ, some fields are identical, and the validation rules are consistent. For example, case and violation information includes: administrative penalty decision number, type of violation, facts of the violation, basis for penalty, penalty category, and penalty content. These fields are identical in both tables, and the validation rules are consistent. However, the two tables differ in several fields. For instance, most fields in the administrative penalty tables for natural persons and legal persons are the same, with only a few fields differing. The legal person table has a unified social credit code, while the natural person table has an identity number field. The validation rules for the social credit code and the identity number are inconsistent.
[0050] In this scenario, both the Natural Person Administrative Penalty Form and the Legal Person Administrative Penalty Form are used as input nodes, forming a multi-input task streaming read; the specific implementation is based on the Apache Flink framework.
[0051] like Figure 3 As shown, the current process execution route is as follows: Branch 1: Legal Entity Table Input - Unified Social Credit Code - Required Field Validation - Rule Validation - Correct Penalty Information Table; Branch 2: Natural Person Table Input - Identity Number Verification - Required Field Verification - Rule Verification - Correct Penalty Information Table.
[0052] The configuration page for the business scenario in this embodiment is as follows: Figure 4 As shown, the specific implementation process is as follows: Figure 5 As shown.
[0053] The production system received a cleaning task message with task number 1001.
[0054] Specific methods for initiating cleaning task messages include: Users can manually send a message to the MQ by clicking a button, or the system can periodically send cleanup messages to the MQ. The MQ listens for the message and then begins consuming it. The message body includes data such as the messageId and the DSL (Domain Specific Language) for process orchestration.
[0055] 2. The production system reads the process DSL and identifies two input branches: Branch batch01: Input from the legal entity table, SQL is select * from jc_sgs_xzcf_fr; Branch batch02: Input from the natural person table, the SQL is select * from jc_sgs_xzcf_zrr.
[0056] 3. The production system creates two branch states in Redis: task:1001:branch:batch01; task:1001:branch:batch02; The DSL of the process is queried based on the message ID. For example, if the message ID is 1001, a total cleaning task task001 is generated. At the beginning, the status of both tasks is "reading" and the total number of files has not yet been determined.
[0057] The production system calculates the number of branches based on DSL, and writes an execution branch log to Redis for each branch to record the total number of files, the total amount of cleaning in the current branch, the execution time, the execution status, etc. Allocate a global file counter variable, sendfileCount, in memory for each branch.
[0058] Create a data tag bit in the Redis shared state store for each branch.
[0059] 4. Batch01 reads the legal entity table in batches of 1000 records. After reading the first 1000 records, a batch file is generated; in this embodiment, the batch file is a columnar file with the .arrow extension, named with messageid + branch number + batch number, and written to disk. 1001_batch01_01.arrow; The production system temporarily stores the file and does not immediately send it to Redis for processing.
[0060] For each batch of files produced, the production system increments sendfileCount by 1 and records it in memory.
[0061] 5. Batch01 continues reading the legal entity table, and after reading another 1000 records, it generates: 1001_batch01_02.arrow; The production system writes the previous temporary file 1001_batch01_01.arrow to the Redis queue; then it sets 1001_batch01_02.arrow as the current temporary file. The sending status of 1001_batch01_01.arrow is updated to "sent".
[0062] 6. Batch01 continues reading, only 347 records are read, generating: 1001_batch01_03.arrow; The production system writes 1001_batch01_02.arrow to a Redis queue; then it sets 1001_batch01_03.arrow as the current temporary file. The sending status of 1001_batch01_02.arrow is updated to "sent".
[0063] 7. Batch01 attempted to read again but found no data. The production system confirmed that batch01 had ended and that 1001_batch01_03.arrow was the last file of that branch.
[0064] 8. The production system writes the sendfileCount stored in memory, which is the actual total number of files in batch 01, into the data flag in Redis: task:1001:branch:batch01. sendfileCount = 3; Then, the last file, 1001_batch01_03.arrow, is written to the Redis queue; Update the sending status of 1001_batch01_03.arrow to "sent".
[0065] 9. Batch02 reads the natural persons table in batches of 1000 records. After reading the first 1000 records, the following is generated: 1001_batch02_01.arrow The system temporarily saves this file.
[0066] 10. Batch02 continues reading the natural persons table, only reading 586 records, generating: 1001_batch02_02.arrow The system writes 1001_batch02_01.arrow to the Redis queue and updates the sending status, and temporarily stores 1001_batch02_02.arrow.
[0067] 11. Batch02 attempted to read again but found no data. The production system confirmed that batch02 had finished and first wrote the value of sendfileCount to the data flag bit: task:1001:branch:batch02. sendfileCount = 2; Then write the last file 1001_batch02_02.arrow to the Redis queue and update the sending status.
[0068] 12. File task messages in Redis, for example: messageId=1001; branchId=batch01; batchNo=01; arrowFilePath=1001_batch01_01.arrow.
[0069] At this point, sendfileCount for both input branches has been written to the data flags. If the data flags for both branches are not empty, it indicates that the input data stream has ended.
[0070] 13. Flink, acting as the processing end, retrieves task messages from the Redis partition queue, reads the corresponding Arrow file, and performs cleaning. For each file successfully processed, it updates the number of consumed files in the corresponding branch and sets the processing status of the Arrow file to "processed."
[0071] Specifically, Flink retrieves unconsumed Redis messages through a combination of polling and short-term blocking waits. The `redis_to_flink` message contains data such as the arrow file path, message ID, and batch ID. The processing end locates and reads the arrow file based on the file path name in the message, executes each cleaning node, and finally stores the cleaning results in memory. Each time an arrow file is successfully processed, the Flink task in Redis is marked as consumed, and the number of consumed files is accumulated.
[0072] 14. When the total number of files in batch01 is 3, the number of files enqueued is 3, and the number of files consumed is 3, the system will update the processing status of branch batch01: task:1001:branch:batch01.status = Completed; 15. When the total number of files in batch02 is 2, the number of files enqueued is 2, and the number of files consumed is 2, the system will update the processing status of branch batch02: task:1001:branch:batch02.status = Completed; 16. The system checks the status of the two branches of task 1001. After confirming that both batch 01 and batch 02 have been completed, the processing status of the overall task 1001 is updated. task:1001.status = Completed.
[0073] Example 2 is a parent-child process business scenario (the child process is executed cyclically based on the query results of the parent process).
[0074] Scenario Description: A raw penalty table (input node) exists in the database, storing penalty data for all departments. Penalty data is identified by a department code field. Periodically, penalty data needs to be cleaned from the raw penalty table (input node) of each department and then transferred to the business penalty table (output node) of each department using cleaning rules. The business penalty tables of each department have the same structure and naming convention, such as JC_SGSXZCF_001, JC_SGSXZCF_002… where 001 and 002 are department codes.
[0075] When the number of departments exceeds one hundred, the amount of data processed in each data cleaning task becomes enormous. Setting up a separate cleaning task for each department is inefficient and cumbersome, requiring a separate cleaning process for each department. To improve data cleaning efficiency and reduce implementation difficulty, this embodiment configures the data cleaning task as a parent and child process structure. The parent process queries the set of department codes and generates the corresponding form name for the business database penalty table based on the department codes. Then, the department codes and the corresponding business database penalty table names are passed as variables to the child process in a loop. For example... Figure 7 The image shows an example of the configuration page for the department and business table input nodes in the parent process, as well as a preview page showing the correspondence between the department and business tables. Figure 6 The embodiment shown includes two sub-processes: a legal person penalty sub-process and a natural person penalty sub-process.
[0076] In the sub-process, after receiving each department code variable from the parent process, the pre-defined cleaning logic is executed separately for each department code to obtain the correct penalty information data. Finally, the pre-stored form in the system is located according to the name of the penalty table in the business database input by the parent process, and the data after all logic processing is executed is stored in the corresponding form. For example... Figure 9 The image shows an example of the input node configuration page and output node configuration page for the original table of a sub-process. Figure 8 As shown, the logical processing nodes that need to be executed in the specific sub-process in this embodiment include: legal person table input, required field validation, data validation, and code validation.
[0077] In this embodiment, the process execution route is as follows: Branch A: Query the set of sub-process departments - legal person penalty sub-process - specific legal person sub-process and clean all nodes; Branch B: Query the sub-process department set - natural person penalty sub-process - specific natural person sub-process clean all nodes.
[0078] like Figure 10 As shown, the execution steps are as follows.
[0079] Step 1: Query the sub-process department set dept_table, and execute its input node: select *from dept_table; This node reads departmental data in batches of up to 1000 records. The first record in the preview data is: TABLE_NAME: YW_L_SGSXZCF5; DEPT_CODE: 008; ID: 8.
[0080] Step 2: The user manually clicks to execute, or the system triggers it periodically, sending a message from the parent process to the MQ; the main content of the parent process message includes: parent messageid, process orchestration DSL, etc.
[0081] In this embodiment, the parent messageid is set to parent-1001; that is, the message of the parent process parent-1001 is sent to the MQ.
[0082] Step 3: After the MQ listens for the parent process message, it consumes the parent process message, queries the parent process DSL based on parent-1001, and generates the parent process task task001.
[0083] Step 4: The parent process task 001 parses the DSL and analyzes it to obtain the number of branches of the child process tasks. The parent process allocates a global file counter variable sendfileCount in memory for each branch; at the same time, it also establishes a data flag bit for each branch in the Redis shared state storage area.
[0084] In this embodiment, two parallel sub-process branches are obtained: Branch A (Legal Entity Branch): Parent-child process of legal entities; Branch B (Natural Person Branch): Natural Person Parent-Child Process.
[0085] The parent process parses the DSL, calculates branch paths, and supports multiple sub-processes serially or in parallel. This example assumes two sub-processes running in parallel, resulting in two branches. The parent process of each branch acts as the production end for processing tasks, while the sub-processes act as the processing end. The parent process generates a batch of data based on the data set of department codes read from the input nodes. However, it doesn't simultaneously send all data from the same batch to both branches; instead, it extracts each department code one by one before sending it to both branches. For example, batch 1 contains 5 department data entries with department codes 001, 002, 003, 004, and 005. The parent process generates a corresponding business database penalty table filename JC_SGSXZCF_001 for department code 001. Then, 001 and JC_SGSXZCF_001 form the first group of department data, which is sent to both branches simultaneously. After that, a second group of department data is generated: 002 and JC_SGSXZCF_002, which are sent to both branches simultaneously… Assumptions: Each batch of data includes N (N≥1) department codes. Each group of department data is simultaneously sent to two branches, and a sub-process is established for each group of department data in each branch. In this embodiment, for each batch of data in the parent process, the actual amount of data consumed by each branch is N, and the total amount of data consumed by the two sub-processes is 2N. That is, in this embodiment, the theoretical total consumption counted by the global file counter variable sendfileCount allocated to each branch by the parent process is the same as the number of sub-processes established in each branch. Therefore, in this embodiment, the theoretical total consumption of each branch is calculated using the number of sub-processes (batch - A / B_child_count).
[0086] The parent process internally constructs a parent process filecount and a parent process number counter. The parent process filecount is used to count the number of batches within the parent process. The parent process branch counter is used to calculate the number of branches after parsing the parent process task DSL and calculating the branch path. Then, the number of subprocesses is calculated based on the number of branches and stored in the process number counter.
[0087] The parent process also has a polling listener set up. Before the parent process processes the last piece of data in the last batch, it sends a polling penalty message to the polling listener to poll the branch status.
[0088] After the parent process reads the batch data and confirms that the batch data is not empty, and also determines that this batch of data is not the last batch of data, it reads the first department code data of this batch, generates the business database penalty table file name, and then does not send the message for the first group of department data to the branch. Instead, it continues to read the second department code data. After confirming that the previous group of data is not the last group of data in this batch, it sends the first group of department data to both branches. The following loop processing is performed for each group of department data in each batch of data.
[0089] Step 5: After the parent process reads the first department data with department code dept_code 008, it simultaneously sends it to two branches, each creating a sub-process: Create child-A001 in the legal entity branch; The natural person branch creates child-B001.
[0090] Both sub-process records store the parent process message ID, the branch it belongs to, and the sub-process ID, with the initial state being PENDING. Simultaneously, the number of created sub-processes (batch-A / B_child_count) for both the legal entity branch and the natural person branch increases by 1.
[0091] Step Six: The parent process maps fields such as dept_code=008 and table_name=YW_L_SGSXZCF5 from the current department data to variables in the child process, and sends them to the legal entity child process (child-A001) and the natural person child process (child-B001) respectively. After the child processes start, their status changes to RUNNING.
[0092] After a branch, acting as a processing end, obtains any set of departmental data, it retrieves the corresponding field values from the current data based on the field mapping relationship configured in the subprocess node, and passes the field values as input variables to the corresponding subprocess. Each branch generates a subprocess cleaning task for each piece of data in the batch and sends it to MQ, where MQ consumes and executes the corresponding subprocess cleaning steps.
[0093] Step 7: After receiving dept_code=008, the legal entity sub-process executes the actual cleansing SQL statement in its "Legal Entity Table Input" node: select*fromjc_sgs_xzcf_frwheredept_code=${dept_code}; Therefore, the actual query retrieved administrative penalty data for the legal entity with department code 008. The query results sequentially passed through nodes such as "Required Field Validation," "Data Validation," and "Code Validation."
[0094] Step 8: When the legal entity sub-process enters the table output node, it uses the table_name passed from the parent process. In this example, the output table name is YW_L_SGSXZCF5, which is where the verified legal entity data is written.
[0095] In the natural person branch, the natural person sub-process also follows the process in steps seven and eight, performing the corresponding data reading, verification, and output steps according to its own DSL.
[0096] The MQ consumes and executes the sub-process cleaning steps to complete the sub-process. It then checks if the current sub-process has ended; if not, it continues executing the sub-process cleaning steps.
[0097] Step 9: The parent process will not wait for child-A001 and child-B001 to finish, but will continue reading the next department data. For example, if it reads: TABLE_NAME: YW_L_SGSXZCF6; DEPT_CODE: 009; ID: 9; For department code 009, generate a business database penalty table file name, construct a set of department data with 009 and the corresponding business database penalty table file name, and send it to two branches. In the two branches, create child-A002 and child-B002, and accumulate them into the number of created subprocesses (batch-A / B _child_count) of the legal person branch and the natural person branch, respectively.
[0098] If the parent process determines that the current batch of data is the last batch of data in this branch, then it saves the final total number of child processes of the two branches as batch-A / B_child_count, and writes the final total number of child processes as the total batch of this branch into the data flag bit in the shared state storage area.
[0099] Step 10: Assume the first batch of data reads 5 department codes from `dept_table`, and the second batch reads 5 department codes. The parent process reads data, confirming the second batch as the last batch, and sends the data to both branches one by one. After sending the fourth data, it sends a polling message to the polling listener to monitor the completion status of each branch and each subprocess.
[0100] Meanwhile, after sending the fourth data, the parent process will calculate and solidify the final total number of child processes for the two branches (batch-A / B _child_count), and then send the last department data to the two branches. The total number of sub-processes in the legal entity branch is batch-A_child_count=5*2=10; The total number of subprocesses in the natural person branch is batch - B_child_count = 5 * 2 = 10.
[0101] At the same time, mark both branches as "No more new subprocesses". Before this, even if the number of created and completed processes in a branch are temporarily the same, it cannot be determined that the branch has ended.
[0102] Once the parent process has completed reading all input data, it writes the theoretical total consumption (10 records) for each branch into the data flag in Redis. Then, it sends the message corresponding to the last department's data in the temporary storage area to both branches. After receiving the last department's data, both branches, following the same procedure, generate corresponding sub-process cleaning tasks for each record and send them to the message queue (MQ).
[0103] The parent process determines whether each branch has been completed based on the number of subprocess tasks created, the number of completed subprocess tasks, and the "no more new subprocesses" flag. When all created subprocess tasks have been completed, the branch status is marked as COMPLETED.
[0104] Step 11: For example, after the legal entity data query, verification, and output of child-A001 are completed, the subprocess will mark itself as COMPLETED and increment the number of completed subprocesses in the legal entity branch by 1. The count will not be incremented repeatedly when the same subprocess is called back or retried.
[0105] Step 12: The parent process polls the branch status to check if all child processes have been completed. The legal entity branch has been closed, and all 10 sub-processes have been completed; the legal entity branch is now complete. The individual branch has been closed, and all 10 sub-processes have been completed. The individual branch is now complete.
[0106] Specifically, the parent-child relationship record is found based on the parent process (parent-1001), branch (branch-A), and child process (child-A001). The parent process polls all parent-child relationship records under all branches, and determines that all child processes have been completed when the number of completed states equals the counter for the number of child processes. The parent process monitors the status of each child process and branch in real time through polling to ensure that the actual number of processes consumed at the processing end can be accurately obtained.
[0107] Step 13: Once both branches are completed, all sub-process steps in task001 of the parent process are finished. The entire parent process is marked as complete only after all subsequent nodes in the parent process have also been executed.
[0108] Using the technical solution of this application, during the streaming generation stage where the total amount is unknown, the latest batch data is only temporarily stored and not sent; when a new batch appears, the previous batch is sent; after the input is confirmed, the actual total amount is first written to a status position that can be read by the processing end, and then the last batch is sent; the processing end simultaneously checks the determined total amount, the number of items sent, and the number of items processed; for duplicate notifications, network retries, or task resumptions of the previous and last batches, a sending status check is performed first. This can avoid premature termination or duplicate processing due to the queue being temporarily empty, the last batch not being sent, or duplicate callbacks.
[0109] This application does not limit the specific method of batch boundaries. Batches can be formed according to a fixed number of rows, a fixed number of bytes, database pagination, message windows, or business rules. The sending queue can be a memory queue, a message middleware queue, or other channels capable of transmitting batches.
[0110] This application does not limit the specific type of input source. The input source can be a single database query, a single file stream, a message stream, or multiple database queries, multiple file streams, or multiple process nodes. In a single-input scenario, the end of the input source is taken as the total closure point; in a multi-input scenario, the end of all input sources within the current execution scope is taken as the total closure point.
Claims
1. A data processing method, characterized by, It includes the following steps: S1: Set a data flag bit for the processing end; The data flag bit is configured to record the theoretical consumption amount of the processing end, and is empty initially; Create a global file counter, where the global file counter includes: a total theoretical consumption amount and a total actual consumption amount; Initialize the total theoretical consumption amount sendfileCount=0 for the data volume consumed by the processing end; the total theoretical consumption amount is calculated based on the total number of sending batches of the production end; Initialize the total actual consumption amount processfileCount=0 for the processing end; the total actual consumption amount is calculated based on the total actual processed volume of the processing end; S2: The production end continuously reads input data and generates the i-th batch of data; i≥1; S3: After the production end generates each batch, perform the following operations; Count the total number of batches that have been generated currently, calculate to obtain the total theoretical consumption amount and assign it to sendfileCount; S4: Determine whether i is greater than 1; If yes, execute step S5; If not, place the newly generated i-th batch of data in the temporary storage area, and then execute step S6; S5: Send the (i-1)-th batch of data to the processing end; Place the newly generated i-th batch of data in the temporary storage area; S6: Determine whether all input data of this processing task has ended; If the input data has ended, execute step S7; If the input data has not ended, cyclically execute steps S2 to S5; S7: Assign the value of sendfileCount to the data flag bit, and then determine whether sendfileCount is 0; If sendfileCount is 0, no batch data is sent, and this processing task is ended; If sendfileCount is not 0, send the batch data in the temporary storage area to the processing end; S8: The processing end confirms whether the data flag bit is empty; If the data flag is empty, wait to receive batch data from the sending end; after each batch of data is received, perform the following operations: Perform predetermined logical processing on the received batch data; count the batch data that has been processed by the processing end currently, calculate to obtain the total actual consumption amount and assign it to processfileCount; If the data flag is not empty, execute step S9; S9: The processing end reads the value DM in the data flag bit; Determine whether the value DM in the data flag bit is 0; If it is 0, directly determine that this processing task is ended; If DM is not 0, execute step S10; S10: Compare the value DM in the data flag bit with processfileCount; If DM=processfileCount, and all logical processing tasks of the current processing end are completed, it is determined that the current processing task is ended; If DM<processfileCount, execute step S11; If DM>processfileCount, issue an abnormality alert; S11: Wait to receive batch data from the processing end, and time the waiting time WaitT; Compare the waiting time WaitT with the preset waiting time threshold WaitD; An anomaly alarm is issued when WaitT > WaitD; If batch data is received while WaitT≤WaitD, then the received batch data is processed according to the predetermined logic; the total number of batches that have been processed by the processing end is counted and the actual consumption total is calculated and assigned to processfileCount, and then step S10 is executed.
2. The data processing method of claim 1, wherein: A shared state storage area is provided between the production end and the processing end, and the production end and the processing end can access the shared state storage area simultaneously; The data marker bits are stored in the shared state storage area.
3. The data processing method of claim 1, wherein: The methods for calculating the theoretical total consumption and the actual total consumption include the following: Scenario 1: The processing end includes only one consumption branch, and the generating end sends a complete batch of data to the branch for consumption each time; a data flag is set for the consumption branch of the processing end; Theoretical total consumption sendfileCount = total number of batches sent from the production end; Actual total consumption (processfileCount) = Total number of batches processed at the processing end; Scenario 2: The processing end has multiple consumption branches. The production end sends a complete batch of data to the processing end for consumption each time, and a data marker bit is established for each consumption branch. Theoretical total consumption sendfileCount = Total number of batches sent from the production side to each branch; Actual total consumption processfileCount = Total number of batches processed in each branch of the processing end; Scenario 3: The processing end has multiple consumption branches. After the production end reads a batch of data, it sends a single data item from the batch to the processing end for consumption. Each branch consumes a single data item at a time. A data marker is created for each consumption branch. Theoretical total consumption sendfileCount = m1+…+mj j +…+m n ; n is the total amount of production end sending batches, m j is the total amount of data included in the jth batch The actual total consumption at the processing end, processfileCount, equals the sum of the number of data items processed in each branch.
4. The data processing method of claim 1, wherein: Data updates in the shared state storage area are performed atomically.
5. The data processing method of claim 2, wherein: The shared state storage area also stores: sending state and processing state; The sending status is used to mark whether the data has been sent. The sending status includes: not sent and sent, with the default status being not sent. The production end generates the sending status of the data synchronously when generating each batch of data. After the data is successfully sent to the processing end, the sending status of the data is modified to sent. Each time the processing terminal receives data, it generates a processing status for the received data. The processing status includes: unprocessed, in progress, and processed, with the default status being unprocessed. After the processing terminal processes the data according to a preset logic, it sets the data to in progress, and after the processing is completed, it modifies the processing status of the data to processed.
6. The data processing method of claim 1, wherein: When the production end sends data, a unique identifier is constructed for this data transmission; the transmission status and the processing status in the shared status storage area are distinguished by the unique identifier. After receiving data, the processing end first checks whether the data has already been consumed; data that has already been consumed will not be received again.
7. The data processing method of claim 1, wherein: In step S6, the methods for determining whether all input data for this processing task has been completed include: The execution is triggered when the end of the file is reached, the database query range has been exhausted and there is no next page of data, the message stream receives an end message, or all input sources within the current execution range have ended.
8. A data processing system, characterized by It includes: the production end Processing end and shared state storage area; The production end connects to the data source to collect input data; The production end and the processing end are connected by a communication link; The production end and the processing end are respectively connected to the shared state storage area; The production end includes: a batch formation module, a temporary storage module, a sending module, and an end confirmation module; The batch generation module is used to communicate with the data source, continuously read input data, and generate a theoretical total consumption. The temporary storage module is used to temporarily store the latest batch of data. The sending module is used to send the previously stored batch when a new batch is formed, and to send the latest batch that is currently stored after the theoretical consumption total is written. The end confirmation module is used to confirm that the input data has ended, and after the input data ends, to calculate the actual theoretical consumption total and write it into the shared state storage area; The processing terminal includes: a receiving module, a logic processing module, and a completion judgment module; The receiving module is used to receive batch data from the production end, and at the same time to count the total actual consumption that the processing end has processed. The logic processing module is used to process each batch of data according to a preset logic; The completion judgment module is used to determine whether the theoretical total consumption has been determined, and whether the number of batches sent and the actual consumption that has been processed have reached the theoretical total consumption; and to confirm whether the task has been completed based on the judgment results.
