A time sequence alignment-based multi-source service data processing method, electronic equipment and program product

Abstract

The application provides a multi-source business data processing method based on time sequence alignment, an electronic device and a program product. According to the correspondence relationship between business data and a split file, structured storage is realized, an offset record writing progress is synchronously updated, valid data is defined according to writing success, the current maximum event time is determined, and then the split file to be published is screened. Compared with a heavy architecture which depends on distributed state management, complex water level line cooperation and distributed transaction coordination, the application does not need to introduce additional distributed components, but only needs to drive the core logic of the publication through simple split mapping, offset record and maximum event time. While realizing ordered storage, non-repetition, non-missing and reliable publication, the application saves the complex mechanisms such as distributed state synchronization and multi-source water level line negotiation, reduces the architecture complexity and resource consumption, balances the data processing integrity and real-time performance, achieves lightweight design, and avoids the problem of high implementation cost caused by component redundancy and complex logic.

Description

Technical Field

[0001] This application relates to the field of business data processing, and more specifically, to a method, electronic device, and program product for processing multi-source business data based on time-series alignment. Background Technology

[0002] In the field of business data monitoring, multi-source business data (such as data from business counter A and business counter B) needs to be processed to achieve accurate monitoring. Multi-source data should ideally arrive in order according to the time of business events, but due to upstream network latency and differences in processing speed, it often arrives out of order (e.g., business data at 10:24 and 10:25 arrives at 10:26), leading to errors in the calculation of event time windows by the stream processing system and affecting the accuracy of monitoring results.

[0003] Existing technologies like Flink use checkpointing to atomically persist the water level state and message queue (MQ) offset to the state backend before executing the MQ acknowledgment (ACK) response, thus achieving multi-source data water level alignment and state consistency. However, this approach relies on state storage components, MQ and its ACK mechanism, and Flink's checkpointing mechanism, resulting in high architectural complexity and significantly increased operational costs for both Flink and MQ.

[0004] Therefore, there is an urgent need for a lightweight solution that can reduce the complexity of architecture and operation and maintenance while achieving multi-source data level alignment and state consistency control. Summary of the Invention

[0005] The purpose of this application is to provide a method, electronic device and program product for processing multi-source business data based on time-series alignment, so as to reduce the complexity of multi-source data maintenance.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, embodiments of this application provide a method for processing multi-source service data based on time-series alignment, including:

[0008] Based on the correspondence between business data and shard files, the business data is written into the corresponding target shard file;

[0009] Update the offset corresponding to the amount of data written for the business data;

[0010] If the business data is successfully written, the business data is determined to be valid business data;

[0011] Based on all the valid business data, the current maximum event time is determined; the current maximum event time is the latest business event time recorded among all the valid business data.

[0012] Based on the current maximum event time, determine all the shard files to be published from the shard files and publish them.

[0013] Secondly, embodiments of this application provide an electronic device, including:

[0014] Memory, used to store one or more programs;

[0015] processor;

[0016] When the one or more programs are executed by the processor, the method as described in any one of the first aspects above is implemented.

[0017] Thirdly, embodiments of this application provide a program product that, when executed by a processor, implements the method as described in any one of the first aspects above.

[0018] Compared to existing technologies, the multi-source business data processing method, electronic device, and program product based on time-series alignment provided in this application achieves structured storage according to the correspondence between business data and sharded files, synchronously updates the offset record writing progress, defines valid data upon successful writing, determines the current maximum event time, and then filters sharded files to be published. Compared to the heavy architecture of existing technologies that rely on distributed state management, complex waterline coordination, and distributed transaction coordination, this method does not require the introduction of additional distributed components. It drives the core logic of publishing through simple shard mapping, offset records, and maximum event time. While achieving ordered data storage, no duplication or omission, and reliable publishing, it eliminates complex mechanisms such as distributed state synchronization and multi-source waterline negotiation, significantly reducing architectural complexity and resource consumption. It balances data processing integrity and real-time performance, achieving a lightweight design and avoiding the problems of high operation and maintenance costs and high deployment thresholds caused by component redundancy and complex logic in existing solutions.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A schematic diagram of a monitoring architecture for multi-source business data;

[0022] Figure 2 A flowchart illustrating a multi-source business data processing method based on time-series alignment provided in an embodiment of the present invention;

[0023] Figure 3 A flowchart illustrating another method for processing multi-source business data based on time-series alignment provided in an embodiment of the present invention;

[0024] Figure 4 This is a signaling interaction diagram of a multi-source service data processing method based on time alignment provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram illustrating business data writing based on fragmented files, provided as an embodiment of the present invention.

[0026] Figure 6 A flowchart illustrating another method for processing multi-source business data based on time-series alignment provided in an embodiment of the present invention;

[0027] Figure 7 A signaling interaction diagram illustrating another multi-source service data processing method based on time alignment provided in an embodiment of the present invention;

[0028] Figure 8 A signaling interaction diagram illustrating another multi-source service data processing method based on time alignment provided in an embodiment of the present invention;

[0029] Figure 9 A signaling interaction diagram illustrating another multi-source service data processing method based on time alignment provided in an embodiment of the present invention;

[0030] Figure 10 A flowchart illustrating another method for processing multi-source business data based on time-series alignment provided in an embodiment of the present invention;

[0031] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0034] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0035] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0036] In existing technologies, monitoring mechanisms for business data are prone to errors. When upstream multi-source business data arrives out of order due to network latency and processing speed differences, it affects the accuracy of the stream processing system's calculation of event time windows, often resulting in erroneous monitoring results. Specifically, Figure 1 See the diagram for the monitoring architecture of multi-source business data. Figure 1 A certain business monitoring system has two different business sources upstream: Counter A and Counter B.

[0037] Due to upstream network latency or differences in upstream processing speed, see [link / reference]. Figure 1 This can cause a delay in the arrival of some data, such as Figure 1 In the data processing, the business data from counter A arrived late on October 24th and 25th. This out-of-order arrival of business data will lead to incorrect monitoring results.

[0038] Based on the above, existing technologies typically use Flink as the mainstream technology in the stream processing field. To address the problem of out-of-order data from multiple sources, it achieves water level alignment and state consistency through a collaborative architecture of "checkpoint mechanism, MQ ACK, and state storage." Its core principle can be broken down into four main stages: "data access - state maintenance - atomic persistence - acknowledgment feedback," as detailed below:

[0039] Multi-source data access layer: Flink connects to multiple upstream message queues (MQ) (such as Kafka and RabbitMQ) through the Source operator. Each MQ corresponds to a business data source (such as business counter A and business counter B). After the upstream multi-source business data is generated according to the "event time" (i.e. the actual time of the business), it is first sent to the corresponding MQ. Flink pulls data from the partitions of each MQ to form a multi-stream parallel input data stream.

[0040] Water level calculation layer: Flink's window operators (such as time windows) calculate the "water level" for each input data stream in real time. Essentially, it represents the "maximum event time observed in the current data stream," used to mark that "all data in the stream before this time has arrived." For example, if the water level for data source A is 10:05, it means that all business data in data source A before 10:05 has been collected; if the water level for data source B is 10:03, it means that all business data in data source B before 10:03 has been collected.

[0041] Global water level alignment logic: To ensure that multi-source data is processed synchronously according to event time (i.e., water level alignment), the Flink window operator takes the minimum value of the water level of all input data streams as the global water level. Taking the above example, the global water level is 10:03, which means that the data from data sources A and B before 10:03 has arrived, and the data calculation within this time window can be triggered, avoiding calculation omissions caused by the late arrival of data from a certain data source.

[0042] State storage component dependency: Flink stores two types of core state in independent "state backends" (such as HDFS and RocksDB): one is "water level state" (water level values ​​for each data stream and globally), and the other is "MQ offset state" (the data offsets read from each MQ partition, used to mark "where in the MQ has been processed"). These two types of state are crucial to ensuring "consistency between computation progress and event time". If the state is lost, Flink cannot determine "which data has been processed and what the current water level is" after restarting, which will lead to duplicate calculations or data omissions.

[0043] Atomicity Guarantee of Checkpoint Mechanism: To avoid inconsistencies between water level status and MQ offset (e.g., water level has been updated but offset not stored, or vice versa), Flink uses a checkpoint mechanism to achieve atomic persistence of two types of states: When a checkpoint is triggered, Flink pauses all data processing and packages the water level of all current data streams and the offsets of each MQ into a "checkpoint snapshot"; the snapshot is written to the state backend via a two-phase commit protocol: the first phase confirms that the state of each operator is ready, and the second phase uniformly persists the snapshot to the state backend; only when the "checkpoint snapshot is successfully written to the state backend" will Flink execute subsequent operations, ensuring that "the water level status and MQ offset are either successfully stored simultaneously or fail and rollback simultaneously," with no intermediate inconsistent states.

[0044] Final MQACK confirmation: After the state atomic persistence is complete, Flink sends an "ACK (acknowledgment signal)" to each upstream message queue (MQ). Upon receiving the ACK, the MQ deletes the data packets that have been read and stored in state by Flink. This step is to prevent the MQ from resending processed data after Flink restarts: if no ACK is sent, the MQ will retain the data packets, and Flink will fetch and process them again after restarting, leading to duplicate calculations; if the ACK is sent prematurely (before the state storage is complete), Flink cannot recover the deleted data packets if the state storage fails, resulting in data loss.

[0045] Therefore, while existing technologies can achieve multi-source data level alignment and state consistency, they inherently suffer from "high architectural complexity and high operational costs." The root cause lies in "over-reliance on multi-component collaboration and complex protocols to ensure consistency," which can be analyzed from three dimensions: "component dependency, protocol complexity, and operational coupling."

[0046] 1. Strong dependencies among multiple components: The stability of the architecture is limited by the compatibility of components.

[0047] Flink's solution relies on three types of external components: "MQ (data ingestion), state backend (state storage), and distributed coordination components (such as ZooKeeper, used for checkpoint coordination)." These components are tightly coupled: if the MQ experiences a partition failure (such as a Kafka partition leader switch), the FlinkSource operator cannot pull data, causing the waterline to fail to advance and global computation to stagnate; if the state backend storage is delayed (such as a decrease in HDFS write performance), checkpoint snapshot writes will time out, triggering checkpoint failure, and Flink will need to re-trigger the checkpoint, resulting in data processing delays; if the coordination component fails (such as a ZooKeeper cluster crash), the two-phase commit of the checkpoint cannot be completed, the state atomic storage mechanism fails, and state consistency is directly compromised.

[0048] This "multi-component strong dependency" architecture makes system stability extremely sensitive, as any component failure will propagate to Flink, causing computation interruptions or data errors. Furthermore, compatibility debugging between components (such as matching Flink and Kafka versions, and adapting the state backend and checkpoint mechanism) is itself a complex engineering task.

[0049] II. Complex protocols and mechanisms: High difficulty in troubleshooting and maintenance

[0050] Flink's mechanisms for ensuring consistency, such as "checkpoint two-phase commit" and "MQACK confirmation," suffer from complex protocol logic and cumbersome handling of exceptional scenarios.

[0051] Checkpoint failures can occur in various ways: such as failure to write state backend, inconsistent operator states, or network timeouts. It is necessary to check the checkpoint logs (such as CheckpointID and snapshot status of each operator) one by one to locate the fault.

[0052] The interaction between MQACK and state storage: If the state backend suddenly fails after Flink sends an ACK, the MQ may delete the data packet but the state may not be stored, which will result in permanent data loss. Additional parameters such as "ACK retry mechanism" and "checkpoint timeout retry policy" need to be configured. The parameter tuning is difficult (too many retries will cause MQ to accumulate, while too few retries will increase the risk of data loss).

[0053] Troubleshooting water level anomalies is difficult: If the global water level is progressing slowly, it may be due to reasons such as late arrival of data from a certain data source, errors in the water level generation logic, or delays in window operator calculation. It requires cross-analysis of multi-stream water level logs and MQ consumption progress logs, which results in low troubleshooting efficiency.

[0054] This complexity requires operations and maintenance personnel to simultaneously master Flink's internal mechanisms (checkpoints, watermarks), MQ principles (partitioning, ACK), and state backend storage logic, resulting in an extremely high maintenance threshold and a long troubleshooting cycle, which affects the real-time performance of business monitoring.

[0055] 3. High resource and cost consumption: poor adaptability to lightweight scenarios

[0056] Flink's distributed architecture and complex mechanisms place high demands on hardware resources.

[0057] The state backend requires a lot of storage resources (e.g., HDFS requires multiple nodes, and RocksDB requires local disk space), and checkpoint snapshot writing consumes network bandwidth (in distributed scenarios, snapshots need to be transmitted across nodes).

[0058] Data processing will be paused when a checkpoint is triggered. Although "incremental checkpoints" can be configured to reduce overhead, it will still cause some processing delay. It is not suitable for lightweight scenarios that are sensitive to latency (such as monitoring small and medium-sized business counters, which do not require a distributed architecture but still have to bear the resource costs of Flink).

[0059] The deployment and maintenance of MQ also require independent resources (such as Kafka requiring multiple broker nodes to ensure high availability). For scenarios with "small amounts of multi-source data and a pursuit of low-cost operation and maintenance" (such as counter monitoring of small financial institutions), the "resource input-output ratio" of the Flink solution is too low.

[0060] To address the aforementioned issues, this application introduces a core approach: "sharded files, offsets corresponding to the amount of data written, the current maximum event time, and published files." Sharded files serve as a structured carrier for storing multi-source data in time-partitioned format, preventing data corruption. Offsets are used to precisely control the data acquisition progress, ensuring data is not duplicated, omitted, or inconsistent with the write status. The current maximum event time is used as a benchmark to determine the publishable range of files, balancing data integrity and real-time processing. Published files output ordered, valid data with zero invalid data to subsequent calculations, ensuring computational accuracy. Ultimately, multi-source data time-series alignment is achieved within a lightweight architecture, ensuring accurate monitoring results while significantly reducing architectural complexity, operational costs, and resource consumption, thus resolving existing technical pain points.

[0061] Optionally, Figure 2 A flowchart illustrating a multi-source service data processing method based on time-series alignment provided in an embodiment of the present invention is shown below. Figure 2 The method includes:

[0062] Step 102: Based on the correspondence between business data and shard files, write the business data into the corresponding target shard file.

[0063] Step 103: Update the offset corresponding to the amount of business data written.

[0064] Step 104: If the business data is successfully written, confirm that the business data is valid.

[0065] Step 105: Determine the current maximum event time based on all valid business data.

[0066] The current maximum event time is the latest business event time recorded in all valid business data.

[0067] Step 106: Based on the current maximum event time, determine the fragment files to be published in all fragment files and publish them.

[0068] The multi-source business data processing method based on time-series alignment provided in this invention implements structured storage according to the correspondence between business data and sharded files, synchronously updates the offset record writing progress, defines valid data based on successful writing, determines the current maximum event time, and then filters sharded files to be published. Compared to the heavy architecture of existing Flink solutions that rely on distributed state management, complex watermark coordination, and distributed transaction coordination, this method does not require the introduction of additional distributed components. It drives the core logic of publishing through simple shard mapping, offset records, and maximum event time. While achieving ordered data storage, no duplication or omission, and reliable publishing, it eliminates complex mechanisms such as distributed state synchronization and multi-source watermark negotiation, significantly reducing architectural complexity and resource consumption. It balances data processing integrity and real-time performance, achieving a lightweight design and avoiding the problems of high operation and maintenance costs and high deployment thresholds caused by component redundancy and complex logic in existing solutions.

[0069] Due to the processing scenario of this application, it will encounter situations involving multi-source out-of-order data. This application provides "time anchoring" through fragmented files, writing matching business data into the response fragmented files. Below is one possible implementation method for constructing fragmented files; specifically... Figure 2 On this basis, Figure 3 A flowchart illustrating another multi-source service data processing method based on time-series alignment provided in this embodiment of the invention is shown below. Figure 3 Before step 102, the following is also included:

[0070] Step 100: Obtain the shard file configuration parameters, which include the business operation period and the sharding time interval.

[0071] Step 101: Generate multiple fragment files with serial numbers based on the business operation period and the fragmentation time interval.

[0072] Each sequence number is matched with a corresponding time interval, which is determined by the business operation period and the slicing time interval.

[0073] Optionally, to better illustrate the configuration process of the fragment file configuration parameters and the generation process of the fragment files, the following explanation is provided with examples. Specifically, Figure 4 This is a signaling interaction diagram of a multi-source service data processing method based on time-series alignment provided in an embodiment of the present invention. (See attached diagram.) Figure 4 The architecture consists of two counters, namely Counter A and Counter B, multi-source data alignment components, databases, and file systems.

[0074] Among them, counter A and counter B generate business data with event times in real time.

[0075] The multi-source data alignment component is used to trigger the creation of shard files and write business data into the corresponding shard files in an orderly manner, and then publish the shard files.

[0076] The database is used to maintain the offset of each business data item and is enabled to reflect the progress of business data collection and writing.

[0077] Specifically, this offset can control incremental data collection and avoid redundant processing: before each collection, the offset in the database is read, and new data is pulled from the "next record of written data" (e.g., if the offset is 10000, collection starts from record 10001), preventing "processed valid data from being repeatedly written to the fragment file" and reducing invalid calculation and storage overhead.

[0078] This offset also ensures breakpoint recovery and avoids data loss: if the system is interrupted (such as by restart or failure), after restarting, only the latest offset in the database needs to be read to continue collecting and writing from the breakpoint without reprocessing historical data, ensuring that "collected data is not lost" and guaranteeing business continuity.

[0079] This offset can also be associated with the write status to ensure consistency: the database transaction binds "successful write of shard file" with "successful offset update". The offset is only updated synchronously after the data is effectively written to the file; if the write fails, the offset remains unchanged, ensuring that the offset always accurately reflects the "amount of valid data written" and avoiding inconsistencies such as "data not written but progress updated" or "data written but progress not recorded".

[0080] The file system is used for structured storage of multi-source business data, specifically storing time-aligned business data based on sharded files.

[0081] Furthermore, see this Figure 4 Step 200: Configure the multi-source data alignment component to obtain the shard file configuration parameters, which include the business operation period and the sharding time interval.

[0082] Step 201: The multi-source data alignment component triggers initialization to the file system.

[0083] Step 202: The file system generates multiple fragment files with sequence numbers based on the business operation time period and the fragmentation time interval.

[0084] Specifically, for a certain business system, its business operation period is [9:30-15:00]. This business system can be initialized daily in the above way. For example, if the sharding time interval is 10 seconds, then [9:30-15:00] will be sharded into 1620 empty shard files (such as tmp files) with incrementing sequence numbers.

[0085] Optionally, Figure 5 This is a schematic diagram illustrating business data writing based on fragmented files, provided as an embodiment of the present invention. See also... Figure 5 Incremental data collected from different counters is first obtained in the form of an out-of-order file queue.

[0086] Furthermore, for business data collected from the same counter and in the same batch, in this... Figure 4 The data is identified as follows: Business data 1, collected from counter A; Business data 2, collected from counter B; Business data 3, collected from counter A; Business data n, collected from counter B; Business data n+1, collected from counter B.

[0087] Within the same batch of business data, there can be different event times. For example, business data 1 contains business data with event times of (5:1:0), (8:1:0), (9:0:2), and (9:0:7).

[0088] Furthermore, based on the above example, multiple shard files with sequence numbers can be generated according to the business operation period and the sharding time interval. For example... Figure 5 As shown, if the sharding interval is 5 seconds, the shard file with sequence number 1 has a time interval of (~9:0:0], meaning that any business data with an event time at or before (9:0:0) should be stored in shard file 1. Similarly, the shard file with sequence number 2 has a time interval of (9:0:0~9:0:5], meaning that any business data with an event time between (9:0:0~9:0:5) should be stored in shard file 2, and so on up to shard file N.

[0089] Based on the event times contained in each batch of business data, it is stored in a shard file corresponding to the time series interval. For example, business data with event times (5:1:0) and (8:1:0) in business data 1 are both written to shard file 1. Business data with event time (9:0:2) in business data 1, and business data with event times (9:0:1) and (9:0:5) in business data 2 are written to shard file 2.

[0090] Furthermore, the release of fragment files can be controlled by inserting watermarks. For example, fragment files 1 through 6 can be released sequentially.

[0091] Optionally, for business data to be written, if the shard file corresponding to its event time has been published, the business data can be written to the unpublished shard file with the closest time interval. For example, business data with event time (9:0:24) in business data n+1 should be written to shard file 6, but since shard file 6 has been published, the business data can be written to shard file 7.

[0092] Optionally, to manage business data validity verification and anomaly tracing, this application introduces transaction tracking identifiers to implement relevant controls. Specifically, the following provides a possible implementation method for writing business data to the target shard file. Figure 2 On this basis, Figure 6 A flowchart illustrating another multi-source service data processing method based on time-series alignment provided in this embodiment of the invention is shown below. Figure 6 Step 102 includes:

[0093] Step 102-1: Obtain the historical offset from the last business data collection.

[0094] Step 102-2: Based on the historical offset, collect the newly added business data and identify the event time of the newly added business data.

[0095] Step 102-3: Based on the correspondence between event time and time interval, determine the target fragment file corresponding to the target sequence number.

[0096] Step 102-4: Generate transaction tracking identifiers for the newly added business data.

[0097] Step 102-5: Write the newly added business data and transaction tracking identifier into the target shard file.

[0098] Optionally, the function of this transaction tracking identifier will be explained below with examples. Specifically, Figure 7 A signaling interaction diagram for another multi-source service data processing method based on time alignment provided in an embodiment of the present invention is shown below. Figure 7 When Figure 5 After the system architecture shown is initialized, business data can begin to be written. The process is as follows:

[0099] Step 203: Obtain the offset of the corresponding business system from the database.

[0100] Specifically, for example, we can obtain the offset of the business system corresponding to counter A. In this example, the historical offset at the current moment is 10000.

[0101] Step 204: The multi-source data alignment component pulls business data A based on the new offset.

[0102] Specifically, this new offset is the additional offset added based on the historical offset. For example... Figure 7 The event time shown is 10:00, and the newly added offset is 5000.

[0103] Step 205: The multi-source data alignment component identifies the event time of the business data.

[0104] Specifically, the event time for business data A is identified as 10:00. Based on the previous example, it can be determined that the segment file corresponding to this event time can be identified.

[0105] Step 206: The multi-source data alignment component transfers business data A to the file system.

[0106] Step 207: The file system generates a corresponding transaction trace identifier (traceID) based on the business data A.

[0107] Specifically, the file system initiates a database transaction and generates a unique transaction tracking identifier for this business data write process. This identifier serves as a globally unique identifier for "a single business data write and its corresponding database transaction," ensuring a one-to-one mapping between transactions and business data.

[0108] Optionally, the possible implementation methods for selecting shard files and writing business data are as follows: Traverse multiple groups of business data. For one group, determine whether the shard file corresponding to the group's sequence number has been published. If published, write the group data to the nearest unpublished file; otherwise, write the group data to the corresponding shard file. During writing, the following steps can be followed: 1) Enable process lock: Add a JVM process lock to the target shard file path; 2) Append a start identifier to the transaction trace identifier: Add [startwrite:traceID] before the first line; 3) Append group data: Append multiple lines of the group data to the end; 4) Release process lock: Release the shard file-level process lock.

[0109] Step 208: The file system writes business data A into the corresponding target shard file.

[0110] Specifically, based on the amount of data in business data A, it can be written to one or more target shard files. For example, based on the event time 10:00, business data A should be written to shard file 4 with the corresponding sequence number. However, shard file 4 can only store a portion of the data, such as 2000 data points. The remaining business data is then stored in the next shard file 5, such as... Figure 7 The data stored is 3000. Therefore, the total offset is 5000.

[0111] Step 209: The file system commits the transaction to the database.

[0112] Specifically, the transaction includes the updated offset, as in example 5000 above.

[0113] Step 210: Update the offset in the database.

[0114] Specifically, if the historical offset is 10000 and the updated offset is 5000, then the new offset after the update is 10000 + 5000, which is 15000.

[0115] Step 211: After the offset update is completed, the database synchronization file system transaction tracking identifier is successfully updated.

[0116] Specifically, if the transaction tracking identifier is successfully updated, indicating that the business data has been successfully written, then step 212 is executed.

[0117] Step 212: Add a committed flag to the transaction tracking flags.

[0118] Specifically, after business data is successfully written, a committed flag is appended to the transaction tracking flag of the corresponding shard file. This, combined with database transaction control, ensures the consistency between the offset position and the actual written shard file. If a transaction succeeds but the file is not appended successfully due to OutOfMemory (OOM) or other reasons, during subsequent file reads, for transaction tracking flags without the committed flag, the database will be searched for the last successfully committed transaction tracking flag. If found, this portion of the data will be recovered.

[0119] Step 213: Read the delayed release time configuration.

[0120] The delayed release time configuration includes a delayed release time threshold. The file sequence number for delayed release is calculated as follows: Publishable Time = Current Maximum Event Time (Maximum time to retrieve business data) - Delayed Release Time Threshold. For example, if the current maximum event time is 10:30 and the delayed release time threshold is 10 minutes, then the publishable time is 10:20, meaning that the shard file corresponding to 10:20 has its business data aligned by default. Combining the mapping relationship between the sequence numbers of each shard file and the time intervals mentioned earlier, the publishable time is converted into the maximum publishable shard file sequence number.

[0121] It should be noted that if an upstream source (such as Counter B) fails to write business data to the corresponding shard file due to a lack of data, the mapping relationship is "time-based shard numbering" rather than "shard numbering based on whether a source has data." As long as the publishable time meets the requirements, the maximum publishable shard number will still be calculated according to the rules. Other upstream sources with data (such as Counter A) with the same shard number can still be published normally and participate in the calculation. The entire process will not be halted due to missing data from a single source, ensuring the continuous progress of the calculation process.

[0122] Step 214: Instruct the release of the fragment files to be published.

[0123] Specifically, the multi-source data alignment component instructs the file system to publish files. Based on the largest publishable shard file sequence number, the file system scans unpublished shard files to determine the shard files to be published and ultimately instructs the publication of the corresponding shard files.

[0124] Step 215: Publish the corresponding files.

[0125] For example, see Figure 7 If fragment file 4 and fragment file 5 are fragment files to be published, then based on this instruction, fragment file 4 and fragment file 5 will be published. The publication watermark will then be updated.

[0126] Optionally, for publishing split files, taking tmp files as an example, the publishing can be completed by renaming them from smallest to largest as release files. After publishing, it can drive subsequent calculations.

[0127] Optionally, combined Figure 7 In some scenarios, database offset updates may fail. To ensure data consistency, a possible implementation of a data rollback mechanism is provided. Specifically, Figure 8 A signaling interaction diagram for another multi-source service data processing method based on time alignment provided in an embodiment of the present invention is shown below. Figure 8 After the business data is written to the sharded file, the process includes the following steps:

[0128] Step 209: The file system commits the transaction to the database.

[0129] Step 210a: Offset update failed.

[0130] Step 211a: The database sends a rollback instruction to the file system.

[0131] Step 212a: The file system appends a rollback flag to the corresponding transaction tracking flag.

[0132] Specifically, if the offset update fails, it means that the corresponding data write failed, and a rollback flag is added to the transaction tracking flag.

[0133] Step 214: Instruct the release of the fragment files to be published.

[0134] Step 215: Publish the corresponding files.

[0135] At this point, combining the above example, both shard file 4 and shard file 5 have been published. However, if the transaction trace identifiers of the data contain a start identifier (Startwrite) and a rollback identifier (Rollback), the data can be defined as invalid data.

[0136] Step 216: Obtain a data read request.

[0137] For details, see Figure 8 If counter A needs to read relevant business data, it sends a data read request to the file system.

[0138] Step 217: Locate the fragment file to be read among the published fragment files.

[0139] Specifically, if the sharded files to be read are sharded file 4 and sharded file 5, then the validity of the data in the sharded files is identified. If the transaction trace identifier contains both a start identifier (Startwrite) and a committed identifier (committed), then the corresponding business data is valid. If the transaction trace identifier only contains a start identifier (Startwrite), or if the transaction trace identifier (TraceID1) contains both a start identifier (Startwrite) and a rollback identifier (Rollback), then the business data is invalid.

[0140] Step 218: Read valid business data and ignore invalid business data.

[0141] Specifically, this can be understood as reading business data with a committed flag and ignoring business data with a rollback flag or no committed flag.

[0142] Optionally, in some scenarios, application layer interruptions may cause the operation of writing business data to sharded data to fail. In this case, the transaction tracing identifier mentioned above may only contain the start identifier (Startwrite). The following is an example illustrating this situation. Specifically, Figure 9 A signaling interaction diagram for another multi-source service data processing method based on time alignment provided in an embodiment of the present invention is shown below. Figure 9 The relevant process includes the following steps:

[0143] Step 207: The file system generates a corresponding transaction trace identifier (traceID) based on the business data A.

[0144] Step 208: The file system writes business data A into the corresponding target shard file.

[0145] Specifically, if the write operation fails at this point, the offset will not be updated.

[0146] Step 214: Instruct the release of the fragment files to be published.

[0147] Step 215: Publish the corresponding files.

[0148] At this point, combining the above example, both shard file 4 and shard file 5 have been published. However, if the transaction trace identifiers of the data contain a start identifier (Startwrite) and a rollback identifier (Rollback), the data can be defined as invalid data.

[0149] Step 216: Obtain a data read request.

[0150] For details, see Figure 9 If counter B needs to read relevant business data, it sends a data read request to the file system.

[0151] Step 217: Locate the fragment file to be read among the published fragment files.

[0152] Specifically, if the sharded files to be read are sharded file 4 and sharded file 5, then the validity of the data in the sharded files is identified. Specifically, if the transaction tracking identifier only contains a start identifier (Startwrite) and does not contain a committed identifier (committed), then the business data is invalid.

[0153] Step 218: Read valid business data and ignore invalid business data.

[0154] Specifically, this can be understood as reading business data with a committed flag and ignoring business data with a rollback flag or no committed flag.

[0155] Optionally, as seen in the example above, multi-source business data may experience delays due to network fluctuations, differences in processing speed, etc. Therefore, by setting up a delayed release mechanism for fragmented files, data omissions caused by premature file release can be avoided, thus ensuring data integrity. Below is one possible implementation method for delayed release; specifically... Figure 3 On this basis, Figure 10 A flowchart illustrating another multi-source service data processing method based on time-series alignment provided in this embodiment of the invention is shown below. Figure 10Step 106 includes:

[0156] Step 106-1: Read the delayed release time configuration.

[0157] This delayed release time configuration includes a delayed release time threshold.

[0158] Step 106-2: Determine the publishable time based on the current maximum event time and the delayed publication time threshold.

[0159] Step 106-3: Determine the maximum publishable sequence number based on the publishable time.

[0160] Step 106-4: Based on the largest publishable sequence number, traverse all unpublished shard files to determine the shard files to be published.

[0161] Step 106-5: Publish the fragmented files to be published.

[0162] This invention also provides an electronic device that can execute all the steps of the examples described above to achieve the corresponding technical effects. Specifically, Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. See also: Figure 11 The electronic device 30 includes: a memory 301 and a processor 300;

[0163] Memory 301 is used to store one or more programs;

[0164] Processor 300;

[0165] When one or more programs are executed by a processor, the electronic device 30 can achieve the steps and corresponding technical effects when it performs the steps shown in the above-described method examples.

[0166] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0167] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0168] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a program product. This program product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0170] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for processing multi-source business data based on time alignment, characterized in that, include: Obtain the shard file configuration parameters, which include the business operation time period and the sharding time interval; Based on the business operation period and the sharding time interval, multiple shard files with serial numbers are generated; wherein each serial number matches a corresponding time interval, and the time interval is determined by the business operation period and the sharding time interval. Obtain the historical offset from the last business data collection; the business data is multi-source business data. Based on the historical offset, newly added business data is collected, and the event time of the newly added business data is identified; Based on the correspondence between the event time and the time interval, determine the target segment file corresponding to the target sequence number; Generate a transaction tracking identifier for the newly added business data; The transaction tracking identifier is generated when a database transaction is started, and the transaction tracking identifier is used to uniquely identify the one-to-one mapping relationship between a single business data write and the database transaction; Write the newly added business data and the transaction tracking identifier into the target shard file; Update the offset corresponding to the amount of data written for the business data; wherein, the offset is successfully bound to the successful writing of the shard file and the successful update of the offset through the database transaction; when data is successfully written to the file, the offset is updated synchronously; when data writing to the file fails, the offset remains unchanged; If the business data is successfully written, the business data is determined to be valid business data; If the writing of the business data to the corresponding target shard file fails, the offset corresponding to the amount of business data written remains unchanged. Based on all the valid business data, the current maximum event time is determined; the current maximum event time is the latest business event time recorded among all the valid business data. Based on the current maximum event time, determine all the shard files to be published from the shard files and publish them.

2. The method of claim 1, wherein, The step of updating the offset corresponding to the amount of data written for the business data includes: Write the offset into the status information table of the relational database; If the write is successful, a committed flag is added to the transaction tracking flags; If the write fails, a rollback flag is added to the transaction tracking flag.

3. The method of claim 2, wherein, If the business data is successfully written, the step of determining that the business data is valid business data includes: If the transaction tracking identifier of the business data is identified and has the committed identifier, then the business data is confirmed to be valid business data.

4. The method of claim 1, wherein, The step of determining the shard files to be published among all the shard files based on the current maximum event time includes: Read the delayed release time configuration; the delayed release time configuration includes a delayed release time threshold. The publishable time is determined based on the current maximum event time and the delayed publication time threshold. The maximum publishable sequence number is determined based on the publishable time. Based on the maximum publishable sequence number, traverse all unpublished shard files to determine the shard file to be published; Publish the fragmented files to be published.

5. The method of claim 2, wherein, Also includes: When a data read request is received, the file to be read is located in the published file shards. Read the business data with the submitted identifier; Ignore business data that has the rollback flag or no submitted flag.

6. An electronic device, comprising: include: Memory, used to store one or more programs; processor; When the one or more programs are executed by the processor, the method as described in any one of claims 1-5 is implemented.

7. A program product, characterized by When the program product contains a program that is executed by a processor, it implements the method as described in any one of claims 1-5.

Images