A method and system applied to online data offline editing fusion

Abstract

The application relates to a method and system applied to online data offline editing fusion, applied to a data center, comprising the following steps: acquiring real-time incremental data and offline historical data, and mapping the data to a unified logical time coordinate system to determine the alignment boundary time to which the real-time incremental data belongs; performing logical judgment on the offline historical data; performing conflict detection and data fusion on the offline historical data and the real-time incremental data based on a business primary key identifier and an alignment timestamp to generate fusion data records; encapsulating metadata extension information of the fusion data records to generate a fusion data stream and output the fusion data stream to a downstream service layer; in summary, the application can solve the problems of offline and real-time data in time sequence consistency and data conflict through the steps of constructing a calculation topology, mapping data to a unified time coordinate system, performing logical judgment, and performing conflict detection and fusion, and has the effects of improving data consistency and reducing system complexity.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for online data offline editing and fusion. Background Technology

[0002] With the deepening of enterprise digital transformation, high-frequency data application scenarios such as the Industrial Internet and Fintech are experiencing explosive growth. Building a unified and efficient data platform architecture has become a core requirement for modern information technology infrastructure construction. Current mainstream technology paradigms generally adopt layered computing models such as Lambda or Kappa architectures, dividing data processing into two parallel paths: the offline computing link relies on tools such as Hive and MPP databases, using batch processing mechanisms to complete full-scale indicator calculations within a T+1 cycle, emphasizing eventual data consistency and high throughput; the real-time computing link is based on stream processing technologies such as Kafka and Flink, dynamically parsing and aggregating millisecond-level events in real time, emphasizing the immediacy of business feedback.

[0003] However, the existing dual-track architecture has revealed the following shortcomings during engineering implementation: First, the existing architecture suffers from a lack of temporal consistency. Offline computing, based on static data partitioning, leads to a lag in the time base. Although real-time stream processing uses event-time semantics, it is affected by data acquisition jitter, heterogeneous network transmission latency, and differences in the timestamp parsing accuracy of storage engines. This causes a misalignment of the time dimension of the same business entity in the offline table and the real-time stream. For example, in financial risk control scenarios, the misalignment of timestamps between real-time transaction streams and historical benchmark data makes it difficult to achieve accurate semantic matching for complex analyses, directly leading to the failure of risk identification. Second, the existing data conflict resolution mechanism has significant deficiencies. Specifically, the calibration data generated by the offline channel and the transient data from the real-time channel frequently overlap at the same granularity. Existing architectures typically rely solely on hard-coded manual intervention or simple overlay logic in upper-layer applications, lacking fine-grained discrimination rules based on data quality, version sequence, and generation time. Furthermore, due to the non-uniformity of the underlying data model, upper-layer applications are forced to repeatedly develop adaptation layers for both offline and real-time APIs. This not only significantly increases the complexity of system operation and maintenance but also prevents the application layer from obtaining a logically unified global data view. When the data platform undergoes architectural upgrades or model evolution, this tight coupling will trigger a chain reaction, significantly reducing the system's responsiveness. Summary of the Invention

[0004] To address the aforementioned shortcomings, this application provides a method and system for online data offline editing and fusion.

[0005] The above-mentioned objective of this application is achieved through the following technical solution: A method for online data offline editing and fusion, applied to a data platform, includes the following steps: Construct a computing topology based on a distributed stream processing framework to acquire real-time incremental data and offline historical data with partition identifiers and business update timestamps. The real-time incremental data includes event timestamps and business payloads. Real-time incremental data and offline historical data are mapped to a unified logical time coordinate system, and the alignment boundary time to which the real-time incremental data belongs is determined by combining the preset alignment time window parameters. Perform predefined logical judgments on offline historical data to verify whether the business update timestamp falls within the closed-open interval starting from the alignment boundary time. If so, it is the alignment timestamp that is consistent between the offline historical data and the real-time incremental data. Identify the business primary key identifiers corresponding to real-time incremental data and offline historical data, and based on the business primary key identifiers and aligned timestamps, perform conflict detection and data fusion on the offline historical data and real-time incremental data that have been associated with aligned timestamps, and generate fused data records; The corresponding metadata extension information is encapsulated for the fused data records, generating a fused data stream, and outputting it to a preset downstream service layer to provide data access services.

[0006] The second objective of this invention is achieved through the following technical solution: A system for online data offline editing and fusion includes: The data acquisition module is used to construct a computing topology based on a distributed stream processing framework, acquire real-time incremental data and offline historical data with partition identifiers and business update timestamps. The real-time incremental data includes event timestamps and business payloads. The data mapping module is used to map real-time incremental data and offline historical data to a unified logical time coordinate system, and determine the alignment boundary time to which the real-time incremental data belongs by combining the preset alignment time window parameters. The logic determination module is used to perform predefined logic determinations on offline historical data, verifying whether the business update timestamp falls within the closed-open interval starting from the alignment boundary time. If so, it is the alignment timestamp that is consistent with the offline historical data and the real-time incremental data. The data generation module is used to identify the business primary key identifiers corresponding to real-time incremental data and offline historical data, and based on the business primary key identifiers and alignment timestamps, to perform conflict detection and data fusion on the offline historical data and real-time incremental data that have been associated with the alignment timestamps, and generate fused data records. The data stream output module is used to encapsulate the corresponding metadata extension information of the fused data records, generate the fused data stream, and output it to the preset downstream service layer to provide data access services.

[0007] This application also relates to a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for online data offline editing and fusion.

[0008] This application also relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for online data offline editing and fusion.

[0009] In summary, the online data offline editing and fusion method and system provided in this application can solve the problems of temporal consistency and data conflict between offline and real-time data by constructing a computing topology, mapping data to a unified time coordinate system, executing logical judgment, and performing conflict detection and fusion. This method and system can improve data consistency and reduce system complexity. Attached Figure Description

[0010] Figure 1 This is a flowchart of an embodiment of a method for online data offline editing and fusion according to this application; Figure 2 This is a flowchart of step S20 in an embodiment of a method for online data offline editing and fusion according to this application; Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0011] The following is in conjunction with the appendix Figures 1-3 This application will be described in further detail.

[0012] In one embodiment, such as Figure 1 As shown, this application discloses a method for online data offline editing and fusion, which specifically includes the following steps: S10: Construct a computing topology based on a distributed stream processing framework, and obtain real-time incremental data and offline historical data with partition identifiers and business update timestamps. The real-time incremental data includes event timestamps and business payloads. In this embodiment, a distributed stream processing framework refers to a computing system capable of processing continuous data streams. It achieves high throughput and low latency data processing capabilities by decomposing data processing tasks and executing them in parallel across multiple computing nodes. This distributed stream processing framework typically supports event-time semantics, can handle out-of-order data, and provides state management and fault tolerance mechanisms. The computational topology refers to the abstract representation of the data processing logic within the distributed stream processing framework. It is typically constructed in the form of a directed acyclic graph, where nodes represent data processing operations and edges represent the direction of data flow. The computational topology usually defines the entire data processing flow from input to output. The processing path; Real-time incremental data refers to data that is generated and captured instantly when a business occurs. It usually exists in the form of an event stream and has strong timeliness. Real-time incremental data usually includes an event timestamp to indicate the actual time the event occurred, as well as the business payload, i.e., the specific content of the event; Offline historical data refers to all or part of the historical data stored in a persistent storage system after batch processing or periodic synchronization. Offline historical data usually has partition identifiers for data management and query optimization, and includes a business update timestamp to indicate the time when the offline historical data record was last updated in the offline system.

[0013] Specifically, the computing topology can be manually configured, for example, by writing configuration files or scripts to define the data source, transformation logic, and data sink; real-time incremental data can be consumed directly from the message queue, which contains event timestamps and business payloads; offline historical data can be periodically read from batch processing systems (such as HDFS or relational databases), which contains partition identifiers and business update timestamps. For example, a data acquisition agent can be set up to periodically extract updated data from the database and mark it as offline historical data.

[0014] S20: Map real-time incremental data and offline historical data to a unified logical time coordinate system, and determine the alignment boundary time to which the real-time incremental data belongs by combining the preset alignment time window parameters. In this embodiment, the logical time coordinate system refers to an abstract time representation used to unify the time base of different data sources. By mapping different timestamps to a unified logical time coordinate system, time deviations caused by differences in physical clocks, network latency, or processing latency can be eliminated, thereby achieving data alignment in the time dimension. The alignment time window parameter refers to a fixed-length time interval defined in the logical time coordinate system, used to discretize continuous event time streams or offline data update times and determine the alignment boundary to which the data belongs. For example, it can be set to 5 minutes, 1 hour, etc. All data falling within the alignment time window parameter will be considered to have occurred within the same time period. The alignment boundary time refers to the starting point of the time interval determined by the alignment time window parameter. When real-time incremental data is mapped to the logical time coordinate system, its event timestamp will be normalized to the nearest alignment boundary time according to the alignment time window parameter, serving as the unified time base for the real-time incremental data.

[0015] Specifically, one implementation is to convert all timestamps to Coordinated Universal Time (UTC) format as the basis for the logical time coordinate system; the alignment time window parameter can be preset to a fixed value, such as 1 hour; the alignment boundary time of real-time incremental data can be determined by rounding down its event timestamp to the starting point of the nearest alignment time window. For example, if the event timestamp is 10:35 and the alignment time window is 1 hour, then the alignment boundary time can be determined as 10:00.

[0016] S30: Perform predefined logical judgment on offline historical data to verify whether the business update timestamp falls within the closed-open interval starting from the alignment boundary time. If so, it is the alignment timestamp that is consistent with the offline historical data and the real-time incremental data. In this embodiment, the alignment timestamp refers to the timestamp associated with offline historical data that is consistent with the real-time incremental data. The alignment timestamp is generated based on the alignment boundary time of the real-time incremental data when the business update timestamp of the offline historical data meets specific conditions, and is used for data fusion on a unified time dimension.

[0017] Specifically, the logical determination can be accomplished by comparing the business update timestamp of the offline historical data with the alignment boundary time of the real-time incremental data. For example, it can be determined whether the business update timestamp of the offline historical data is greater than or equal to the alignment boundary time of the real-time incremental data, and less than the alignment boundary time plus the alignment time window parameter. If the above conditions are met, the alignment boundary time of the real-time incremental data is used as the alignment timestamp of the offline historical data for association.

[0018] S40: Identify the business primary key identifiers corresponding to real-time incremental data and offline historical data, and based on the business primary key identifiers and alignment timestamps, perform conflict detection and data fusion on the offline historical data and real-time incremental data that have been associated with the alignment timestamps, and generate fused data records; In this embodiment, the business primary key identifier refers to a set or a field used to uniquely identify a business entity. During the data fusion process, the business primary key identifier is used to associate records of the same business entity from different data sources and is the basis for data conflict detection and fusion. The fused data record refers to a new data record generated by real-time incremental data and offline historical data after conflict detection and data fusion processing. It contains the business payload that has been adjudicated and merged, representing the latest and most accurate state of the business entity at a specific point in time.

[0019] Specifically, business primary key identifiers can be predefined in the data model, such as order ID, user ID, etc. Conflict detection can be implemented by maintaining a hash table in memory, where the key is a combination of the business primary key identifier and the alignment timestamp. When new real-time incremental data or offline historical data is received, the hash table is queried to see if there is a record with the same key combination. If it exists, a conflict is determined and data fusion is triggered. Data fusion can adopt an overwriting strategy, for example, always overwriting the content of offline historical data with the content of real-time incremental data, or vice versa, to generate fused data records.

[0020] S50: Encapsulates the corresponding metadata extension information for the fused data record, generates a fused data stream, and outputs it to the preset downstream service layer to provide data access services.

[0021] In this embodiment, metadata extension information refers to auxiliary information attached to the fused data record, used to describe the source, processing process, quality, etc. of the fused data record. For example, it may include data source tracking identifier, fusion logic version number, computation processing delay statistics, and data quality assessment indicators, so as to facilitate data governance and backtracking in the downstream service layer. The downstream service layer refers to the application layer or data consumption layer that receives the fused data stream and provides data access services. It may include data query services, data analysis applications, data visualization platforms, etc., and provides high-quality fused data to the outside world through a unified interface.

[0022] Specifically, metadata extensions can be simple sets of key-value pairs, such as a field indicating the data source and a field indicating the processing time. This metadata can be directly appended to the end of the merged data record. The merged data stream can be sent to another message queue or written directly to storage media accessible to downstream service layers, such as a unified data lake or data warehouse, for use by services such as data querying, report generation, or real-time analysis.

[0023] This embodiment effectively solves the problem of lack of temporal consistency in the fusion of offline and real-time data in traditional data platforms by constructing a unified logical time coordinate system and alignment mechanism, ensuring the accuracy of cross-source data association. At the same time, the conflict detection and data fusion mechanism based on business primary key identifiers and alignment timestamps can handle conflicts in multi-source data, avoiding logical inconsistencies caused by manual intervention and simple overwriting. In addition, by encapsulating metadata extension information and outputting it to the downstream service layer, a unified and high-quality data view is provided for upper-layer applications, which has the effect of improving the credibility and flexibility of data services.

[0024] In one embodiment, such as Figure 2 As shown, step S20 includes: S21: Obtain the event timestamps built into the real-time incremental data and calculate the total offset of the event timestamps relative to the preset time coordinate origin; In this embodiment, when acquiring the event timestamps built into the real-time incremental data and calculating the total offset of the event timestamps relative to a preset time coordinate origin, the real-time incremental data usually has built-in event timestamps when it is generated or collected. These event timestamps are the actual time when the data is generated. In order to standardize the event timestamps and perform unified calculations, they need to be converted into the total offset relative to a preset time coordinate origin. The preset time coordinate origin can be a fixed historical time point, such as the Unix epoch or the system startup time. The total offset is the time difference between the event timestamp and the origin, usually expressed in a unified time unit such as milliseconds, seconds, or minutes.

[0025] S22: Perform a predefined modulo operation based on the total offset and the alignment time window parameters to generate the window offset; In this embodiment, when generating the window offset by performing a predefined modulo operation based on the total offset and the alignment time window parameter, the alignment time window parameter defines a fixed time period length, such as 5 minutes or 1 hour. The modulo operation is a mathematical operation used to calculate the remainder after dividing one number by another. By performing a modulo operation on the total offset of the real-time incremental data, it can be mapped to a relative position within the alignment time window, thereby generating the window offset. The window offset represents the relative position of the event timestamp within the current alignment time window and is a key intermediate quantity for determining the alignment boundary time.

[0026] S23: Calculate and generate the alignment boundary time based on the window offset and the alignment time window parameter, where the length of the closed-open interval is equal to the alignment time window parameter.

[0027] In this embodiment, the alignment boundary time refers to the starting point used to divide the time window. By subtracting the window offset from the total offset, the starting time of the time window to which the event timestamp belongs, i.e., the alignment boundary time, can be obtained. The alignment boundary time will serve as a unified time reference for all data within the window, ensuring that the data is aligned in logical time. The length of the closed-open interval is set to be equal to the alignment time window parameter, indicating that each time window has the same fixed length, and the starting point of the window is included, while the ending point is not included, thereby avoiding the repetition or omission of time points and improving the rigor of time window division.

[0028] Specifically, by acquiring the event timestamps built into the real-time incremental data and calculating their total offset relative to the preset time coordinate origin, and then performing a modulo operation based on the alignment time window parameters to generate the window offset, a unified alignment boundary time is finally calculated based on the window offset. This ensures that all real-time incremental data falling within the same preset time window can be mapped to the same logical time starting point, thereby providing an accurate and consistent time reference for subsequent offline historical data association and alignment timestamps and data fusion operations. This improves the accuracy and efficiency of data fusion and reduces or even avoids data conflicts or fusion errors caused by inaccurate time alignment.

[0029] In one embodiment, the computation topology operates on a directed acyclic graph structure across several cooperating computation nodes, and step S20 includes: S201: Obtain the current system physical time of the compute node and calculate the difference between the system physical time and the event timestamp built into the real-time incremental data; In this embodiment, a directed acyclic graph (DAG) is a commonly used computational model in distributed stream processing frameworks. It decomposes data processing tasks into a series of directed, acyclic computational operations or operators. Data flows unidirectionally along these computational operations or operators. This structure can clearly define the logical flow of data processing and supports parallel and distributed execution on multiple "cooperative computing nodes". These computing nodes can be physical servers, virtual machines, or containers. They communicate and coordinate through a network to jointly complete data processing tasks, thereby achieving high throughput and low latency data processing capabilities.

[0030] In this embodiment, system physical time refers to the local clock time maintained by the computing node currently executing the data processing task, which is typically provided by the operating system's hardware clock. In a distributed system, each computing node has its own independent system physical time. These system physical times may have slight differences due to hardware variations, operating system scheduling, or network time protocol synchronization errors, i.e., clock drift. Event timestamps refer to the time information carried by real-time incremental data when it is generated at the source. They reflect the actual moment the event occurred. By calculating the difference between the current processing time and the event occurrence time, the latency experienced by the data from generation to processing by the current node can be quantified. Furthermore, the calculated difference can reflect the network latency on the data transmission path, the queuing time in the message queue, and the data backlog that may be encountered during processing.

[0031] S202: Determine whether the calculated difference exceeds a preset deviation threshold. If it exceeds the deviation threshold, correct the calculated window offset according to a preset weighting factor to compensate for the delay caused by clock drift or data backlog.

[0032] In this embodiment, the deviation threshold is a configurable parameter used to define the maximum acceptable time difference between the system's physical time and the event timestamp. When the calculated difference exceeds the deviation threshold, it indicates a significant time deviation, which may indicate serious clock drift or data backlog issues, requiring intervention. Furthermore, the deviation threshold setting should be adjusted according to the specific requirements of the business for data timeliness and accuracy. The window offset is an intermediate value generated by a predefined modulo operation based on the event timestamp of the real-time incremental data and the alignment time window parameters. When a significant time deviation is detected, the window offset needs to be adjusted to compensate for this time deviation. The weighting factor is a pre-set value that determines the strength or proportion of the correction. For example, the window offset can be appropriately increased or decreased based on the magnitude and direction of the deviation, combined with a weighting factor, to more accurately reflect the actual time context of the data. Clock drift refers to the gradual deviation between the clocks of different computing nodes in a distributed system, which is difficult to completely eliminate even with a time synchronization mechanism. Data backlog refers to the situation where the input rate of the data stream exceeds the system's processing capacity, causing data to wait in the buffer or queue for a long time, thus introducing additional latency. By dynamically acquiring the system's physical time, calculating the difference with the event timestamp, determining whether it exceeds the deviation threshold, and deciding whether to make corrections based on the determination results, this application can effectively offset the impact of uncertainties on time calculation, thereby ensuring the accuracy of the alignment boundary time.

[0033] Specifically, by monitoring the difference between the system physical time of computing nodes and the timestamps of real-time incremental data events in real time, and setting a reasonable deviation threshold, potential time synchronization anomalies can be identified. Once a significant deviation is detected, the key window offset is corrected according to a preset weighting factor, thereby accurately compensating for the delay. This ensures that even in complex distributed topologies, the determined alignment boundary time remains highly accurate and reliable, avoiding data fusion errors caused by time inconsistencies. It improves the overall accuracy and consistency of the fusion of online data and offline historical data, providing downstream service layers with higher quality and more timely data access services.

[0034] In one embodiment, step S30 includes: S31: Using the alignment time window parameter as the time window length, perform alignment calculations on the event timestamps of the real-time incremental data to obtain the baseline alignment time; In this embodiment, step S31 aims to determine the standardized starting point of the time window, i.e., the baseline alignment time, from the event timestamps of the real-time incremental data and in combination with a preset alignment time window parameter. This baseline alignment time will serve as the time basis for subsequent judgment on whether offline historical data participates in the fusion. Specifically, the baseline alignment time can be calculated by dividing the event timestamp of the real-time incremental data by the alignment time window parameter, rounding down, and then multiplying by the alignment time window parameter. For example, if the alignment time window parameter is set to 5 minutes and the event timestamp of the real-time incremental data is 10:07:30, then the baseline alignment time will be calculated as 10:05:00. Alternatively, the baseline alignment time can also be obtained by performing a modulo operation on the event timestamp, i.e., subtracting the modulo of the event timestamp with respect to the alignment time window parameter from the event timestamp, ensuring that all real-time incremental data falling within the same time window will correspond to the same baseline alignment time.

[0035] S32: Determine whether the business update timestamp of the offline historical data is greater than or equal to the baseline alignment time and less than the sum of the baseline alignment time and the alignment time window parameter. If the condition is met, the baseline alignment time is used as the alignment timestamp and associated with the corresponding offline historical data. If the condition is not met, the corresponding offline historical data is associated with a flag indicating that it will not participate in the fusion.

[0036] In this embodiment, it is further determined whether the business update timestamp of the offline historical data is greater than or equal to the baseline alignment time and less than the sum of the baseline alignment time and the alignment time window parameter. The precise time range for offline historical data to participate in fusion is clearly defined, namely, a closed-open interval with the baseline alignment time as the starting point and the alignment time window parameter as the length. If the business update timestamp of the offline historical data falls within this closed-open interval, it indicates that the offline historical data is temporally correlated with the current real-time incremental data. At this time, the baseline alignment time is used as the alignment timestamp and associated with the corresponding offline historical data, so that the offline historical data can obtain an alignment identifier consistent with the real-time incremental data in the logical time coordinate system. Conversely, if the business update timestamp of the offline historical data does not meet the above conditions, a flag indicating that the corresponding offline historical data does not participate in fusion is associated with it. This flag can be a specific Boolean value, a null value, or a predefined string. Its function is to clearly indicate that the offline historical data should not participate in the subsequent data fusion process, thereby avoiding the processing of irrelevant data.

[0037] Specifically, using the alignment time window parameter as the time window length, alignment calculations are performed on the event timestamps of real-time incremental data to determine a clear baseline alignment time. This baseline alignment time serves as the starting point of the time window and, together with the alignment time window parameter, defines a precise closed-open interval. By determining whether the business update timestamps of offline historical data fall within this closed-open interval, accurate filtering of offline historical data can be achieved. Specifically, if the condition is met, the baseline alignment time is used as the alignment timestamp and associated with the corresponding offline historical data, ensuring effective alignment between real-time incremental data and offline historical data in the time dimension. Conversely, if the condition is not met, it is explicitly marked as not participating in the fusion, avoiding the processing of invalid data. Through the above logical judgment mechanism, the problem of quickly and accurately identifying historical data associated with real-time incremental data in massive offline historical data can be effectively solved, avoiding misjudgments caused by ambiguous time window definitions or inaccurate calculations, thereby improving the accuracy of data fusion. At the same time, by explicitly excluding offline historical data that does not participate in the fusion, the processing burden of subsequent conflict detection and data fusion stages can be reduced, improving the efficiency and resource utilization of the data fusion process.

[0038] In one embodiment, step S40 includes: S41: Based on the business primary key identifier, key-controlled partitioning is performed on real-time incremental data and offline historical data with associated aligned timestamps, and the business primary key identifier and the aligned timestamp form a composite key to maintain the corresponding existing data records in the stream processing state. In this embodiment, keyed partitioning refers to a commonly used data distribution strategy in distributed systems. Specifically, it routes data records with the same key to the same processing unit by performing hash calculations or other partitioning algorithms on one or more fields of the data record. In this embodiment, using the business primary key identifier as the basis for keyed partitioning ensures that all real-time incremental data and offline historical data of the same business entity, regardless of their source or arrival time, can be sent to the same processing logic or storage location, avoiding cross-partition data lookup and communication overhead, thereby improving processing efficiency. A composite key is a unique identifier composed of multiple fields. In this embodiment, the business primary key identifier and the alignment timestamp are combined into a composite key, which is used to uniquely identify and maintain each existing data record in the stream processing state. The stream processing state refers to the mechanism in the distributed stream processing framework used to store and manage intermediate computation results or historical data. It allows processing units to access and update previous data when processing new data. By using the composite key, the historical state of a specific business entity within a specific alignment time window can be accurately located, providing accurate contextual information for subsequent conflict detection and data fusion. This ensures that during the processing, the different states of the same business entity at different points in time can be distinguished, thereby achieving fine-grained data management.

[0039] S42: When new real-time incremental data or offline historical data is received, based on its business primary key identifier and alignment timestamp, query whether there is an existing data record with the same composite key in the stream processing status. If it exists, it is determined that a data conflict has occurred, and a preset conflict resolution process is triggered to generate a merged data record. If it does not exist, the current existing data record is stored in the stream processing status.

[0040] In this embodiment, when any new data record is received, its business primary key identifier and alignment timestamp are first extracted, and a query composite key is constructed accordingly. Further, the composite key is used to query whether a completely matching existing data record already exists in the current stream processing state. This query is the core of conflict detection, quickly determining whether the newly arrived data is duplicated or potentially conflicting with existing data under the same business primary key and alignment timestamp. If the query result indicates that an existing data record with the same composite key already exists in the stream processing state, it means that the newly arrived data is completely identical to existing data in terms of business primary key and alignment timestamp, and a data conflict is determined to have occurred. To resolve the data conflict, a preset conflict resolution process is triggered. This conflict resolution process is a predefined set of rules. This process is used to determine how to select, merge, or calculate the final business load from conflicting data records when a conflict occurs, thereby generating a merged data record. The conflict resolution process can adopt different strategies according to business needs, such as "based on new data", "based on old data", or "merging based on specific field priority", to ensure the logic and accuracy of data fusion. If the query results show that there are no existing data records with the same composite key in the stream processing state, it indicates that the newly arrived data is the first appearance of the business entity in the alignment time window or a completely new record. In this case, the new data record is directly stored in the stream processing state as the current existing data record, ensuring that all non-conflicting new data can be correctly included in the stream processing state management.

[0041] Specifically, by using keyed partitioning based on business primary key identifiers, data of the same business entity is logically processed centrally, significantly reducing the complexity and resource consumption of data comparison. Furthermore, by combining the business primary key identifier with the alignment timestamp to form a composite key and maintaining existing data records in the stream processing state, accurate identification of different data versions of the same business entity within a specific time window can be achieved, effectively avoiding data duplication or erroneous overwriting. When a data conflict is detected, a preset conflict resolution process is triggered to generate fused data records according to business rules, ensuring the accuracy and consistency of data fusion. This improves the efficiency and accuracy of data processing and provides conflict-free fused data for downstream services, thereby enhancing the stability and reliability of the data platform.

[0042] In one embodiment, the preset conflict resolution process includes: S421: Obtain and compare the sequence numbers corresponding to the real-time incremental data and offline historical data that have data conflicts, wherein the sequence number is a monotonically increasing value that reflects the order of execution of business logic; In this embodiment, the sequence number refers to a monotonically increasing value that reflects the order in which business logic is executed. For example, it can be a data version number, operation sequence number, or timestamp generated by the business system. By introducing the sequence number, a clear business timeline can be provided for each data record, so that a preliminary judgment can be made based on the order of business logic when a conflict occurs.

[0043] S422: If the sequence number of the real-time incremental data is higher than that of the offline historical data, the real-time incremental data is determined to have timeliness authority, and its business load is used as the business load of the fused data record. In this embodiment, if the sequence number of real-time incremental data is higher than that of offline historical data, the real-time incremental data is determined to have timeliness and authority, and its business payload is used as the business payload of the fused data record. This means that when the real-time incremental data has a higher sequence number, it will be considered as a newer or more authoritative business status, and its business payload will be adopted first to ensure that the fused data reflects the latest business facts.

[0044] S423: If the sequence number of the real-time incremental data is the same as that of the offline historical data, or if the sequence number of any data in the real-time incremental data or the offline historical data is missing, then a preset attribute-level fusion mode is entered, wherein the attribute-level fusion mode includes: The business load of real-time incremental data and offline historical data is decomposed into several atomic attribute fields respectively; For each atomic attribute field, based on the merging strategy corresponding to its predefined sensitivity level, field values ​​are selected or calculated from real-time incremental data and offline historical data where data conflicts occur, in order to generate the business payload of the fused data record.

[0045] In this embodiment, if the sequence number of the real-time incremental data is the same as that of the offline historical data, or if the sequence number of either the real-time incremental data or the offline historical data is missing, a preset attribute-level fusion mode is entered. This typically occurs when two data sets have the same point in time in terms of business logic, or when a reliable sequence number cannot be provided due to data source limitations. Specifically, the attribute-level fusion mode includes: decomposing the business payloads of the real-time incremental data and the offline historical data into several atomic attribute fields. For example, the business payload of a customer record may contain multiple fields such as name, address, phone number, and email address; these fields are treated as atomic attributes. By decomposing the business payload into these independent... Each field can be individually conflict-resolved. For each atomic attribute field, based on its predefined sensitivity level and corresponding merging strategy, field values ​​are selected or calculated from real-time incremental data and offline historical data where data conflicts have occurred to generate the business payload of the merged data record. For example, for the "Name" field, a "subject to offline editing" strategy may be adopted; for the "Inventory Quantity" field, a "cumulative" or "take the latest" strategy may be adopted; for the "Remarks" field, a "merge text" strategy may be adopted. The merging strategies mentioned above are usually pre-configured according to business needs and data characteristics, and can be differentiated according to the sensitivity level of the field.

[0046] Specifically, firstly, by comparing serial numbers, data with higher timeliness and authority can be effectively identified and adopted, thereby ensuring that the fused data reflects the latest business status. When the serial number cannot provide a clear priority judgment, such as when the serial numbers are the same or missing, the system switches to a preset attribute-level fusion mode. This attribute-level fusion mode allows for fine-grained processing of each atomic attribute field of the data record. Combined with predefined sensitivity levels and merging strategies, it ensures that each field can be merged in the most reasonable way. This not only avoids the information loss that may be caused by simply overwriting or discarding data, but also improves the accuracy, completeness, and business availability of the fused data records, providing high-quality data access services to the downstream service layer, thereby effectively supporting the online data offline editing and fusion needs of the data platform.

[0047] In one embodiment, step S50 includes: S51: Insert a metadata header of a preset length into the fused data record. The metadata header includes a data source tracking identifier, a fusion logic version number, computation processing delay statistics, and data quality assessment indicators. In this embodiment, a metadata header of a preset length is inserted for each fused data record. This metadata header includes a data source tracking identifier, used to uniquely identify the original source of the data, such as a specific database, data table, or offline editing task, thereby achieving end-to-end data traceability. Furthermore, the metadata header also includes a fusion logic version number, which records the version information of the algorithm or business logic used in the current data fusion. Additionally, the metadata header includes computational processing latency statistics, such as the timestamp from data reception to fusion completion or the total time consumed, which helps monitor the performance of the data processing pipeline and the freshness of the data. Furthermore, the metadata header also includes data quality assessment metrics, which can be confidence scores, error flags, or integrity percentages, used to convey the reliability level of the current fused data to the downstream service layer.

[0048] S52: Set a boolean edit rollback flag in the metadata header. If the current merged data record is a correction of the historical state triggered by offline editing, set the edit rollback flag to true to trigger the corresponding cache refresh logic in the downstream service layer.

[0049] In this embodiment, to effectively address scenarios where offline editing corrects historical data, a boolean edit backtracking flag is set in the metadata header. When it is detected that the current fused data record has been corrected due to an offline editing operation that modified an existing historical state, this edit backtracking flag will be set to true. For example, when offline historical data is edited and then fused with real-time incremental data, and the fusion result is an update of data at a certain point in the past, this edit backtracking flag is activated and directly triggers the preset cache refresh logic in the downstream service layer. The downstream service layer usually caches data to improve access efficiency, but the correction caused by offline editing means that some cached data may have become invalid. Through the edit backtracking flag, the downstream service layer can accurately identify the data that needs to be refreshed, thereby avoiding the provision of outdated information and optimizing the utilization of cache resources.

[0050] Specifically, the introduction of data source tracking identifiers, fusion logic version numbers, computational processing latency statistics, and data quality assessment indicators can improve data traceability, manageability, and transparency. More specifically, downstream service layers can more accurately understand the data lifecycle, processing procedures, and quality status based on these metadata elements, thereby optimizing data consumption strategies and improving the reliability of data services. Furthermore, the setting of a Boolean-type edit backtracking identifier can effectively solve the problem of inconsistent online service data after offline editing corrects historical data. When the edit backtracking identifier is true, the downstream service layer can promptly trigger the corresponding cache refresh logic, ensuring that online users always access the latest and corrected data, avoiding data inconsistencies or business errors caused by cache lag. This improves the real-time performance and accuracy of data services and significantly reduces the complexity of data synchronization caused by offline editing, thereby enhancing the user experience.

[0051] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0052] In one embodiment, a system for online data offline editing and fusion is provided, which corresponds one-to-one with a method for online data offline editing and fusion described in the above embodiments. The system for online data offline editing and fusion includes: The data acquisition module is used to construct a computing topology based on a distributed stream processing framework, acquire real-time incremental data and offline historical data with partition identifiers and business update timestamps. The real-time incremental data includes event timestamps and business payloads. The data mapping module is used to map real-time incremental data and offline historical data to a unified logical time coordinate system, and determine the alignment boundary time to which the real-time incremental data belongs by combining the preset alignment time window parameters. The logic determination module is used to perform predefined logic determinations on offline historical data, verifying whether the business update timestamp falls within the closed-open interval starting from the alignment boundary time. If so, it is the alignment timestamp that is consistent with the offline historical data and the real-time incremental data. The data generation module is used to identify the business primary key identifiers corresponding to real-time incremental data and offline historical data, and based on the business primary key identifiers and alignment timestamps, to perform conflict detection and data fusion on the offline historical data and real-time incremental data that have been associated with the alignment timestamps, and generate fused data records. The data stream output module is used to encapsulate the corresponding metadata extension information of the fused data records, generate the fused data stream, and output it to the preset downstream service layer to provide data access services.

[0053] For specific limitations regarding a system for online data offline editing and fusion, please refer to the limitations regarding a method for online data offline editing and fusion described above, which will not be repeated here. The various modules in the aforementioned system for online data offline editing and fusion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0054] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 3 As shown; the computer device includes a processor, memory, network interface, and database connected via a system bus; the processor provides computing and control capabilities; the memory includes a non-volatile storage medium and internal memory; the non-volatile storage medium stores an operating system, computer programs, and a database; the internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium; the network interface is used to communicate with external terminals via a network connection; when the computer program is executed by the processor, it implements a method for online data offline editing and fusion.

[0055] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for online data offline editing and fusion.

[0056] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a method for online data offline editing and fusion.

[0057] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for integrating online and offline data editing, applied to a data platform, characterized in that: Including the following steps: Construct a computing topology based on a distributed stream processing framework to acquire real-time incremental data and offline historical data with partition identifiers and business update timestamps. The real-time incremental data includes event timestamps and business payloads. Real-time incremental data and offline historical data are mapped to a unified logical time coordinate system, and the alignment boundary time to which the real-time incremental data belongs is determined by combining the preset alignment time window parameters. Perform predefined logical judgments on offline historical data to verify whether the business update timestamp falls within the closed-open interval starting from the alignment boundary time. If so, it is the alignment timestamp that is consistent between the offline historical data and the real-time incremental data. Identify the business primary key identifiers corresponding to real-time incremental data and offline historical data, and based on the business primary key identifiers and aligned timestamps, perform conflict detection and data fusion on the offline historical data and real-time incremental data that have been associated with aligned timestamps, and generate fused data records; The corresponding metadata extension information is encapsulated for the fused data records, generating a fused data stream, and outputting it to a preset downstream service layer to provide data access services.

2. The method for online data offline editing and fusion according to claim 1, characterized in that: The step of mapping real-time incremental data and offline historical data to a unified logical time coordinate system, and determining the alignment boundary time to which the real-time incremental data belongs, in conjunction with preset alignment time window parameters, includes the following steps: Obtain the event timestamps built into the real-time incremental data and calculate the total offset of the event timestamps relative to the preset time coordinate origin; A predefined modulo operation is performed based on the total offset and the alignment time window parameters to generate the window offset; The alignment boundary time is calculated based on the window offset and the alignment time window parameter, where the length of the closed-open interval is equal to the alignment time window parameter.

3. The method for online data offline editing and fusion according to claim 2, characterized in that: The computational topology operates on several cooperating computational nodes in a directed acyclic graph structure. The step of mapping real-time incremental data and offline historical data to a unified logical time coordinate system, and determining the alignment boundary time to which the real-time incremental data belongs, in conjunction with preset alignment time window parameters, includes the following steps: Obtain the current system physical time of the compute node and calculate the difference between the system physical time and the event timestamp built into the real-time incremental data; It is determined whether the calculated difference exceeds a preset deviation threshold. If it does, the calculated window offset is corrected according to a preset weighting factor to compensate for the delay caused by clock drift or data backlog.

4. The method for online data offline editing and fusion according to claim 1, characterized in that: The step of performing a predefined logical judgment on offline historical data to verify whether the business update timestamp falls within the closed-open interval starting from the alignment boundary time, and if so, the offline historical data is associated with an aligned timestamp consistent with the real-time incremental data, including the following steps: Using the alignment time window parameter as the time window length, perform alignment calculations on the event timestamps of real-time incremental data to obtain the baseline alignment time; Determine whether the business update timestamp of the offline historical data is greater than or equal to the baseline alignment time and less than the sum of the baseline alignment time and the alignment time window parameter. If the condition is met, the baseline alignment time is used as the alignment timestamp and associated with the corresponding offline historical data. If the condition is not met, a flag indicating that the corresponding offline historical data will not participate in the fusion is associated with it.

5. The method for online data offline editing and fusion according to claim 1, characterized in that: The steps of identifying the business primary key identifiers corresponding to real-time incremental data and offline historical data, and performing conflict detection and data fusion on the offline historical data and real-time incremental data with associated aligned timestamps based on the business primary key identifiers and aligned timestamps to generate fused data records include the following steps: Based on the business primary key identifier, keyed partitioning is performed on real-time incremental data and offline historical data with associated aligned timestamps, and a composite key is formed by the business primary key identifier and the aligned timestamp, and the corresponding existing data records are maintained in the stream processing state. When new real-time incremental data or offline historical data is received, based on its business primary key identifier and alignment timestamp, the system queries whether there is an existing data record with the same composite key in the stream processing status. If it exists, a data conflict is determined, and a preset conflict resolution process is triggered to generate a merged data record. If it does not exist, the current existing data record is stored in the stream processing status.

6. The method for online data offline editing and fusion according to claim 5, characterized in that: The preset conflict resolution process includes the following steps: Obtain and compare the sequence numbers corresponding to the real-time incremental data and offline historical data that have data conflicts. The sequence numbers are monotonically increasing values ​​that reflect the order of execution of business logic. If the sequence number of the real-time incremental data is higher than that of the offline historical data, the real-time incremental data is determined to have timeliness and authority, and its business load is used as the business load of the fused data record. If the sequence number of the real-time incremental data is the same as that of the offline historical data, or if the sequence number of any data in the real-time incremental data or the offline historical data is missing, then a preset attribute-level fusion mode is entered, wherein the attribute-level fusion mode includes: The business load of real-time incremental data and offline historical data is decomposed into several atomic attribute fields respectively; For each atomic attribute field, based on the merging strategy corresponding to its predefined sensitivity level, field values ​​are selected or calculated from real-time incremental data and offline historical data where data conflicts occur, in order to generate the business payload of the fused data record.

7. The method for online data offline editing and fusion according to claim 1, characterized in that: The steps of encapsulating corresponding metadata extension information on the fused data records, generating a fused data stream, and outputting it to a preset downstream service layer to provide data access services include the following steps: A metadata header of a preset length is inserted into the fused data record. The metadata header includes a data source tracking identifier, a fusion logic version number, computation processing delay statistics, and data quality assessment indicators. Set a boolean edit rollback flag in the metadata header. If the current merged data record is a correction of the historical state triggered by offline editing, set the edit rollback flag to true to trigger the corresponding cache refresh logic in the downstream service layer.

8. A system for online data offline editing and fusion, characterized in that, include: The data acquisition module is used to construct a computing topology based on a distributed stream processing framework, acquire real-time incremental data and offline historical data with partition identifiers and business update timestamps. The real-time incremental data includes event timestamps and business payloads. The data mapping module is used to map real-time incremental data and offline historical data to a unified logical time coordinate system, and determine the alignment boundary time to which the real-time incremental data belongs by combining the preset alignment time window parameters. The logic determination module is used to perform predefined logic determinations on offline historical data, verifying whether the business update timestamp falls within the closed-open interval starting from the alignment boundary time. If so, it is the alignment timestamp that is consistent with the offline historical data and the real-time incremental data. The data generation module is used to identify the business primary key identifiers corresponding to real-time incremental data and offline historical data, and based on the business primary key identifiers and the aligned timestamps, to perform conflict detection and data fusion on the offline historical data and real-time incremental data that have been associated with the aligned timestamps, and generate fused data records. The data stream output module is used to encapsulate the corresponding metadata extension information of the fused data records, generate the fused data stream, and output it to the preset downstream service layer to provide data access services.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a method for online data offline editing and fusion as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of a method for online data offline editing and fusion as described in any one of claims 1-7.

Images