Cross-system data synchronization method and system for cake electronic coupon mall connection system

Abstract

The application relates to a cross-system data synchronization method and system of a cake electronic coupon mall docking system, which comprises a local operation monitoring module and an interface adapter of each cooperative system deployed in the mall, captures heterogeneous data streams and sends the data streams to a message middleware; a pre-trained analysis model is used for intelligent analysis of the data streams, static templates and dynamic parameters are identified through the deep learning capacity of the analysis model, and a structured data stream in a unified format is output; based on a distributed stream computing framework, staged incremental calculation is performed on the structured data, including non-blocking merging and compression of DML operations and fusion calculation with DDL operations to generate an incremental data snapshot; and the snapshot is routed and synchronized to a target system according to a data source, so that stable and efficient bidirectional data synchronization between the cake electronic coupon mall and the cooperative system is realized, cross-system data collaboration capability is effectively improved, and data support is provided for core businesses such as electronic coupon state synchronization and inventory updating.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology, and in particular to a cross-system data synchronization method and system for a cake e-coupon mall docking system. Background Technology

[0002] An e-coupon mall is an online management platform for e-coupon resources. Its core business is the full lifecycle management of virtual vouchers such as cake vouchers, gift cards, and coupons. A typical business process begins with a user purchasing vouchers on the mall platform, triggering real-time updates and synchronization of voucher status. This extends to the redemption and fulfillment process when the user makes a purchase in-store. Each step establishes a continuous data exchange channel between the mall and multiple heterogeneous coupon identification systems. These cross-system operations form a complex data flow network, including key data interactions such as synchronizing the binding relationship between user accounts and coupon packages, real-time updates of coupon inventory status, and accurate transmission of redemption records. Because each system uses different data formats and technical architectures, data collaboration in this heterogeneous environment faces multiple challenges related to real-time performance, consistency, and reliability. Especially during peak promotional periods with massive concurrent transactions, traditional data synchronization solutions struggle to support the instantaneous surge in data flow processing demands, leading to delays in coupon status synchronization and lag in redemption result feedback, directly impacting user experience and platform reputation.

[0003] Currently, a common method in the industry for achieving this type of data synchronization is to use ETL tools based on scheduled batch processing. The e-coupon mall system periodically and proactively calls the query interfaces provided by various external systems to obtain full or timestamp-filtered incremental data snapshots. Subsequently, the system internally performs format conversion and mapping on the obtained heterogeneous data, and finally updates the processed data to the mall database in batches to maintain data consistency between the two sides.

[0004] Regarding the aforementioned technical methods, the ETL tools have poor real-time performance. The fixed synchronization cycle results in data delays of up to several minutes, making it impossible to reflect changes in user coupon status and coupon redemption results in a timely manner. Secondly, their data processing capabilities are insufficient. This method only focuses on data transfer and format conversion, lacking the ability to intelligently merge and compress continuous operation streams, resulting in the synchronization of a large amount of invalid intermediate state data and increasing unnecessary system load. Summary of the Invention

[0005] To address the issues of poor real-time performance of ETL tools, where fixed synchronization cycles result in data delays of up to several minutes, failing to reflect changes in user coupon status and coupon redemption results in a timely manner; secondly, their data processing capabilities are insufficient, as the method focuses only on data transfer and format conversion, lacking the ability to intelligently merge and compress continuous operation streams, leading to the synchronization of a large amount of invalid intermediate state data and increasing unnecessary system load, this application provides a cross-system data synchronization method and system for a cake e-coupon mall integration system.

[0006] Firstly, this application provides a cross-system data synchronization method for a cake e-coupon mall integration system, which adopts the following technical solution: including: The local operation module deployed in the cake e-coupon mall captures user transaction operations, generates and outputs a first raw data stream to the message middleware; the interface adapters deployed in each partner system subscribe to incremental data pushes from the partner system, generates and outputs a second raw data stream to the message middleware. The first raw data stream and the second raw data stream are input in parallel into a pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format. The structured data stream includes at least the data source, operation object, operation type, timestamp, and operation data body. The structured data stream is then published to a specified topic in the message middleware. The structured data stream is obtained from the message middleware. Based on the distributed stream computing framework, the structured data stream is subjected to phased incremental computation. The data records are classified according to the operation type. The Data Manipulation Language (DML) type operations are merged and compressed non-blockingly based on the operation object and time sequence to obtain intermediate results. At the same time, the Data Definition Language (DDL) type operations are fused with the intermediate results to generate a high-quality incremental data snapshot. Based on the data source, the route of the incremental data snapshot is determined and synchronized to the corresponding target system, thus completing the bidirectional data synchronization across systems.

[0007] Preferably, before inputting the first and second raw data streams in parallel into the pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format, the method further includes: Historical heterogeneous data streams are obtained from multiple partner systems to form the original training dataset, and the original training dataset is preprocessed, including word segmentation, word vector encoding, and sample clustering. Word segmentation involves segmenting each historical data record into words to obtain a word sequence; word vector encoding uses a character-level word embedding model to encode each word into a word vector; sample clustering uses the K-Means clustering algorithm to cluster the word vectors, generating multiple clusters, where data records within the same cluster are labeled as positive samples, and data records between different clusters are labeled as negative samples; To build a log text classifier and a context similarity comparator for the BiGRU-Attention model, the log text classifier includes an input layer, a hidden layer and an output layer, and the hidden layer includes a BiGRU layer and an Attention layer. For each target word, its context word vector is input into the BiGRU layer and the Attention layer to obtain the context encoding vector; By inputting the context encoding vector and the word vector of the target word into the output layer, the classification result of the target word belonging to the template or parameter is obtained; The similarity between the target word and the context encoding vectors of positive and negative samples is calculated using a context similarity comparator, and the contrast loss is calculated based on the information noise contrast estimation loss function. The total loss is obtained by combining the classification loss and contrast loss generated from the classification results, and the model parameters are updated through the backpropagation algorithm. When the model training reaches the preset convergence condition or number of iterations, the trained BiGRU-Attention parsing model is output.

[0008] Preferably, the step of inputting the first and second raw data streams in parallel into a pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format, and publishing the structured data stream to a designated topic in the message middleware, includes: Consume the first raw data stream and the second raw data stream in parallel from the message middleware; Each raw data record obtained from consumption is segmented into word sequences, and each word is encoded into a word vector using a character-level word embedding model. The word vector sequence is input into the pre-trained BiGRU-Attention parsing model, which classifies and predicts each word in the sequence, identifying whether it belongs to the static template part or the dynamic parameter part. The static template part is integrated into an event template, and the dynamic parameter part is integrated into a parameter list. The timestamp information in the original data record is also extracted. The event template, parameter list, timestamp information, along with the source system identifier, operation object identifier, and operation type identifier of the data record, are combined and encapsulated to generate the structured data stream in a unified format, and the structured data stream is published to the specified topic of the message middleware.

[0009] Preferably, the step of inputting the word vector sequence into the pre-trained BiGRU-Attention parsing model, whereby the parsing model classifies and predicts each word in the sequence to identify whether it belongs to the static template part or the dynamic parameter part, includes: The context word vector sequence of the target word is input into the hidden layer of the BiGRU-Attention parsing model; The BiGRU layer is used to learn the bidirectional semantic dependencies of the context word vector sequence and output the hidden state matrix. The hidden state matrix is ​​input into the Attention layer to calculate the attention weight of each context word, and a context encoding vector representing the contextual semantic features of the target word is generated by weighted summation. The context encoding vector is combined with the word vector of the target word itself to form the final feature representation; The final feature representation is input into the classifier of the output layer, and the classifier calculates the probability distribution of the target word belonging to the static template part or the dynamic parameter part; Based on the probability distribution, the category with the higher probability value is determined as the final classification result of the target word. The final classification result is used to determine the integration of the event template and the parameter list.

[0010] Preferably, the step of obtaining the structured data stream from the message middleware, performing phased incremental computation on the structured data stream based on a distributed stream computing framework, classifying data records according to operation type, and performing non-blocking merging and compression computation on Data Manipulation Language (DML) type operations based on operation object and time sequence to obtain intermediate results, and simultaneously fusing the Data Definition Language (DDL) type operations with the intermediate results to generate a high-quality incremental data snapshot, including: Consume the structured data stream from the specified topic of the message middleware, extract key information from each record in the structured data stream, and obtain its operation object identifier, operation type and timestamp; Based on the operation type, each data record is marked as a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation; data records marked as DML operations are routed to the first processing channel, and data records marked as DDL operations are routed to the second processing channel. The DML operation records in the first processing channel are grouped according to their operation object identifiers, and within each group, the DML operation records are sorted according to the timestamp order. Based on preset operation type merging rules, non-blocking merging and compression calculations are performed on consecutive DML operations on the same operation object to eliminate intermediate state data covered by subsequent operations and generate the intermediate calculation results. The intermediate calculation results are merged with the DDL operation records in the second processing channel according to the global timestamp. The merged data records are then globally sorted according to the timestamp. The sorted data records are then grouped according to the operation object identifier. Within each group, serial processing is performed in chronological order: when a DDL operation is encountered, its impact on the data structure is identified; state transitions or data migrations are performed on the DML operation records affected by the DDL operation to generate the high-quality incremental data snapshot.

[0011] Preferably, the non-blocking merging and compression calculation of consecutive DML operations on the same operation object based on preset operation type merging rules includes: For a sequence of DML operations on the same object, establish the operation state transition relationship according to the timestamp order; Based on preset state transition rules, consecutive operation sequences are merged and optimized. These state transition rules include: when an insert operation on the same object is immediately followed by an update operation, the insert and update operations are merged into a single insert operation, which contains the final data state after the update operation; when an insert operation on the same object is immediately followed by a delete operation, both the insert and delete operations are removed from the operation sequence; when an update operation on the same object is immediately followed by another update operation, the two update operations are merged into a single update operation, which reflects the final data change state; and when an update operation on the same object is immediately followed by a delete operation, the update and delete operations are merged into a single delete operation. Based on the execution result of the operation type merging rule, a compressed DML operation sequence is generated as the intermediate calculation result, wherein the original operation record being merged is replaced by the corresponding single operation record in the intermediate calculation result.

[0012] Preferably, the step of determining the route of the incremental data snapshot based on the data source and synchronizing it to the corresponding target system to complete cross-system bidirectional data synchronization includes: Based on the incremental data snapshot, the incremental data snapshot is routed to the target system according to the data source: If the data source is a partner system, the incremental data snapshot will be synchronously updated to the database of the cake e-coupon mall; If the data source is the cake e-coupon mall, the incremental data snapshot will be converted into a new format and then the corresponding interface adapter will be used to call the partner's system interface to perform the corresponding operation. This enables two-way, real-time data synchronization between the cake e-coupon mall and multiple heterogeneous partner systems.

[0013] Secondly, this application discloses a cross-system data synchronization device for a cake e-coupon mall integration system, which adopts the following technical solution, including: The data capture module is used to capture user transaction operations through the local operation module deployed in the cake e-coupon mall, generate and output a first raw data stream to the message middleware; and subscribe to incremental data pushes from partner systems through interface adapters deployed in each partner system, generate and output a second raw data stream to the message middleware. The model parsing module is used to input the first raw data stream and the second raw data stream in parallel into the pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format. The structured data stream includes at least the data source, operation object, operation type, timestamp, and operation data body. The module also publishes the structured data stream to a specified topic of the message middleware. The data compression module is used to obtain the structured data stream from the message middleware, perform phased incremental calculations on the structured data stream based on the distributed stream computing framework, classify data records according to operation type, and perform non-blocking merging and compression calculations on Data Manipulation Language (DML) type operations based on operation object and time sequence to obtain intermediate results. At the same time, it merges and calculates the Data Definition Language (DDL) type operations with the intermediate results to generate high-quality incremental data snapshots. The data synchronization module is used to determine the route of the incremental data snapshot based on the data source and synchronize it to the corresponding target system, thereby completing bidirectional data synchronization across systems.

[0014] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the cross-system data synchronization method of the cake e-coupon mall docking system described above.

[0015] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above regarding the cross-system data synchronization method of the cake e-coupon mall docking system.

[0016] In summary, this application captures heterogeneous incremental data streams from e-commerce user operations and various partner systems via local modules and interface adapters, and sends them to message middleware. The core component utilizes a pre-trained BiGRU-Attention parsing model to perform parallel parsing of multi-source data streams. Through its deep learning capabilities, it intelligently identifies and separates static templates and dynamic parameters, outputting a structured data stream in a unified format, thereby decoupling downstream processing from the upstream data source format. Subsequently, the system performs phased incremental computation on the structured data stream based on a distributed stream computing framework. First, it categorizes operations according to operation type and performs non-blocking merging and compression of Data Manipulation Language (DML) operations according to the operation object and time sequence to eliminate redundant intermediate states and generate intermediate results. Then, it fuses and computes DDL operations with these intermediate results, handling data state transitions caused by structural changes, and finally generates a high-quality incremental data snapshot. Finally, it intelligently routes this snapshot according to the data source and synchronizes it to the e-commerce database or the corresponding partner system, completing bidirectional data synchronization. Thus, through intelligent parsing and streaming incremental computation, the operation stream compression mechanism effectively filters redundant intermediate data, significantly reducing network transmission load and storage pressure. Simultaneously, DDL / DML fusion computation ensures business continuity during data structure changes. Regarding system reliability, the message queue-based persistence mechanism and the fault tolerance capabilities of the distributed computing framework jointly construct a highly available data pipeline, ensuring uninterrupted service and no data loss even in the event of a single point of failure. In terms of scalability, the deep learning parsing model has powerful generalization capabilities. When a new brokerage firm joins, only incremental updates to the model parameters are required instead of reconstructing the parsing logic, reducing system expansion costs by more than half and significantly accelerating business deployment. This effectively addresses the shortcomings of traditional methods in terms of heterogeneity, real-time performance, data quality, and scalability, achieving highly real-time, highly reliable, and easily scalable cross-system data synchronization, significantly improving the data processing capabilities and user experience of the electronic brokerage mall. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a cross-system data synchronization method for a cake e-coupon mall integration system.

[0018] Figure 2 This is a structural block diagram of a cross-system data synchronization device for a cake e-coupon mall integration system. Detailed Implementation

[0019] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0020] Existing technologies face inherent bottlenecks such as heterogeneity, latency, and data redundancy when handling data synchronization between electronic coupon malls and multiple systems. To address these issues, this application designs a data synchronization system that integrates intelligent sensing and real-time decision-making. The core of this system lies in constructing a full-link technical solution from data sensing, intelligent parsing, streaming processing to precise distribution. To enable those skilled in the art to fully understand and reproduce the preferred embodiments of this invention, the specific implementation details of the technical solution and the execution flow of key algorithms will be elaborated and explained below in conjunction with the accompanying drawings and examples.

[0021] Reference Figure 1 The embodiments of this application include at least steps S10 to S40.

[0022] S10 captures user transaction operations through the local operation module deployed in the cake e-coupon mall, generates and outputs the first raw data stream to the message middleware; and subscribes to the incremental data push of the partner system through the interface adapter deployed in each partner system, generates and outputs the second raw data stream to the message middleware.

[0023] S20: Input the first and second raw data streams in parallel into the pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format, and publish the structured data stream to the specified topic of the message middleware.

[0024] S30 obtains structured data streams from message middleware, performs phased incremental computations on the structured data streams based on a distributed stream computing framework, classifies data records according to operation types, and performs non-blocking merging and compression computations on Data Manipulation Language (DML) type operations based on operation objects and time sequences to obtain intermediate results. At the same time, it integrates DDL type operations with the intermediate results to generate high-quality incremental data snapshots.

[0025] S40 determines the route of incremental data snapshots based on the data source and synchronizes them to the corresponding target system, completing bidirectional data synchronization across systems.

[0026] The structured data stream includes at least the data source, the object of operation, the type of operation, the timestamp, and the operation data body.

[0027] Specifically, the process begins with the collection of heterogeneous data from multiple sources, unifying e-commerce operations and brokerage push data into a message middleware. Next, the SBiGRU-Attention parsing model transforms the heterogeneous data into a standardized data stream containing data source, operation object, type, timestamp, and data body. Then, based on a distributed stream computing framework, phased incremental computation is implemented, generating high-quality incremental data snapshots through classification processing, DML operation compression, and DDL / DML fusion computation. Finally, a routing mechanism accurately synchronizes the data snapshots to the target system. Thus, through the synergy of intelligent parsing and stream computing, the process from data access and intelligent processing to precise distribution is realized, effectively solving the challenges of real-time performance, reliability, and scalability in data synchronization between heterogeneous systems.

[0028] In some embodiments, the construction and training steps of the BiGRU-Attention parsing model are as follows for training the parsing model: Historical heterogeneous data streams are obtained from multiple partner systems to form the original training dataset. The original training dataset undergoes data preprocessing, including word segmentation, word vector encoding, and sample clustering. Word segmentation involves segmenting each historical data record to obtain a word sequence. Word vector encoding uses a character-level word embedding model to encode each word into a word vector. Sample clustering uses the K-Means clustering algorithm to cluster the word vectors, generating multiple clusters. Data records within the same cluster are labeled as positive samples, while data records between different clusters are labeled as negative samples. A log text classifier and context similarity comparison are then constructed for the BiGRU-Attention model. The log text classifier consists of an input layer, a hidden layer, and an output layer. The hidden layer includes a BiGRU layer and an Attention layer. For each target word, its context word vector is input into the BiGRU layer and the Attention layer to obtain a context encoding vector. The context encoding vector and the target word's word vector are input into the output layer to obtain the classification result of whether the target word belongs to the template or parameter. The similarity between the target word and the context encoding vectors of positive and negative samples is calculated using a context similarity comparator, and the contrastive loss is calculated based on the information noise contrastive estimation loss function. The classification loss and contrastive loss generated by the classification results are combined to obtain the total loss, and the model parameters are updated through the backpropagation algorithm. When the model training reaches the preset convergence condition or the number of iterations, the trained BiGRU-Attention parsing model is output.

[0029] The forward and backward hidden states of the BiGRU layer in the model are calculated according to the following formula: ; ; ; in This represents the input vector at time t. and These are the forward and reverse hidden states, respectively. and For the corresponding weighting coefficients, Let be the bias term at time t. Describe the final hidden state of the BiGRU at time t; The attention layer calculates attention weights and generates context encoding vectors using the following formula: ; ; ; in, The hidden state matrix, , , For attention parameters, The attention weights at time t are normalized using softmax; n is the sequence length. The hidden state at the i-th time step, s is the context encoding vector; the context encoding vector and the word vector of the target word are input to the output layer, and the classification probability is calculated by the sigmoid function. This yields the classification results for the target word belonging to the template or parameter; among which, This represents the context encoding vector of the target word i. The weight parameters of the classifier are represented. The bias parameters of the classifier.

[0030] The similarity between the target word and the context encoding vectors of positive and negative samples is calculated using a context similarity comparator, and the contrastive loss is calculated based on the information-noise contrastive estimation loss function. The calculation formula is as follows: ; in, This represents the information-noise contrast estimation loss, where k represents the sample size. The encoded vector of the query sample, For positive sample encoding vectors, For negative sample encoding vectors, The cosine similarity calculation function is used; the classification loss and contrast loss generated by the classification results are combined to obtain the total loss, and the model parameters are updated through the backpropagation algorithm; when the model training reaches the preset convergence condition or the number of iterations, the trained BiGRU-Attention parsing model is output.

[0031] Specifically, the model first performs word segmentation, character-level word vector encoding, and K-Means sample clustering on historical data from multiple securities firms to construct positive and negative sample sets. Then, a dual-path network architecture including a BiGRU layer and an Attention layer is established. Semantic features are captured through context-encoded vectors, and template and parameter classification is completed by combining the target word's own vector. Simultaneously, a context similarity comparator is introduced, generating a contrastive loss by calculating the similarity with positive and negative samples, which, together with the classification loss, constitutes the optimization objective. Finally, parameters are iteratively updated through backpropagation until the model converges. Thus, through the combination of deep learning and contrastive learning, the model can accurately identify static templates and dynamic parameters in heterogeneous data, laying the foundation for subsequent data synchronization.

[0032] In some embodiments, step S20 specifically includes the following steps: consuming the first and second raw data streams in parallel from the message middleware; performing word segmentation on each raw data record obtained from consumption, splitting it into a sequence of words, and encoding each word into a word vector using a character-level word embedding model; inputting the word vector sequence into a pre-trained BiGRU-Attention parsing model, where the parsing model classifies and predicts each word in the sequence, identifying whether it belongs to the static template part or the dynamic parameter part, integrating the static template part into an event template, integrating the dynamic parameter part into a parameter list, and extracting the timestamp information from the raw data record; combining and encapsulating the event template, parameter list, timestamp information, along with the source system identifier, operation object identifier, and operation type identifier of the data record, to generate a structured data stream in a unified format, and publishing the structured data stream to a specified topic in the message middleware.

[0033] Specifically, the system first acquires multi-source raw data streams in parallel from the message middleware. Each data record is then transformed into a word vector sequence through word segmentation and character-level word embedding encoding. The sequence is then input into a pre-trained BiGRU-Attention parsing model. This model uses deep semantic analysis to classify each word into templates or parameters, integrating them into event templates and parameter lists respectively. Simultaneously, timestamp information is extracted from the data records, and combined with metadata such as data source, operation object, and type, a structured data stream containing complete contextual information is finally assembled and redeployed to the message middleware. This achieves efficient conversion from multi-source heterogeneous data to a standard format, facilitating the provision of reliable data for subsequent streaming computation.

[0034] Furthermore, considering that the trained model calculates the classification result based on the target word, the corresponding steps are as follows: Input the context word vector sequence of the target word into the hidden layer of the BiGRU-Attention parsing model; learn the bidirectional semantic dependency relationship of the context word vector sequence through the BiGRU layer and output the hidden state matrix; input the hidden state matrix into the Attention layer, calculate the attention weight of each context word, and generate a context encoding vector representing the contextual semantic features of the target word through weighted summation; combine the context encoding vector with the target word's own word vector to form the final feature representation; input the final feature representation into the classifier of the output layer, and the classifier calculates the probability distribution of the target word belonging to the static template part or the dynamic parameter part; based on the probability distribution, determine the category with the higher probability value as the final classification result of the target word, and the final classification result is used to determine the integration of the event template and the parameter list.

[0035] Specifically, the process begins by inputting the context word vectors of the target word into a BiGRU layer to capture the bidirectional semantic dependencies between the preceding and following texts, generating a hidden state matrix. Next, an Attention layer calculates the attention weights of each context word, and a weighted average is used to obtain a context encoding vector that condenses semantic features. This vector is then fused with the target word's own word vector to form a composite feature representation that combines local features with global semantics. Finally, an output layer classifier calculates the probability distribution of the template and parameters, and the final classification result of the word is determined based on the principle of maximizing probability. Thus, by combining bidirectional semantic modeling with an attention mechanism, accurate identification of fixed patterns and dynamic content in data records is achieved, facilitating the subsequent generation of structured data.

[0036] In some embodiments, step S30 specifically includes the following steps: consuming a structured data stream from a specified topic in the message middleware; extracting key information from each record in the structured data stream to obtain its operation object identifier, operation type, and timestamp; marking each data record as a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation based on the operation type; routing the data records marked as DML operations to a first processing channel and the data records marked as DDL operations to a second processing channel; grouping the DML operation records in the first processing channel according to their operation object identifiers, and sorting the DML operation records within each group according to their timestamp order; Based on preset operation type merging rules, continuous DML operations on the same operation object are merged and compressed non-blockingly to eliminate intermediate state data covered by subsequent operations and generate intermediate calculation results. The intermediate calculation results are merged with the DDL operation records in the second processing channel according to the global timestamp. The merged data records are then globally sorted according to the timestamp. The sorted data records are grouped according to the operation object identifier. Within each group, serial processing is performed in chronological order: when a DDL operation is encountered, its impact on the data structure is identified; state transitions or data migrations are performed on the DML operation records affected by the DDL operation to generate high-quality incremental data snapshots.

[0037] Furthermore, based on preset operation type merging rules, non-blocking merging and compression calculations are performed on consecutive DML operations on the same operation object. Specifically, this includes: establishing operation state transition relationships according to timestamp order for a sequence of DML operations on the same operation object; merging and optimization of consecutive operation sequences based on preset state transition rules, which include: when an insert operation on the same operation object is immediately followed by an update operation, the insert and update operations are merged into a single insert operation, which contains the final data state after the update operation; when an insert operation on the same operation object is immediately followed by a delete operation, the insert and delete operations are removed from the operation sequence; when an update operation on the same operation object is immediately followed by another update operation, the two update operations are merged into a single update operation, which reflects the final data change state; when an update operation on the same operation object is immediately followed by a delete operation, the update and delete operations are merged into a single delete operation; and generating a compressed DML operation sequence as an intermediate calculation result based on the execution result of the operation type merging rules, where the original operation records that were merged are replaced by the corresponding single operation records in the intermediate calculation result.

[0038] Specifically, the system first categorizes and processes data records into DML and DDL operations: DML operations are grouped by operation object, and non-blocking merging calculations are performed based on time-series state transition rules. This compresses continuous sequences of insert-update, insert-delete, update-update, and update-delete operations into single operations, effectively eliminating intermediate state redundancy. Simultaneously, DDL operations are independently routed. After globally sorting intermediate DML results by timestamp, they are grouped and processed serially. The system identifies the impact of DDL structure changes on related DML data and performs state transitions accordingly. Through intelligent operation flow compression and DDL / DML fusion processing, the data synchronization load is significantly reduced, while ensuring data integrity and business continuity in scenarios involving table structure changes. This provides the e-coupon mall with high-throughput, low-latency data processing capabilities.

[0039] In some embodiments, step S40 specifically includes the following steps: based on the incremental data snapshot, the incremental data snapshot is routed to the target system according to the data source: if the data source is a partner system, the incremental data snapshot is synchronously updated to the database of the cake e-coupon mall; if the data source is the cake e-coupon mall, the incremental data snapshot is converted into a format and then the partner system interface is called through the corresponding interface adapter to perform the corresponding operation; thereby completing the bidirectional, real-time data synchronization between the cake e-coupon mall and multiple heterogeneous partner systems.

[0040] Specifically, the system dynamically routes data based on the data source attributes of incremental data snapshots. When the data originates from a partner system, the snapshot is directly synchronized to the e-commerce database. When the data originates from local operations within the e-commerce platform, the snapshot is formatted and then the corresponding interface adapter is used to call the partner system's interface to execute the operation. This unified pipeline enables the orderly scheduling of bidirectional data flows, ensuring both efficient aggregation of brokerage data to the e-commerce platform and reliable transmission of user commands to brokerages. Ultimately, this builds a data synchronization ecosystem, allowing the e-brokerage to maintain the heterogeneity and independence of each partner system while achieving a unified and real-time consistent overall data view.

[0041] The implementation principle of the cross-system data synchronization method for a cake e-coupon mall integration system in this application embodiment is as follows: The heterogeneous incremental data streams from mall user operations and various partner systems are captured through a local module and interface adapter and sent to a message middleware. The core step involves using a pre-trained BiGRU-Attention parsing model to perform parallel parsing of the multi-source data streams. Through its deep learning capabilities, it intelligently identifies and separates static templates and dynamic parameters, outputting a structured data stream in a unified format, thereby decoupling downstream processing from the upstream data source format. Subsequently, the system performs phased incremental computation on the structured data stream based on a distributed stream computing framework. First, it classifies operations according to operation type and performs non-blocking merging and compression of Data Manipulation Language (DML) operations according to the operation object and time sequence to eliminate redundant intermediate states and generate intermediate results. Then, it fuses and calculates the Data Definition Language (DDL) operations with these intermediate results, handling data state transitions caused by structural changes, and finally generating a high-quality incremental data snapshot. Finally, it intelligently routes this snapshot according to the data source and synchronizes it to the mall database or the corresponding partner system, completing bidirectional data synchronization. Thus, through intelligent parsing and streaming incremental computation, the operation stream compression mechanism effectively filters redundant intermediate data, significantly reducing network transmission load and storage pressure. Simultaneously, DDL / DML fusion computation ensures business continuity during data structure changes. Regarding system reliability, the message queue-based persistence mechanism and the fault tolerance capabilities of the distributed computing framework jointly construct a highly available data pipeline, ensuring uninterrupted service and no data loss even in the event of a single point of failure. In terms of scalability, the deep learning parsing model has powerful generalization capabilities. When a new brokerage firm joins, only incremental updates to the model parameters are required instead of reconstructing the parsing logic, reducing system expansion costs by more than half and significantly accelerating business deployment. This effectively addresses the shortcomings of traditional methods in terms of heterogeneity, real-time performance, data quality, and scalability, achieving highly real-time, highly reliable, and easily scalable cross-system data synchronization, significantly improving the data processing capabilities and user experience of the electronic brokerage mall.

[0042] Figure 1 This is a flowchart illustrating a cross-system data synchronization method for a cake e-coupon mall integration system in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0043] Based on the same technical concept, referring to Figure 2 This application also provides a cross-system data synchronization device for a cake e-coupon mall integration system, which adopts the following technical solution: The device includes: The data capture module is used to capture user transaction operations through the local operation module deployed in the cake e-coupon mall, generate and output the first raw data stream to the message middleware; and subscribe to the incremental data push of the partner system through the interface adapters deployed in each partner system, generate and output the second raw data stream to the message middleware. The model parsing module is used to input the first and second raw data streams in parallel into the pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format. The structured data stream contains at least the data source, operation object, operation type, timestamp, and operation data body, and publishes the structured data stream to the specified topic of the message middleware. The data compression module is used to obtain structured data streams from message middleware. Based on a distributed stream computing framework, it performs phased incremental computation on the structured data streams, classifies data records according to operation type, and performs non-blocking merging and compression computation on Data Manipulation Language (DML) type operations based on operation objects and time sequence to obtain intermediate results. At the same time, it integrates Data Definition Language (DDL) type operations with intermediate results to generate high-quality incremental data snapshots. The data synchronization module is used to determine the route of incremental data snapshots based on the data source and synchronize them to the corresponding target system, thereby completing bidirectional data synchronization across systems.

[0044] In some embodiments, the model parsing module is also used to obtain historical heterogeneous data streams from multiple partner systems to form an original training dataset, and to perform data preprocessing on the original training dataset, including: word segmentation, word vector encoding, and sample clustering; Word segmentation involves segmenting each historical data record into words to obtain a word sequence; word vector encoding uses a character-level word embedding model to encode each word into a word vector; sample clustering uses the K-Means clustering algorithm to cluster word vectors, generating multiple clusters, where data records within the same cluster are labeled as positive samples, and data records between different clusters are labeled as negative samples. To build a log text classifier and a context similarity comparator for the BiGRU-Attention model, the log text classifier includes an input layer, a hidden layer and an output layer, and the hidden layer includes a BiGRU layer and an Attention layer. For each target word, its context word vector is input into the BiGRU layer and the Attention layer to obtain the context encoding vector; By inputting the context encoding vector and the word vector of the target word into the output layer, the classification result of the target word belonging to the template or parameter is obtained; The similarity between the target word and the context encoding vectors of positive and negative samples is calculated using a context similarity comparator, and the contrast loss is calculated based on the information noise contrast estimation loss function. The total loss is obtained by combining the classification loss and contrast loss generated from the classification results, and the model parameters are updated through the backpropagation algorithm. When the model training reaches the preset convergence condition or number of iterations, the trained BiGRU-Attention parsing model is output.

[0045] In some embodiments, the model parsing module is specifically used to consume the first raw data stream and the second raw data stream in parallel from the message middleware; Each raw data record obtained from consumption is segmented into word sequences, and each word is encoded into a word vector using a character-level word embedding model. The word vector sequence is input into a pre-trained BiGRU-Attention parsing model. The parsing model classifies and predicts each word in the sequence, identifying whether it belongs to the static template part or the dynamic parameter part. The static template part is integrated into an event template, and the dynamic parameter part is integrated into a parameter list. The timestamp information in the original data record is also extracted. The event template, parameter list, and timestamp information, along with the source system identifier, operation object identifier, and operation type identifier of the data record, are combined and encapsulated to generate a structured data stream in a unified format, and the structured data stream is published to the specified topic of the message middleware.

[0046] In some embodiments, the model parsing module is also used to input the context word vector sequence of the target word into the hidden layer of the BiGRU-Attention parsing model; The BiGRU layer learns the bidirectional semantic dependencies of the context word vector sequence and outputs the hidden state matrix. The hidden state matrix is ​​input into the Attention layer, the attention weight of each context word is calculated, and a context encoding vector representing the contextual semantic features of the target word is generated by weighted summation. The context encoding vector is combined with the word vector of the target word itself to form the final feature representation; The final feature representation is input into the classifier of the output layer, and the classifier calculates the probability distribution of the target word belonging to the static template part or the dynamic parameter part; Based on the probability distribution, the category with the higher probability value is determined as the final classification result of the target word. The final classification result is used to determine the integration of the event template and the parameter list.

[0047] In some embodiments, the data compression module is specifically used to consume structured data streams from a specified topic of the message middleware, extract key information from each record in the structured data stream, and obtain its operation object identifier, operation type, and timestamp. Based on the operation type, each data record is marked as a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation; data records marked as DML operations are routed to the first processing channel, and data records marked as DDL operations are routed to the second processing channel. The DML operation records in the first processing channel are grouped according to their operation object identifiers, and within each group, the DML operation records are sorted according to the timestamp order. Based on preset operation type merging rules, non-blocking merging and compression calculations are performed on consecutive DML operations on the same operation object to eliminate intermediate state data covered by subsequent operations and generate intermediate calculation results. The intermediate calculation results are merged with the DDL operation records in the second processing channel according to the global timestamp. The merged data records are then globally sorted according to the timestamp. The sorted data records are then grouped according to the operation object identifier. Within each group, serial processing is performed in chronological order: when a DDL operation is encountered, its impact on the data structure is identified; state transitions or data migrations are performed on the DML operation records affected by the DDL operation to generate high-quality incremental data snapshots.

[0048] In some embodiments, the data compression module is also used to establish operation state transition relationships for a sequence of DML operations on the same operation object according to the timestamp order; Based on preset state transition rules, continuous operation sequences are merged and optimized. These rules include: when an insert operation on the same object is immediately followed by an update operation, the insert and update operations are merged into a single insert operation, which contains the final data state after the update operation; when an insert operation on the same object is immediately followed by a delete operation, both the insert and delete operations are removed from the operation sequence; when an update operation on the same object is immediately followed by another update operation, the two update operations are merged into a single update operation, which reflects the final data change state; and when an update operation on the same object is immediately followed by a delete operation, the update and delete operations are merged into a single delete operation. Based on the execution result of the operation type merging rule, a compressed DML operation sequence is generated as an intermediate calculation result, in which the original operation record being merged is replaced by the corresponding single operation record in the intermediate calculation result.

[0049] In some embodiments, the data synchronization module is further configured to route incremental data snapshots to the target system based on the data source: If the data source is a partner system, the incremental data snapshot will be synchronously updated to the database of the cake e-coupon mall; If the data source is the cake e-coupon mall, the incremental data snapshot will be converted into a new format and then the corresponding interface adapter will be used to call the partner's system interface to perform the corresponding operation. This enables two-way, real-time data synchronization between the cake e-coupon mall and multiple heterogeneous partner systems.

[0050] This application also discloses a control device.

[0051] Specifically, the control device includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and executed to implement the cross-system data synchronization method of the aforementioned cake e-coupon mall docking system.

[0052] This application also discloses a computer-readable storage medium.

[0053] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the cross-system data synchronization method of the aforementioned cake e-coupon mall docking system. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A cross-system data synchronization method for a cake e-coupon mall integration system, characterized in that, include: The local operation module deployed in the cake e-coupon mall captures user transaction operations, generates and outputs a first raw data stream to the message middleware; the interface adapters deployed in each partner system subscribe to incremental data pushes from the partner system, generates and outputs a second raw data stream to the message middleware. To build a log text classifier and a context similarity comparator for the BiGRU-Attention model, the log text classifier includes an input layer, a hidden layer and an output layer, and the hidden layer includes a BiGRU layer and an Attention layer. For each target word, its context word vector is input into the BiGRU layer and the Attention layer to obtain the context encoding vector; By inputting the context encoding vector and the word vector of the target word into the output layer, the classification result of the target word belonging to the template or parameter is obtained; The similarity between the target word and the context encoding vectors of positive and negative samples is calculated using a context similarity comparator, and the contrast loss is calculated based on the information noise contrast estimation loss function. The total loss is obtained by combining the classification loss and contrast loss generated from the classification results, and the model parameters are updated through the backpropagation algorithm. When the model training reaches the preset convergence condition or number of iterations, the trained BiGRU-Attention parsing model is output. The first raw data stream and the second raw data stream are input in parallel into a pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format. The structured data stream includes at least the data source, operation object, operation type, timestamp, and operation data body. The structured data stream is then published to a specified topic in the message middleware. The structured data stream is obtained from the message middleware. Based on the distributed stream computing framework, phased incremental computation is performed on the structured data stream. Data records are classified according to operation type. Data manipulation language (DML) type operations are merged and compressed non-blockingly based on operation object and time sequence to obtain intermediate results. At the same time, data definition language (DDL) type operations are fused with the intermediate results to generate a high-quality incremental data snapshot. Specifically, the structured data stream is consumed from the specified topic of the message middleware. Key information is extracted from each record in the structured data stream to obtain its operation object identifier, operation type and timestamp. Based on the operation type, each data record is marked as a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation; data records marked as DML operations are routed to the first processing channel, and data records marked as DDL operations are routed to the second processing channel. The DML operation records in the first processing channel are grouped according to their operation object identifiers, and within each group, the DML operation records are sorted according to the timestamp order. Based on preset operation type merging rules, non-blocking merging and compression calculations are performed on consecutive DML operations on the same operation object to eliminate intermediate state data covered by subsequent operations and generate the intermediate result. The intermediate results are merged with the DDL operation records in the second processing channel according to the global timestamp. The merged data records are then globally sorted according to the timestamp. The sorted data records are then grouped according to the operation object identifier. Within each group, serial processing is performed in chronological order: when a DDL operation is encountered, its impact on the data structure is identified; state transitions or data migrations are performed on the DML operation records affected by the DDL operation to generate the high-quality incremental data snapshot. Based on the data source, the route of the incremental data snapshot is determined and synchronized to the corresponding target system, thus completing the bidirectional data synchronization across systems.

2. The cross-system data synchronization method for a cake e-coupon mall integration system according to claim 1, characterized in that, Before inputting the first and second raw data streams in parallel into the pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format, the method further includes: Historical heterogeneous data streams are obtained from multiple partner systems to form the original training dataset, and the original training dataset is preprocessed, including word segmentation, word vector encoding, and sample clustering. Word segmentation involves segmenting each historical data record into words to obtain a word sequence; word vector encoding uses a character-level word embedding model to encode each word into a word vector; sample clustering uses the K-Means clustering algorithm to cluster the word vectors, generating multiple clusters, where data records within the same cluster are labeled as positive samples, and data records between different clusters are labeled as negative samples.

3. The cross-system data synchronization method for a cake e-coupon mall integration system according to claim 2, characterized in that, The step of inputting the first and second raw data streams in parallel into a pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format, and publishing the structured data stream to a designated topic in the message middleware, includes: Consume the first raw data stream and the second raw data stream in parallel from the message middleware; Each raw data record obtained from consumption is segmented into word sequences, and each word is encoded into a word vector using a character-level word embedding model. The word vector sequence is input into the pre-trained BiGRU-Attention parsing model, which classifies and predicts each word in the sequence, identifying whether it belongs to the static template part or the dynamic parameter part. The static template part is integrated into an event template, and the dynamic parameter part is integrated into a parameter list. The timestamp information in the original data record is also extracted. The event template, parameter list, timestamp information, along with the source system identifier, operation object identifier, and operation type identifier of the data record, are combined and encapsulated to generate the structured data stream in a unified format, and the structured data stream is published to the specified topic of the message middleware.

4. The cross-system data synchronization method for a cake e-coupon mall integration system according to claim 3, characterized in that, The step of inputting the word vector sequence into the pre-trained BiGRU-Attention parsing model, whereby the parsing model classifies and predicts each word in the sequence to identify whether it belongs to the static template part or the dynamic parameter part, includes: The context word vector sequence of the target word is input into the hidden layer of the BiGRU-Attention parsing model; The BiGRU layer is used to learn the bidirectional semantic dependencies of the context word vector sequence and output the hidden state matrix. The hidden state matrix is ​​input into the Attention layer to calculate the attention weight of each context word, and a context encoding vector representing the contextual semantic features of the target word is generated by weighted summation. The context encoding vector is combined with the word vector of the target word itself to form the final feature representation; The final feature representation is input into the classifier of the output layer, and the classifier calculates the probability distribution of the target word belonging to the static template part or the dynamic parameter part; Based on the probability distribution, the category with the highest probability value is determined as the final classification result of the target word. The final classification result is used to determine the integration of the event template and the parameter list.

5. The cross-system data synchronization method for a cake e-coupon mall integration system according to claim 1, characterized in that, The method of performing non-blocking merging and compression calculations on consecutive DML operations of the same operation object based on preset operation type merging rules includes: For a sequence of DML operations on the same object, establish the operation state transition relationship according to the timestamp order; Based on preset state transition rules, consecutive operation sequences are merged and optimized. These state transition rules include: when an insert operation on the same object is immediately followed by an update operation, the insert and update operations are merged into a single insert operation, which contains the final data state after the update operation; when an insert operation on the same object is immediately followed by a delete operation, both the insert and delete operations are removed from the operation sequence; when an update operation on the same object is immediately followed by another update operation, the two update operations are merged into a single update operation, which reflects the final data change state; and when an update operation on the same object is immediately followed by a delete operation, the update and delete operations are merged into a single delete operation. Based on the execution result of the operation type merging rule, a compressed DML operation sequence is generated as the intermediate result, wherein the original operation records that were merged are replaced by the corresponding single operation records in the intermediate result.

6. The cross-system data synchronization method for a cake e-coupon mall integration system according to claim 1, characterized in that, The step of determining the route of the incremental data snapshot based on the data source and synchronizing it to the corresponding target system to complete cross-system bidirectional data synchronization includes: Based on the incremental data snapshot, the incremental data snapshot is routed to the target system according to the data source: If the data source is a partner system, the incremental data snapshot will be synchronously updated to the database of the cake e-coupon mall; If the data source is the cake e-coupon mall, the incremental data snapshot will be converted into a new format and then the corresponding interface adapter will be used to call the partner's system interface to perform the corresponding operation. This enables two-way, real-time data synchronization between the cake e-coupon mall and multiple heterogeneous partner systems.

7. A cross-system data synchronization device for a cake e-coupon mall integration system, characterized in that, The device includes: The data capture module is used to capture user transaction operations through the local operation module deployed in the cake e-coupon mall, generate and output a first raw data stream to the message middleware; and subscribe to incremental data pushes from partner systems through interface adapters deployed in each partner system, generate and output a second raw data stream to the message middleware. The model parsing module is used to input the first raw data stream and the second raw data stream in parallel into the pre-trained BiGRU-Attention parsing model to obtain a structured data stream in a unified format. The structured data stream includes at least the data source, operation object, operation type, timestamp, and operation data body. The module also publishes the structured data stream to a specified topic of the message middleware. The data compression module is used to obtain the structured data stream from the message middleware, perform phased incremental calculations on the structured data stream based on the distributed stream computing framework, classify data records according to operation type, and perform non-blocking merging and compression calculations on Data Manipulation Language (DML) type operations based on operation object and time sequence to obtain intermediate results. At the same time, it merges and calculates the Data Definition Language (DDL) type operations with the intermediate results to generate high-quality incremental data snapshots. The data synchronization module is used to determine the route of the incremental data snapshot according to the data source and synchronize it to the corresponding target system, thereby completing bidirectional data synchronization across systems. The model parsing module is also used to build a log text classifier and a context similarity comparator for the BiGRU-Attention model. The log text classifier includes an input layer, a hidden layer and an output layer. The hidden layer includes a BiGRU layer and an Attention layer. For each target word, its context word vector is input into the BiGRU layer and the Attention layer to obtain the context encoding vector; By inputting the context encoding vector and the word vector of the target word into the output layer, the classification result of the target word belonging to the template or parameter is obtained; The similarity between the target word and the context encoding vectors of positive and negative samples is calculated using a context similarity comparator, and the contrast loss is calculated based on the information noise contrast estimation loss function. The total loss is obtained by combining the classification loss and contrast loss generated from the classification results, and the model parameters are updated through the backpropagation algorithm. When the model training reaches the preset convergence condition or number of iterations, the trained BiGRU-Attention parsing model is output. Specifically, the data compression module is used to consume the structured data stream from a specified topic of the message middleware, extract key information from each record in the structured data stream, and obtain its operation object identifier, operation type and timestamp; Based on the operation type, each data record is marked as a Data Manipulation Language (DML) operation or a Data Definition Language (DDL) operation; data records marked as DML operations are routed to the first processing channel, and data records marked as DDL operations are routed to the second processing channel. The DML operation records in the first processing channel are grouped according to their operation object identifiers, and within each group, the DML operation records are sorted according to the timestamp order. Based on preset operation type merging rules, non-blocking merging and compression calculations are performed on consecutive DML operations on the same operation object to eliminate intermediate state data covered by subsequent operations and generate the intermediate result. The intermediate results are merged with the DDL operation records in the second processing channel according to the global timestamp. The merged data records are then globally sorted according to the timestamp. The sorted data records are then grouped according to the operation object identifier. Within each group, serial processing is performed in chronological order: when a DDL operation is encountered, its impact on the data structure is identified; state transitions or data migrations are performed on the DML operation records affected by the DDL operation to generate the high-quality incremental data snapshot.

8. A control device, characterized in that, The device includes: A memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 6.

Images