Data synchronization method and device, equipment, storage medium and program product
By defining a standardized message body format and implementing multiple verification processes, the problems of low data synchronization efficiency and insufficient accuracy are solved, achieving efficient, reliable, and orderly data synchronization in distributed systems.
Patent Information
- Application Number
- CN202511202522.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-26
- Publication Date
- 2025-12-19
AI Technical Summary
In existing technologies, data synchronization in distributed systems is inefficient, inaccurate, and involves complex message processing logic, which can easily lead to problems such as duplicate message processing, version conflicts causing data overwriting and loss.
Define a standardized message body format, listen to the message body in the message queue, parse the message body information, and perform duplicate information verification, expiration verification, and validity verification to ensure efficient, reliable, and orderly data synchronization.
It improves the efficiency and accuracy of data synchronization, avoids message duplication, version conflicts and data loss, and ensures the reliability and maintainability of the system.
Smart Images
Figure CN121173818A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of distributed technology, and particularly relates to a data synchronization method and device, equipment, a storage medium and a program product. BACKGROUND
[0002] Data synchronization is the key to maintaining data consistency in distributed systems. In microservice architecture and cross-system data interaction scenarios, asynchronous message middleware is used to achieve data consistency.
[0003] In the prior art, the producer sends change events as messages to the message middleware, and the consumer subscribes to these messages and updates the corresponding database according to these messages. However, the message processing mechanism lacks standardized specifications, resulting in complex and difficult-to-maintain message processing logic, which may cause message duplication processing, version conflict, data coverage and loss, and other situations, affecting system reliability and maintainability.
[0004] Therefore, in the prior art, there is a problem of low data synchronization efficiency and insufficient accuracy. SUMMARY
[0005] The present application provides a data synchronization method, device, equipment, storage medium and program product to solve the technical problem of low data synchronization efficiency and insufficient accuracy.
[0006] In a first aspect, the present application provides a data synchronization method applied to a message consumer, comprising:
[0007] Listening to the message body in the message queue; wherein the message body is obtained by the message producer in response to the information change operation, and the message body is sent to the message queue after the message body information is generated according to the message body information; wherein the message body information includes primary key information, content information, time information, version number information and retry count information;
[0008] Parsing the message body to obtain the message body information of the message body;
[0009] According to the message body information, the message body is subjected to repeated information checking processing;
[0010] If the repeated information checking passes, the message body is subjected to expiration checking processing according to the message body information;
[0011] If the expiration checking passes, the message body is subjected to validity checking processing according to the message body information;
[0012] If the validity checking passes, the data synchronization operation is executed according to the message body.
[0013] In a second aspect, the present application provides a data synchronization method applied to a message producer, comprising:
[0014] In response to the information change operation, the message body information is acquired; wherein the message body information includes primary key information, content information, time information, version number information and retry count information;
[0015] The message body is generated according to the message body information;
[0016] The message body is sent to the message queue, so that the message consumer listens to the message body in the message queue, parses the message body to obtain the message body information of the message body, performs repeated information checking processing on the message body according to the message body information, if the repeated information checking passes, performs expired checking processing on the message body according to the message body information, if the expired checking passes, performs valid checking processing on the message body according to the message body information, if the valid checking passes, executes the data synchronization operation according to the message body.
[0017] In a third aspect, the present application provides a data synchronization device, comprising:
[0018] The listening module is configured to listen to the message body in the message queue; wherein the message body is sent to the message queue by the message producer in response to the information change operation, and the message body information is acquired, and the message body is generated according to the message body information; wherein the message body information includes primary key information, content information, time information, version number information and retry count information;
[0019] The parsing module is configured to parse the message body to obtain the message body information of the message body;
[0020] The first checking module is configured to perform repeated information checking processing on the message body according to the message body information;
[0021] The second checking module is configured to perform expired checking processing on the message body according to the message body information if the repeated information checking passes;
[0022] The third checking module is configured to perform valid checking processing on the message body according to the message body information if the expired checking passes;
[0023] The execution module is configured to execute the data synchronization operation according to the message body if the valid checking passes.
[0024] In a fourth aspect, the present application provides an electronic device, comprising: a processor, and a memory connected with the processor in communication;
[0025] The memory stores computer execution instructions;
[0026] The processor executes the computer execution instructions stored in the memory to realize various possible implementation manners of the first aspect or the second aspect.
[0027] In a fifth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the various possible implementation manners of the first aspect or the second aspect.
[0028] In a sixth aspect, the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the various possible implementation manners of the first aspect or the second aspect.
[0029] The data synchronization method, device, equipment, storage medium and program product provided by the present application can listen to the message body in the message queue, analyze the message body to obtain the message body information of the message body, and sequentially perform repeated information checking processing, expiration checking processing and validity checking processing on the message body. If the checking is passed, the data synchronization operation is performed according to the message body. Compared with the prior art, the method of the present application solves the problems of message repetition, disorder of version sequence processing and imperfect retry mechanism in the prior art by defining a standardized message body format and multiple checking, improves the efficiency and accuracy of data synchronization, and ensures efficient, reliable and orderly synchronization of data in a distributed system. BRIEF DESCRIPTION OF DRAWINGS
[0030] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0031] Figure 1 Flowchart of a data synchronization method provided by the present application Figure 1 ;
[0032] Figure 2 Flowchart of a data synchronization method provided by the present application Figure 2 ;
[0033] Figure 3 Flowchart of a data synchronization method provided by the present application Figure 3 ;
[0034] Figure 4 Structure diagram of a data synchronization device provided by the present application
[0035] Figure 5 Structure diagram of an electronic device provided by an embodiment of the present application.
[0036] The above drawings have shown the specific embodiments of the present application, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0037] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not representative of all possible embodiments consistent with the present application. Instead, it is merely intended to provide an example of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.
[0038] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, public order and good customs are not violated, and appropriate operation portals are provided for users to choose authorization or refusal.
[0039] And the present application involves big data analysis of user information (including but not limited to personal biological characteristics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and uses artificial intelligence technology for automatic decision-making, and makes technical solutions based on automatic decision-making results that have a significant impact on personal rights and interests, provides appropriate operation portals for users to choose to agree or refuse automatic decision-making results; if the user chooses to refuse, enter the expert decision-making process.
[0040] It should be noted that the data synchronization method, electronic device, storage medium and program product provided by the present application can be used in the field of distributed technology, and can also be used in any field other than the field of distributed technology. The application field of the data synchronization method, electronic device, storage medium and program product in the present application is not limited.
[0041] Message middleware is a software component based on distributed system architecture, mainly used to realize functions including asynchronous communication, decoupling integration, and traffic peak clipping between different application programs, services or systems. By introducing a message queue between the sender and the receiver, the sender and the receiver do not need to interact directly, but complete data exchange by passing messages.
[0042] Therefore, in the micro-service architecture and cross-system data interaction scenario, an asynchronous message passing mechanism is used to process data update and synchronization, wherein the asynchronous message passing mechanism refers to that the producer sends change events as messages to the message middleware, and the consumer subscribes to these messages and updates the data in the corresponding database according to the subscribed messages to realize data synchronization.
[0043] However, the existing data synchronization scheme lacks a standardized message body format, and different consumers need to parse the obtained message body through their own parsing logic, which leads to complex message processing logic and difficult system maintenance. Moreover, the retry mechanism of the existing data synchronization scheme is not perfect, lacking flexible retry strategies and number control, which easily causes data loss, data inconsistency, or message duplication, resulting in poor data synchronization accuracy and waste of system resources. In the scenario of concurrent updates, if multiple operations attempt to modify the same data, data may be overwritten or lost due to version conflicts.
[0044] Therefore, in the prior art, there are technical problems of data synchronization including message duplication, version order processing disorder, and imperfect retry mechanism, which leads to low data synchronization efficiency and insufficient accuracy, thereby affecting the reliability of the system.
[0045] To solve the above problems, the core idea of the present application is to define a standardized message body format, so that the consumer listens to the message body in the message queue and parses it to obtain message body information, and performs repeated information checking processing, expiration checking processing, and validity checking processing on the message body according to the message body information. If the message body passes the repeated information checking processing, expiration checking processing, and validity checking processing, the data synchronization operation is executed to ensure efficient, reliable, and orderly synchronization of data in the distributed system.
[0046] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.
[0047] Figure 1 Flowchart of a data synchronization method provided by the present application Figure 1 , applied to a message consumption end, as shown in Figure 1 , the method comprises:
[0048] S101, listen to the message body in the message queue; wherein the message body is generated by the message production end in response to the information change operation, and the message body information is sent to the message queue; wherein the message body information includes primary key information, content information, time information, version number information and retry number information.
[0049] In the embodiment, the primary key information refers to a unique identifier of the message body, used for idempotency check and positioning of the data to be updated, ensuring the uniqueness and traceability of the message body; the content information refers to business data or operation instructions of the change content, such as addition, update or deletion operation, and business data of the change content corresponding to the addition, update or deletion operation; the time information refers to a timestamp when the message body is generated, used for checking whether the message is expired and performing log recording; the version number information refers to an identifier recording the version of the data in the message body, used for identifying version conflict in concurrent update, ensuring the orderliness and correctness of data update; and the retry number information refers to the retry number of the message, used for executing the retry strategy and controlling the retry number.
[0050] S102, parsing the message body to obtain message body information of the message body.
[0051] In the embodiment, the message body is obtained from the message queue, and the message body is deserialized to extract the message body information of the message body.
[0052] S103, performing duplicate information check processing on the message body according to the message body information.
[0053] In the embodiment, according to the primary key information in the message body information, the primary key information of the historical message body information is queried, if the primary key information in the message body information is consistent with the primary key information of the historical message body information, it indicates that the message body is duplicated, and the message body does not pass the duplicate information check processing; if not, it indicates that the message body passes the duplicate information check processing.
[0054] Further, the duplicate information check processing on the message body according to the message body information further includes:
[0055] S1031, calculating an idempotency check value according to the primary key information, the content information and the version number information.
[0056] In the embodiment, the primary key information, the content information and the version number information are converted into a unified standardized format, the hash value of the content information is calculated, and then the primary key information, the hash value of the content information and the version number information are spliced according to a preset order, and the spliced primary key information, content information and version number information are calculated again to obtain the idempotency check value. Through twice hash value calculation of the content information, the privacy and security of the data are ensured.
[0057] S1032, checking whether the message body is a duplicate message body according to the idempotency check value and a historical idempotency check value of a historical message body.
[0058] In the embodiment, the message body is checked whether it is a repeated message body by comparing the idempotency check value of the current message body with the historical idempotency check value of the historical message body; if the idempotency check value of the current message body is same as the historical idempotency check value of the historical message body, it means the message body is repeated, and the message body fails the repeated information check processing; if not, it means the message body passes the repeated information check processing.
[0059] The idempotency check value is calculated by the primary key information, the content information and the version number information, and the repeated information check processing is performed on the message body according to the idempotency check value, which improves the accuracy of the repeated detection of the message body; meanwhile, the data state of the message body is ensured to be consistent when the same message body is processed multiple times.
[0060] Optionally, if the repeated information check fails, the message body is skipped.
[0061] In the embodiment, if the repeated information check fails, the message body is skipped, which avoids repeated sending of the message body and waste of system resources.
[0062] S104, if the repeated information check passes, the message body is subjected to an expired check processing according to the message body information.
[0063] In the embodiment, the time information in the message body information is used to check whether the message body is expired, and if the expired message body is detected, a cleaning operation is automatically performed to reduce the load of the message middleware.
[0064] Further, the expired check processing on the message body according to the message body information includes:
[0065] S1041, the time stamp and the expiration time are obtained according to the time information of the message body.
[0066] In the embodiment, the corresponding time stamp and expiration time are extracted from the time information of the message body; and a time window is set as the expiration time.
[0067] S1042, the message body is checked whether it is an expired message body according to the time stamp and the expiration time.
[0068] In the embodiment, if the current time is less than or equal to the expiration time, the message body is not an expired message body; if the current time is greater than the expiration time, the message body is an expired message body.
[0069] In the embodiment, the time stamp and the expiration time are obtained from the time information of the message body, and the message body is checked whether it is an expired message body, which makes the timeliness of the message clear and supports message-level custom expiration time, and improves the flexibility of the message timeliness.
[0070] Optionally, if the current time is about to reach the expiration time of the message body, corresponding alarm information is generated to reduce the expiration rate of the message body. Optionally, if the expiration information check fails, the message body is skipped.
[0071] In this embodiment, for the message body that fails the expiration information check, the current message body is skipped, and the expiration information check of the next message body is performed, so as to avoid affecting the processing progress of the message body.
[0072] Optionally, when the number of message bodies that fail the expiration information check reaches a preset accumulation number, or when the current time reaches a preset cleaning time limit, the system automatically cleans up the message bodies that fail the expiration information check in batches to release system resources.
[0073] S105, if the expiration check passes, the message body is subjected to effective check processing according to the message body information.
[0074] In this embodiment, the message body is subjected to effective check processing through the version number information in the message body information, so as to ensure the consistency of data in a distributed environment, so that the data is synchronously ordered, and the phenomenon that the old version data covers the new version data due to improper version management when multiple consumers concurrently update the same data is prevented.
[0075] Further, the message body is subjected to effective check processing according to the message body information, including:
[0076] S1051, current version number information of current data is obtained.
[0077] In this embodiment, the current version number information in the current data to be synchronized is obtained.
[0078] S1052, if the current version number message is lower than the version number information of the message body, it is determined that the effective check passes.
[0079] In this embodiment, if the current version number message is lower than the version number information of the message body, it indicates that the data in the message body is updated data and needs to be synchronized, the message body passes the effective check, and the current version number message is updated according to the version number information of the message body.
[0080] S1053, if the current version number message is not lower than the version number information of the message body, it is determined that the effective check fails.
[0081] In this embodiment, if the current version number message is not lower than the version number information of the message body, it indicates that the current data is the latest version data, and the data in the message body is old version data, so the message body fails the effective check; then the message body is skipped, and a processing log is recorded.
[0082] In the embodiment, the current version number information of the current data and the version number information of the message body are acquired, and whether the message body passes the effective verification is determined by comparing the current version number information and the version number information of the message body. The old version data is avoided from covering the new version data to cause data rollback, and only the message of version update is processed to reduce the operation of invalid messages, improve the efficiency of message synchronization, record the processing log, and realize the traceability of version conflict.
[0083] S106, if the effective verification passes, performing a data synchronization operation according to the message body.
[0084] In the embodiment, the data synchronization operation of invalid message bodies is effectively reduced by sequentially performing the repeated information verification processing, the expiration verification processing and the effective verification processing on the message body, the version conflict is avoided, and the message body synchronization efficiency and reliability are ensured.
[0085] Further, performing a data synchronization operation according to the message body, including:
[0086] S1061, performing a data synchronization operation according to the content information in the message body.
[0087] In the embodiment, the content information in the message body is transactionally written into the database or the data source to be synchronized; if the data synchronization operation is successful, a data synchronization success message is returned.
[0088] S1062, if the data synchronization operation fails, performing a data synchronization retry operation according to the retry number information in the message body.
[0089] In the embodiment, if the data synchronization operation fails and the retry number information in the message body is less than the preset maximum retry number or the current retry interval is less than the preset maximum retry interval, a corresponding data synchronization retry operation is performed according to the failure type, and the retry number information in the message body and the record data synchronization operation failure reason are updated.
[0090] Optionally, an exponential backoff strategy is adopted to gradually increase the retry interval according to the retry number to reduce the system load. If the retry number information in the message body reaches the maximum retry number or the retry interval of the message body reaches the maximum retry interval, the retry operation is stopped, an error log is recorded, an alarm message of the data synchronization operation failure is generated, and a corresponding operation and maintenance work order is generated.
[0091] For example, if the data synchronization operation fails due to network jitter, the data synchronization retry operation is immediately performed; if the data synchronization operation fails due to system overload, a delay time is set in the attribute file, and the data synchronization retry operation is delayed according to the delay time.
[0092] In the embodiment, the success rate of data synchronization is improved by introducing an intelligent retry strategy,
[0093] The data synchronization method provided in the application can effectively solve the problems of message repetition, disorder in version sequence processing and imperfect retry mechanism in the prior art, improve the efficiency and accuracy of data synchronization, and ensure efficient, reliable and orderly synchronization of data in a distributed system.
[0094] Figure 2 A flowchart of a data synchronization method provided in the application Figure 2 , applied to a message production end, as shown in Figure 3 , the method comprises the following steps.
[0095] S201, in response to an information change operation, message body information is acquired, wherein the message body information comprises primary key information, content information, time information, version number information and retry count information.
[0096] In the embodiment, the message production end listens to change events of a data source, and in response to a message change operation, corresponding change content is acquired as message body information.
[0097] S202, a message body is generated according to the message body information.
[0098] In the embodiment, the message body information is assembled into a message body according to a predefined message body format.
[0099] For example, the format of the message body can be: "primary key: content: time: version number: retry count".
[0100] S203, the message body is sent to a message queue, so that a message consumer listens to the message body in the message queue, parses the message body to obtain message body information of the message body, performs repeated information check processing on the message body according to the message body information, if the repeated information check passes, performs expired check processing on the message body according to the message body information, if the expired check passes, performs valid check processing on the message body according to the message body information, and if the valid check passes, performs a data synchronization operation according to the message body.
[0101] In this embodiment, a standardized message body format ensures complete data transmission from the message producer to the message consumer, preventing data loss. Furthermore, separating content information from version number and retry count information decouples business logic from technical control, facilitating data content expansion and adapting to multiple message consumers for on-demand message body parsing.
[0102] Figure 3 A flowchart illustrating a data synchronization method provided in this application. Figure 4 ,like Figure 4 As shown: If business data changes, the message producer generates a message body in the format "primary key: content: time: version number: retry count" based on the changed business data and sends it to the message queue of the message middleware. The message consumer listens to the message queue. When it detects a message body in the message queue, it pulls the message body and parses the data in the message body. It uses the primary key, content, and version number to check for duplicate information and determine whether it is a duplicate message. If it is a duplicate, the message body is skipped. If it is not a duplicate, the message body is checked for expiration. If the message body has expired, it is skipped. If the message body has not expired, it is checked for validity. If the local data version number is higher than or equal to the version number in the message body, it indicates that the validity check has not passed, so the message body is skipped. If the local data version number is lower than the version number in the message body, it indicates that the validity check has passed, so the message is synchronized to the local database and the version number is updated. It checks whether the data synchronization is successful. If the data synchronization fails, it updates the retry count in the message body and sends a retry message to the message middleware so that the message middleware resends the message body, realizing the data synchronization retry operation.
[0103] For example, if the message producer is a bank system and the message consumer is a credit analysis system, when the bank system detects a change in user information, the bank system generates a message body including a user identifier, user information, a timestamp, a version number, and a retry count, and sends it to the message middleware; the credit analysis system subscribes to the message body of the message middleware and parses the message body to obtain the corresponding user identifier, user information, timestamp, version number, and retry count; an idempotency check is performed on the user identifier, user information, and version number to determine whether there is a duplicate message in the local database of the credit analysis system; if there is no duplicate message, it is determined whether the message body is expired according to the timestamp; if the message body is expired, the version number of the message body is compared with the version number of the corresponding message in the local database; if the version number of the message body is not higher than the version number of the corresponding message in the local database, the message body is ignored; if the version number of the message body is higher than the version number of the corresponding message in the local database, a message synchronization operation is performed, and if the message synchronization operation fails due to a connection failure of the local database, the retry count in the message body is increased, and a retry request is sent to the message middleware for automatic retry operation until the message synchronization operation is successful or the maximum number of retries is reached, and the corresponding processing log is recorded.
[0104] Figure 5 A structural schematic diagram of a data synchronization device provided in the present application is shown in Figure 5 The data synchronization device includes:
[0105] The listening module 401 is configured to listen to the message body in the message queue; wherein the message body is generated by the message producer in response to the information change operation, and the message body information is generated according to the message body information and then sent to the message queue; wherein the message body information includes primary key information, content information, time information, version number information, and retry count information.
[0106] The analysis module 402 is configured to analyze the message body to obtain the message body information of the message body.
[0107] The first checking module 403 is configured to perform duplicate information checking processing on the message body according to the message body information.
[0108] The second checking module 404 is configured to perform expiration checking processing on the message body according to the message body information if the duplicate information checking passes.
[0109] The third checking module 405 is configured to perform validity checking processing on the message body according to the message body information if the expiration checking passes.
[0110] The execution module 406 is configured to perform data synchronization operation according to the message body if the validity checking passes.
[0111] In a possible implementation, the first checking module 403 can be further configured to:
[0112] According to the primary key information, the content information and the version number information, an idempotency check value is calculated.
[0113] According to the idempotency check value and a historical idempotency check value of the historical message body, it is checked whether the message body is a repeated message body.
[0114] Optionally, if the repetition information check fails, the message body is skipped.
[0115] In a possible implementation, the second checking module 404 can be further configured to:
[0116] According to the time information of the message body, a timestamp and an expiration time are obtained.
[0117] According to the timestamp and the expiration time, it is checked whether the message body is an expired message body.
[0118] Optionally, if the expiration information check fails, the message body is skipped.
[0119] In a possible implementation, the third checking module 405 can be further configured to:
[0120] Obtaining current version number information of current data;
[0121] If the current version number information is lower than the version number information of the message body, it is determined that the validity check passes;
[0122] If the current version number information is not lower than the version number information of the message body, it is determined that the validity check fails.
[0123] In a possible implementation, the execution module 406 can be further configured to:
[0124] According to the content information in the message body, a data synchronization operation is performed;
[0125] If the data synchronization operation fails, a data synchronization retry operation is performed according to the retry times information in the message body.
[0126] In a possible implementation, the data synchronization apparatus further includes:
[0127] The obtaining module is configured to obtain message body information in response to an information change operation; wherein the message body information includes primary key information, content information, time information, version number information and retry times information.
[0128] The generating module is configured to generate a message body according to the message body information.
[0129] The sending module is configured to send the message body to a message queue, so that a message consumer listens to the message body in the message queue, parses the message body to obtain message body information of the message body, performs repeated information checking processing on the message body according to the message body information, if the repeated information checking passes, performs expired checking processing on the message body according to the message body information, if the expired checking passes, performs valid checking processing on the message body according to the message body information, and if the valid checking passes, executes a data synchronization operation according to the message body.
[0130] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0131] It should be further noted that, although each step in the flowchart is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise stated in this article, the execution of these steps has no strict sequence limitation, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps.
[0132] It should be understood that the above-mentioned device embodiments are only schematic, and the device of the present application can also be realized by other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and another division way can be used in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0133] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application can be integrated in one unit / module, or each unit / module can exist physically, or two or more units / modules can be integrated together. The integrated unit / module can be realized in the form of hardware or in the form of software program module.
[0134] A structural schematic diagram of an electronic device provided by the embodiment of the present application is shown in As shown, the electronic device includes at least one processor 501 and a memory 502. Optionally, the electronic device further includes a communication component 503. Among them, the processor 501, the memory 502 and the communication component 503 are connected through the bus 504.
[0135] In the specific implementation process, the at least one processor 501 executes the computer execution instructions stored in the memory 502, so that the at least one processor 501 executes the above-mentioned method.
[0136] The specific implementation process of the processor 501 can refer to the above-mentioned method embodiment, which has similar implementation principles and technical effects, and will not be described here in detail.
[0137] In the above-mentioned embodiment, it should be understood that the processor can be a central processing unit (English: Central Processing Unit, for short: CPU), and can also be other general-purpose processors, digital signal processors (English: Digital Signal Processor, for short: DSP), application specific integrated circuits (English: Application Specific Integrated Circuit, for short: ASIC) and the like. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly embodied as hardware processor execution, or executed by hardware and software modules in the processor.
[0138] The memory can contain a random access memory (Random Access Memory, RAM), and can also include a non-volatile memory (Non-volatile Memory, NVM), for example, at least one disk memory.
[0139] The bus can be an industry standard architecture (Industry Standard Architecture, ISA) bus, a peripheral component interconnect (Peripheral Component, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit only one bus or one type of bus.
[0140] The present application also provides a computer program product, comprising a computer program, which is executed by the processor to realize the above-mentioned method.
[0141] The application further provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions.
[0142] The readable storage medium can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0143] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in a special integrated circuit. Of course, the processor and the readable storage medium can also exist as discrete components in the device.
[0144] The division of units is only a logical function division, and in actual implementation, there can be another division mode, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0145] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.
[0146] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.
[0147] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.
[0148] It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.
[0149] Finally, it should be noted that: those skilled in the art will easily conceive other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or conventional technical means in the art that are not disclosed in the present application, and is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is only limited by the appended claims.
Claims
1. A data synchronization method, characterized in that, Applied to the message consumption end, the method includes: Listen to the message body in the message queue; wherein, the message body is generated by the message producer in response to an information change operation, obtains message body information, generates a message body based on the message body information, and sends it to the message queue; wherein, the message body information includes primary key information, content information, time information, version number information, and retry count information; Parse the message body to obtain the message body information of the message body; Based on the message body information, perform duplicate information verification processing on the message body; If the duplicate information verification passes, then the message body is subjected to expiration verification based on the message body information. If the expiration check passes, then the message body is subjected to valid verification processing based on the message body information; If the valid verification passes, then data synchronization is performed according to the message body.
2. The method according to claim 1, characterized in that, The step of performing duplicate information verification on the message body based on the message body information includes: Calculate the idempotency check value based on the primary key information, the content information, and the version number information; Based on the idempotency check value and the historical idempotency check value of the historical message body, verify whether the message body is a duplicate message body.
3. The method according to claim 2, characterized in that, After performing duplicate information verification on the message body based on the message body information, the method further includes: If duplicate information verification fails, the message body is skipped.
4. The method according to claim 1, characterized in that, The step of performing expiration verification processing on the message body based on the message body information includes: Based on the time information in the message body, obtain the timestamp and expiration time; Based on the timestamp and the expiration time, verify whether the message body is an expired message body.
5. The method according to claim 4, characterized in that, After performing expiration verification on the message body based on the message body information if the duplicate information verification passes, the method further includes: If the expiration information verification fails, the message body is skipped.
6. The method according to claim 1, characterized in that, The step of performing effective verification processing on the message body based on the message body information includes: Get the current version number information of the current data; If the current version number message is lower than the version number information in the message body, then the valid verification is determined to have passed; If the current version number message is not lower than the version number information in the message body, then the valid verification is determined to have failed.
7. The method according to any one of claims 1 to 6, characterized in that, The step of performing data synchronization operation according to the message body includes: Based on the content information in the message body, perform a data synchronization operation; If the data synchronization operation fails, a data synchronization retry operation will be performed based on the retry count information in the message body.
8. A data synchronization method, characterized in that, Applied to the message production end, the method includes: In response to an information change operation, message body information is obtained; wherein, the message body information includes primary key information, content information, time information, version number information, and retry count information; Generate a message body based on the message body information; The message body is sent to the message queue so that the message consumer listens to the message body in the message queue, parses the message body to obtain the message body information, performs duplicate information verification on the message body according to the message body information, performs expiration verification on the message body according to the message body information, performs validity verification on the message body according to the message body information, and performs data synchronization operation according to the message body.
9. A data synchronization device, characterized in that, include: The listening module is used to listen to the message body in the message queue; wherein, the message body is generated by the message producer in response to an information change operation, obtains message body information, generates a message body based on the message body information, and sends it to the message queue; wherein, the message body information includes primary key information, content information, time information, version number information, and retry count information; A parsing module is used to parse the message body to obtain the message body information of the message body; The first verification module is used to perform duplicate information verification processing on the message body based on the message body information; The second verification module is used to perform expiration verification on the message body according to the message body information if the duplicate information verification passes. The third verification module is used to perform valid verification processing on the message body based on the message body information if the expiration verification passes. The execution module is used to perform data synchronization operations based on the message body if the valid verification passes.
10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement any one of claims 1 to 7 or to implement the method as described in claim 8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement any one of claims 1 to 7 or to implement the method as described in claim 8.
12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements any one of claims 1 to 7 or implements the method as described in claim 8.