Data synchronization method and device
By storing data update messages into the message queue and synchronous processing using timing tasks, the complexity and message backlog of data in distributed systems are solved, and data synchronization stability and consistency under large traffic are achieved.
Patent Information
- Application Number
- CN202310870128.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-14
- Publication Date
- 2025-01-14
AI Technical Summary
In the prior art, the real-time interface call method leads to strong dependence and high complexity between distributed application systems. The asynchronous monitoring method will lead to a backlog of messages and slow service response in large data scenarios, affecting the user experience.
By depositing the listened data update message into the first message queue, the message is stored in the second message queue using a timing task, and obtaining system data from the item system according to the item identification to update local data, achieving smooth and consistent data synchronization .
In the scenarios of large traffic and large data volume, the stable and orderly consumption of data update messages is achieved, the stability and consistency of data synchronization processing is ensured, and the problems of message backlog and slow response are avoided.
Smart Images
Figure CN119311766A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data technology, and in particular to a method and apparatus for data synchronization. Background Art
[0002] In distributed system architectures, business activities typically require the collaboration of multiple application systems, inevitably leading to data synchronization issues during data interaction among these systems. Currently, there are two main solutions for data synchronization: one is through real-time API calls, and the other is through asynchronous message listening.
[0003] In the process of realizing this invention, the inventors discovered the following problems in the prior art:
[0004] Real-time API calls create strong dependencies between distributed application systems and require data integrity assurance, increasing system complexity and impacting throughput. Asynchronous listening, in scenarios with large data volumes, can cause message backlogs even when the business system is running at full capacity. Furthermore, prolonged full-capacity operation can affect other business processes, leading to slow overall system response and negatively impacting user experience. Summary of the Invention
[0005] In view of this, embodiments of the present invention provide a method and apparatus for data synchronization. The method involves storing monitored data update messages in a first message queue, using a scheduled task to store the first message from the first message queue into a second message queue, and then retrieving corresponding system data from the item system based on the item identifier obtained from the second message queue to update local data. This embodiment of the present invention utilizes a first message queue to cache data update messages, and then uses a scheduled retrieval of data update messages to construct a new message queue for data synchronization. This achieves smooth and orderly consumption of data update messages in scenarios with large data volumes, ensuring the stability and consistency of data synchronization processing under high traffic.
[0006] To achieve the aforementioned objective, according to one aspect of the present invention, a data synchronization method is provided, comprising:
[0007] In response to the detection of a data update message from the item system, the data update message is stored in the first message queue;
[0008] A specified number of first messages are periodically pulled from the first message queue by a scheduled task, and the pulled first messages are stored in the second message queue.
[0009] The item identifier of the item is obtained from the message in the second message queue. Based on the item identifier, the system data of the item is obtained from the item system. The local data of the item is updated according to the system data.
[0010] Optionally, the method further includes: in response to receiving a dispatch task from the item system, initializing the data information of the items in the dispatch task to local data, and recording the item identifier of the items to obtain an item identifier set.
[0011] Optionally, the data update message has an item identifier; storing the data update message in the first message queue includes: filtering the data update message using the item identifier set, selecting data update messages that match the item identifiers in the item identifier set, and storing the selected data update messages in the first message queue.
[0012] Optionally, obtaining system data of the item from the item system based on the item identifier, and updating the local data of the item according to the system data, includes: querying system data of the item identifier from the item system, wherein the system data and local data have a corresponding relationship; obtaining local data of the item identifier; comparing the system data with the local data; and if the local data does not conform to the corresponding relationship, synchronously updating the local data corresponding to the system data to the local data of the item.
[0013] Optionally, the method further includes: in the event that the data update fails, resynchronizing and updating the failed data through a preset retry mechanism.
[0014] Optionally, the system data of the item is the item's shelf status or the item's de-shelf status, and the local data of the item is the item's shelf status or the item's de-shelf status. The item's shelf status corresponds to the item's shelf status, and the item's de-shelf status corresponds to the item's de-shelf status.
[0015] Optionally, updating the local data of the item according to the system data includes: when the system data indicates that the item is in a "listed" state, determining whether the local data corresponding to the item is in a "listed" or "unlisted" state according to a preset item local data setting rule.
[0016] According to a second aspect of the present invention, a data synchronization apparatus is provided, comprising:
[0017] The first message queue module is used to store the data update message into the first message queue in response to the data update message of the item system being listened to.
[0018] The second message queue module is used to periodically pull a specified number of first messages from the first message queue through a scheduled task and store the pulled first messages into the second message queue.
[0019] The data synchronization module is used to obtain the item identifier of the item from the messages in the second message queue, obtain the system data of the item from the item system according to the item identifier, and update the local data of the item according to the system data.
[0020] According to a third aspect of the present invention, an electronic device for data synchronization is provided, comprising:
[0021] One or more processors;
[0022] a storage device for storing one or more programs,
[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiments of the present invention.
[0024] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method provided in the first aspect of the present invention.
[0025] One embodiment of the invention has the following advantages or beneficial effects: by responding to the detected data update message of the item system, the data update message is stored in a first message queue; a specified number of first messages are periodically pulled from the first message queue by a timed task, and the pulled first messages are stored in a second message queue; the item identifier of the item is obtained from the messages in the second message queue, the system data of the item is obtained from the item system according to the item identifier, and the local data of the item is updated according to the system data. This technical solution realizes the smooth and orderly consumption of data update messages in scenarios with large data volume, ensuring the stability and consistency of data synchronization processing under high traffic. Attached Figure Description
[0026] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:
[0027] Figure 1 This is a schematic diagram of the main flow of the data synchronization method according to an embodiment of the present invention;
[0028] Figure 2 This is a schematic diagram illustrating the process of initializing the data information of an item according to an embodiment of the present invention;
[0029] Figure 3 This is a detailed flowchart illustrating the data synchronization process according to an embodiment of the present invention;
[0030] Figure 4 This is a schematic diagram of the main modules of a data synchronization device according to an embodiment of the present invention;
[0031] Figure 5 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;
[0032] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. DETAILED DESCRIPTION
[0033] It should be noted that the collection, updating, analysis, use, transmission, and storage of user personal information involved in the technical solution of this invention all comply with relevant laws and regulations, are used for legitimate and reasonable purposes, and are not shared, disclosed, or sold outside of these legitimate uses, and are subject to supervision and management by national regulatory authorities. Necessary measures should be taken to selectively block the use or access to personal information data to prevent unauthorized access to such personal information data, ensure that personnel authorized to access personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Furthermore, once this user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data.
[0034] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0035] Currently, there are two main solutions for data synchronization. The real-time interface call solution creates strong dependencies between various distributed application systems and requires ensuring data integrity, increasing system complexity and impacting throughput. The asynchronous listening solution, in scenarios with large data volumes, can cause message backlog even when the business system is running at full load. Furthermore, prolonged full-load operation can affect other business processes, leading to slow response times for the entire business system, impacting user experience, and failing to adequately meet the needs of practical applications.
[0036] To address the aforementioned problems in existing technologies, this invention proposes a data synchronization method. The method involves storing monitored data update messages in a first message queue, then using a scheduled task to store the first message from the first message queue into a second message queue. Finally, based on the item identifier obtained from the second message queue, corresponding system data is retrieved from the item system to update the local data. This embodiment of the invention utilizes a first message queue to cache data update messages, and then uses a scheduled retrieval of data update messages to construct a new message queue for data synchronization. This achieves smooth and orderly consumption of data update messages even in scenarios with large data volumes, ensuring the stability and consistency of data synchronization processing under high traffic.
[0037] In the description of the embodiments of the present invention, the terms involved and their meanings are as follows:
[0038] Middle platform: an internet term, generally refers to building a flexible and rapid architecture that can quickly respond to changes, rapidly implement front-end requirements, avoid redundant construction, and improve work efficiency;
[0039] A topic can be understood as a subject or theme. Message producers can send messages to a specific topic, and message consumers can subscribe to one or more topics as needed to receive messages sent by the producer.
[0040] Figure 1 This is a schematic diagram of the main flow of the data synchronization method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the data synchronization method of this embodiment of the invention includes the following steps S101 to S103.
[0041] Step S101: In response to the detection of a data update message from the item system, the data update message is stored in the first message queue.
[0042] Specifically, in a distributed system architecture for comprehensive goods management (such as the management of goods on e-commerce platforms and in-store merchandise), the goods system acts as a middle platform, providing various basic goods service capabilities, such as goods query and goods maintenance. Data update messages from the goods system drive data synchronization processing in various downstream business systems. Faced with hundreds of millions of data update messages, consumers can either add servers to improve the business system's ability to consume these messages, or reduce business processing logic to gain more resources for consuming data update messages. However, these approaches either increase costs or require sacrificing some business processing logic, neither of which adequately meets practical needs.
[0043] In this embodiment of the invention, when a data update message from the item system is detected, the detected data update message is stored in a local Redis cache in the form of a message queue, thus obtaining the first message queue. Typically, the Redis cache capacity can meet the needs of high-volume scenarios with data volumes reaching hundreds of millions. However, to further ensure the reliability of Redis, this embodiment of the invention employs a scaling mechanism. When the used capacity of Redis reaches a preset threshold, Redis is scaled up to ensure that the business system can quickly respond to sudden surges in data volume, avoiding system failures due to insufficient cache capacity.
[0044] According to an embodiment of the present invention, the method further includes: in response to receiving a dispatch task from the item system, initializing the data information of the items in the dispatch task to local data, and recording the item identifiers of the items to obtain an item identifier set.
[0045] Specifically, when a downstream business system receives an approved allocation task from the item system, it executes the item allocation process, stores the item data information such as item number and quantity in the allocation task into a local database, and initializes the item in the business system. After the item data information is successfully stored in the database, the corresponding item identifier is stored in a pre-allocated cache to record the set of item identifiers corresponding to the downstream business system, which is the specific business scope of the downstream business system.
[0046] Figure 2 This is a flowchart illustrating the initialization process of item data information in an embodiment of the present invention. The item system issues allocation tasks. For approved allocation tasks, the downstream business system first performs idempotency verification to prevent item allocation tasks from being treated as multiple allocation tasks and processed repeatedly. The item data information in allocation tasks that pass idempotency verification is stored in a local database. If storage is successful, the item identifier corresponding to the successfully stored item data information is recorded in a cache to obtain an item identifier set. If storage fails, a retry mechanism is initiated to retry storing the item data information in the database. Furthermore, allocation tasks that fail idempotency verification do not require further processing and can be discarded directly.
[0047] According to another embodiment of the present invention, the data update message has an item identifier; storing the data update message into a first message includes: filtering the data update message using the item identifier set, selecting data update messages that match the item identifiers in the item identifier set, and storing the selected data update messages into a first message queue.
[0048] Specifically, in a distributed business scenario of integrated item management, the item system manages item data information from various downstream business systems. Therefore, the item system's data update messages include data update messages for all items in the downstream business data. To conserve resources, in this embodiment of the invention, when a downstream business system listens for a data update message, it uses the item identifier in the data update message and performs business filtering using the aforementioned set of item identifiers recorded in the cache. The data update messages corresponding to items matching the item identifiers in the set are then used as the data update messages for that business system, and the selected data update messages are stored in the first message queue. This not only effectively avoids irrelevant data update messages occupying the business system's cache resources but also significantly reduces the workload of subsequent business systems in processing data update messages.
[0049] Step S102: A specified number of first messages are periodically retrieved from the first message queue by a scheduled task, and the retrieved first messages are stored in the second message queue.
[0050] Specifically, the first message queue stores all data update messages monitored in the item system. In order to buffer the impact of excessive message volume on the overall performance of the business system, this embodiment of the invention uses a scheduled task to retrieve a specified number of first messages from the first message queue in batches, such as pulling 100 data update messages at a time. The retrieved first messages are then stored in the constructed second message queue as new topic messages. This is equivalent to the business system producing the second message queue itself based on the data update messages.
[0051] Step S103: Obtain the item identifier of the item from the message in the second message queue, obtain the system data of the item from the item system according to the item identifier, and update the local data of the item according to the system data.
[0052] Specifically, based on the second message queue mentioned above, messages in the second message queue are consumed by listening to the messages in the second message queue. The consumed messages are parsed to obtain the item identifier. The business system retrieves the corresponding system data from the item system based on the obtained item identifier, and then updates the local data of the corresponding item based on the system data of the item system.
[0053] According to one embodiment of the present invention, the system data of the item is the item on the shelf or the item off the shelf, and the local data of the item is the item on the shelf or the item off the shelf. The item on the shelf corresponds to the item on the shelf, and the item off the shelf corresponds to the item off the shelf.
[0054] According to another embodiment of the present invention, obtaining system data of the item from the item system based on the item identifier and updating the local data of the item according to the system data includes: querying system data of the item identifier from the item system, wherein the system data and local data have a corresponding relationship; obtaining local data of the item identifier; comparing the system data with the local data; and if the local data does not conform to the corresponding relationship, synchronously updating the local data corresponding to the system data to the local data of the item.
[0055] Specifically, in a distributed business scenario for item transaction management, the item system's status updates mainly include item listing and item removal. An item being listed indicates it is in stock and ready for sale; conversely, an item being removed indicates it is out of stock and cannot be sold. In the downstream business system, item status updates mainly include item availability and item removal. Listed items are allowed to be sold, while removed items are not. Generally, when the item system updates its status to "listed," the corresponding item status in the downstream business system should also be "available," and vice versa.
[0056] The item system needs to synchronize item status update data to downstream business systems so that these systems can synchronize based on the item status data in the item system. The business system obtains the item identifier by parsing the message, queries the item system for the latest system data corresponding to that identifier, and then retrieves the local data for that identifier within the business system. The system data and local data are compared. If they don't match, the local data in the current business system is updated to reflect the system data. For example, if the system data shows "off shelf," but the local data in the current business system shows "on shelf," this is clearly inconsistent with the system data's corresponding local data. In this case, the "off shelf" status is updated in the local data to prevent downstream business systems from selling items that are not actually in stock. It should be noted that the local data in the business system generally does not correspond to the system data. However, in cases of message out-of-order listening or consumption, the local data in the business system may be identical to the system data. In such cases, updating the local data in the business system is unnecessary. In addition, the data synchronization logic of retrieving the latest system data from the item system through the item identifier and then comparing it with the local status can synchronize local data based on the latest system data, effectively solving the problem of data synchronization errors caused by out-of-order data update messages.
[0057] According to another embodiment of the present invention, updating the local data of the item according to the system data includes: when the system data shows the item as being on the shelf, determining the local data corresponding to the item as being on the shelf or off the shelf according to a preset item local data setting rule.
[0058] Specifically, considering the individual differences in item transaction management, when the system data shows an "listed" status, the downstream business system needs to adaptively update the item's local data to either an "listed" or "delisted" status based on the item's local data setting rules. In this embodiment of the invention, the item's local data setting rules can be formulated based on the business system's sales permissions. Specifically, these rules may include item identifiers, expiration dates, and whether sales permissions exist. For items with a "listed" status in the system data, the system checks whether sales permissions exist based on the item identifier and the current update time, according to the item's local data setting rules. If there are no sales permissions, the local data corresponding to the "listed" status should be in a "delisted" status; if there are sales permissions, the local data corresponding to the "listed" status should be in a "delisted" status. Of course, the item's local data setting rules can also be determined based on transaction strategies. For example, for items A, B, and C with similar functions, if the current local data for B and C are both in a "listed" sales status, then even if A's system data is updated to a "listed" status, item A will no longer be sold. In this case, the local data corresponding to item A's "listed" status should be in a "delisted" status, meaning transactions are not supported.
[0059] By setting rules based on local data of items and then verifying the corresponding local data for the system data, the status data of items in the business system can be managed more granularly to meet the needs of actual scenarios.
[0060] According to another embodiment of the present invention, the method further includes: in the event that the data update fails, resynchronizing and updating the failed data again through a preset retry mechanism.
[0061] Specifically, this embodiment of the invention not only designs a retry mechanism during data initialization but also considers a retry mechanism for data update failures. By setting the maximum number of retries and the time interval between each retry through the message queue retry mechanism, if an exception occurs when updating the local data corresponding to the system data to the local data of the item, causing the update to fail, multiple attempts can be made to update the local data corresponding to the system data to the local data of the item according to the set retry time interval and maximum number of retries, until the local data corresponding to the system data is successfully updated to the local data of the item within the maximum number of retries, thus ensuring the success rate of data updates.
[0062] Understandably, retry mechanisms can also be used for key processes in local data updates, such as consuming and pulling the first message from the first message queue, or obtaining system data based on item identifiers. The maximum number of retries and the time interval between each retry can be set according to different scenarios to ensure that no data is lost during the data update process and improve the reliability of data synchronization.
[0063] Figure 3 This is a detailed flowchart illustrating the data synchronization process of an embodiment of the present invention. Upon detecting a data update message from the item system, the system filters these messages using an item identifier set. Messages matching the item identifiers in the item identifier set are stored in a first message queue. A scheduled task periodically retrieves a specified number of first messages from the first message queue and sends them to a second message queue for storage. Messages in the second message queue are consumed, and based on the item identifiers obtained from parsing the messages, the system data for the item is retrieved from the item system. The local data for the item in the business system is compared with the corresponding local data in the system. If they differ, the local data for the item in the business system is updated according to the system data; otherwise, the data update process ends.
[0064] This invention provides a data synchronization method that enhances message consumption capabilities without scaling costs. It consumes data update messages in large-volume scenarios using a first message queue, and then uses a scheduled task to pull data update messages from the first message queue in batches and store them in a constructed second message queue for data synchronization processing. By separating the consumption of data update messages from their processing, the impact of excessive message volume on the overall performance of the business system is buffered.
[0065] This invention implements a combination of mechanisms, including constructing a first message queue, periodically pulling messages from the first message queue, constructing a second message queue, and comparing data from the system based on item identifiers. This enables the smooth and orderly consumption of data update messages, ensuring the stability and consistency of data synchronization processing under high traffic.
[0066] Figure 4 This is a schematic diagram of the main modules of a data synchronization device according to an embodiment of the present invention. Figure 4 As shown, the data synchronization device 400 mainly includes a first message queue module 401, a second message queue module 402, and a data synchronization module 403.
[0067] The first message queue module 401 is used to store the data update message into the first message queue in response to the detection of a data update message from the item system.
[0068] The second message queue module 402 is used to periodically pull a specified number of first messages from the first message queue through a timed task and store the pulled first messages into the second message queue.
[0069] The data synchronization module 403 is used to obtain the item identifier of the item from the message in the second message queue, obtain the system data of the item from the item system according to the item identifier, and update the local data of the item according to the system data.
[0070] According to an embodiment of the present invention, the data synchronization device 400 further includes an initialization module (not shown in the figure), configured to: in response to receiving a dispatch task from the item system, initialize the data information of the items in the dispatch task to local data, and record the item identifiers of the items to obtain an item identifier set.
[0071] According to another embodiment of the present invention, the data update message has an item identifier; the first message queue module 401 is further configured to: filter the data update message using the item identifier set, select data update messages that match the item identifiers in the item identifier set, and store the selected data update messages into the first message queue.
[0072] According to another embodiment of the present invention, the data synchronization module 403 can also be used to: query the system data of the item identifier from the item system, wherein the system data and local data have a corresponding relationship; obtain the local data of the item identifier, compare the system data with the local data, and if the local data does not conform to the corresponding relationship, synchronize and update the local data corresponding to the system data to the local data of the item.
[0073] According to another embodiment of the present invention, the data synchronization device 400 further includes a retry module (not shown in the figure), which is used to: in the event that the data update fails, re-synchronize and update the failed data through a preset retry mechanism.
[0074] According to another embodiment of the present invention, the system data of the item is the item on shelf status or the item off shelf status, and the local data of the item is the item on the shelf status or the item off the shelf status, wherein the item on shelf status corresponds to the item on the shelf status, and the item off shelf status corresponds to the item off the shelf status.
[0075] According to another embodiment of the present invention, the data synchronization module 403 can also be used to: determine whether the local data corresponding to the item is in the "listed" or "unlisted" state according to a preset item local data setting rule when the system data is in the "item listed" state.
[0076] Figure 5 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied.
[0077] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0078] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications, such as data synchronization applications, can be installed on terminal devices 501, 502, and 503 (for example only).
[0079] Terminal devices 501, 502, and 503 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0080] Server 505 can be a server providing various services, such as a background management server supporting data synchronization performed by users using terminal devices 501, 502, and 503 (for example only). The background management server can, in response to receiving a data update message from the item system, store the data update message in a first message queue; periodically retrieve a specified number of first messages from the first message queue via a scheduled task, and store the retrieved first messages in a second message queue; obtain the item identifier from the messages in the second message queue, retrieve the item's system data from the item system based on the item identifier, update the item's local data according to the system data, and process the data, and then feed back the processing result (e.g., local data – for example only) to the terminal device.
[0081] It should be noted that the data synchronization method provided in this embodiment of the invention is generally executed by server 505, and correspondingly, the data synchronization device is generally set in server 505.
[0082] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.
[0083] Reference below Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing terminal devices or servers of the present invention. Figure 6The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.
[0084] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0085] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.
[0086] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.
[0087] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0089] The units described in the embodiments of the present invention can be implemented in software or in hardware. The described units can also be housed in a processor; for example, a processor can be described as including: a first message queue module, a second message queue module, and a data synchronization module.
[0090] In some cases, the names of these modules do not constitute a limitation on the module itself. For example, the first message queue module can also be described as "a module for storing data update messages into the first message queue in response to the detection of data update messages of the item system".
[0091] On the other hand, the present invention also provides a computer-readable medium, which may be included in the device described in the embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: in response to listening for a data update message from an item system, storing the data update message in a first message queue; periodically pulling a specified number of first messages from the first message queue via a timed task, and storing the pulled first messages in a second message queue; obtaining an item identifier from messages in the second message queue, retrieving system data of the item from the item system based on the item identifier, and updating the local data of the item according to the system data.
[0092] The technical solution of the present invention has the following advantages or beneficial effects: by responding to the data update message of the item system being listened to, the data update message is stored in a first message queue; a specified number of first messages are periodically pulled from the first message queue by a timed task, and the pulled first messages are stored in a second message queue; the item identifier of the item is obtained from the messages in the second message queue, the system data of the item is obtained from the item system according to the item identifier, and the local data of the item is updated according to the system data, the technical solution realizes that in scenarios with a large amount of data, the data update message is consumed smoothly and orderly, ensuring the stability and consistency of data synchronization processing under high traffic.
[0093] The specific embodiments described herein do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for data synchronization, characterized in that: include: In response to monitoring a data update message of the item system, storing the data update message into a first message queue; Pulling a specified number of first messages from the first message queue regularly through a scheduled task, and storing the pulled first messages in a second message queue; An item identifier of the item is obtained from the message in the second message queue, system data of the item is acquired from the item system according to the item identifier, and local data of the item is updated according to the system data.
2. The method according to claim 1, characterized in that The method further comprises: In response to receiving the allocation task from the item system, the data information of the items in the allocation task is initialized to local data, and the item identification of the item is recorded to obtain an item identification set.
3. The method according to claim 2, characterized in that The data update message has an item identifier; storing the data update message in a first message queue comprises: The data update messages are filtered using the item identification set to filter out data update messages that match the item identifications in the item identification set, and the filtered data update messages are stored in a first message queue.
4. The method according to claim 1, characterized in that: Acquiring system data of the item from the item system according to the item identifier, and updating local data of the item according to the system data, including: Querying the system data of the item identification from the item system, wherein the system data and the local data have a corresponding relationship; The local data of the item identification is obtained, and the system data is compared with the local data. If the local data does not conform to the corresponding relationship, the local data corresponding to the system data is synchronously updated to the local data of the item.
5. The method according to claim 1, characterized in that: The method further comprises: In the case where the data update fails, the data that failed to be updated is synchronized and updated again through a preset retry mechanism.
6. The method according to claim 1, characterized in that The system data of the item is the item on-shelf status or the item off-shelf status, and the local data of the item is the item on-shelf status or the item off-shelf status. The item on-shelf status corresponds to the item on-shelf status, and the item off-shelf status corresponds to the item off-shelf status.
7. The method according to claim 6, characterized in that Updating the local data of the item according to the system data, including: In the case where the system data is the on-shelf status of an item, the local data corresponding to the item is determined to be the on-shelf status or the off-shelf status according to a preset item local data setting rule.
8. A data synchronization device, characterized in that: include: A first message queue module, configured to store the data update message into a first message queue in response to monitoring a data update message of the item system; A second message queue module is used to pull a specified number of first messages from the first message queue regularly through a scheduled task, and store the pulled first messages into a second message queue; The data synchronization module is used to obtain the item identification of the item from the message in the second message queue, obtain the system data of the item from the item system according to the item identification, and update the local data of the item according to the system data.
9. A mobile electronic device terminal, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.
10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.