Data synchronization method and device, equipment and storage medium

Abstract

The invention discloses a data synchronization method and device, equipment and a storage medium, and the method comprises the steps: building a commodity service monitoring environment, and configuring a relational database, message middleware and analysis middleware in the commodity service monitoring environment; managing and storing commodity data by using a relational database, and establishing a data synchronization channel between the message-oriented middleware and each downstream system corresponding to the commodity data; monitoring whether the commodity data of the relational database is changed or not in real time by utilizing the analysis middleware; when the analysis middleware monitors that the commodity data is changed, acquiring a changed data packet; and synchronizing the change data packet to the message middleware, so that the message middleware distributes the change data packet to each downstream system through the data synchronization channel. The method can improve the monitoring efficiency without depending on the operational capability of the system, enables the commodity data to be synchronously updated efficiently and accurately, guarantees the normal popularization and selling, and improves the consumption experience of consumers.

Description

Technical Field

[0001] This application relates to the technical field of data synchronization, and specifically relates to a data synchronization method, device, equipment, and storage medium. Background Art

[0002] With the rapid development of Internet technology and the wide application of fields such as distributed systems, cloud computing, and the Internet of Things, data synchronization technology has become one of the key technologies to ensure system consistency and real-time performance. Data synchronization is mainly used to transfer and share data between different nodes, systems, or platforms to ensure data consistency and integrity. Taking the supply chain system as an example, product data is often widely used in multiple systems, and it is necessary to obtain product information in real time to complete the synchronization process for specified products.

[0003] In traditional data synchronization modes, the common means is to rely on the computing power of nodes, systems, or platforms themselves. However, their computing powers are not equal, so the overall data synchronization efficiency is relatively low, which will affect the normal promotion and sale of products and bring a bad experience to consumers. Summary of the Invention

[0004] In view of this, this application provides a data synchronization method, device, equipment, and storage medium to solve the problem that in traditional data synchronization modes, the common means is to rely on the computing power of nodes, systems, or platforms themselves, but their computing powers are not equal, so the overall data synchronization efficiency is relatively low, which will affect the normal promotion and sale of products and bring a bad experience to consumers.

[0005] To achieve the above objectives, the following solutions are proposed:

[0006] In a first aspect, a data synchronization method includes:

[0007] Build a product service monitoring environment, and configure a relational database, a message middleware, and a parsing middleware in the product service monitoring environment;

[0008] Use the relational database to manage and store product data, and at the same time establish a data synchronization channel between the message middleware and each downstream system corresponding to the product data;

[0009] Use the parsing middleware to monitor in real time whether the product data in the relational database has changed;

[0010] When the parsing middleware monitors that the product data has changed, obtain the change data packet;

[0011] Synchronize the change data packet to the message middleware for the message middleware to distribute the change data packet to each downstream system through the data synchronization channel.

[0012] Preferably, configuring a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment includes:

[0013] Installing a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment;

[0014] Enabling the binary log function of the relational database;

[0015] Configuring the global parameter file in the parsing middleware;

[0016] Invoking the instance configuration file in the parsing middleware, and configuring the connection information of the relational database and the relevant information of the message middleware in the instance configuration file.

[0017] Preferably, when the parsing middleware monitors that the commodity data has changed, obtaining a change data packet includes:

[0018] When the parsing middleware monitors that the commodity data has changed, obtaining each piece of changed data;

[0019] For each piece of the changed data, performing semantic parsing on the piece of changed data to identify the keyword of the piece of changed data;

[0020] Obtaining the commodity name corresponding to the piece of changed data;

[0021] Constructing a change entry from the commodity name and the keyword;

[0022] Combining the change entry with the changed data to form a standard change message;

[0023] Summarizing each piece of the standard change messages in the form of a table to obtain a change data packet.

[0024] Preferably, synchronizing the change data packet to the message middleware includes:

[0025] Decomposing the change data packet to obtain each piece of standard change message;

[0026] Matching each piece of the standard change messages with each preset information type to determine the information type to which each piece of the standard change message belongs;

[0027] Summarizing the pieces of standard change messages belonging to the same information type respectively to obtain each change information set;

[0028] Synchronizing each of the change information sets to the message middleware respectively.

[0029] Preferably, synchronizing each of the change information sets to the message middleware includes:

[0030] Determining the information types corresponding to each of the change information sets respectively;

[0031] For each of the change information sets, if the information type corresponding to the change information set is a critical type, synchronize the change information set to the message middleware in a real-time synchronization manner;

[0032] If the information type corresponding to the change information set is a product description type, synchronize the change information set to the message middleware in a batch synchronization manner;

[0033] If the information type corresponding to the change information set is a historical record type, synchronize the change information set to the message middleware in a delayed synchronization manner.

[0034] Preferably, synchronizing the change information set to the message middleware in a batch synchronization manner includes:

[0035] Set a dynamic time window, accumulate the change information set and other change information sets within the window period of the dynamic time window, and store them in a data packet; wherein, the data packet is set with a capacity upper limit;

[0036] Real-time monitor whether the total data volume of all change information sets in the data packet reaches the capacity upper limit;

[0037] When the dynamic time window ends or the data volume reaches the capacity upper limit, synchronize all the change information sets in the data packet to the message middleware.

[0038] Preferably, the information types include critical types, product description types, and historical record types;

[0039] The standard change information corresponding to the critical type includes product price, product inventory, product on-shelf and off-shelf status, and time-limited promotion activities;

[0040] The standard change information corresponding to the product description type includes product introduction, product classification, product attributes, product pictures, and merchant information;

[0041] The standard change information corresponding to the historical record type includes product historical data, product evaluations and ratings, logistics information, and product statistical analysis data.

[0042] In a second aspect, a data synchronization device, characterized by comprising:

[0043] Configuration module, used to build a commodity service monitoring environment, and configure a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment;

[0044] Data synchronization channel establishment module, used to manage and store commodity data using the relational database, and at the same time establish a data synchronization channel between the message middleware and each downstream system corresponding to the commodity data;

[0045] Change judgment module, used to use the parsing middleware to monitor in real time whether the commodity data in the relational database has changed;

[0046] Changed data packet acquisition module, used to obtain a changed data packet when the parsing middleware monitors that the commodity data has changed;

[0047] Data synchronization module, used to synchronize the changed data packet to the message middleware, so that the message middleware distributes the changed data packet to each of the downstream systems through the data synchronization channel.

[0048] In a third aspect, a data synchronization device includes a memory and a processor;

[0049] The memory is used to store programs;

[0050] The processor is used to execute the program to implement each step of the data synchronization method described in any item of the first aspect.

[0051] In a fourth aspect, a storage medium stores a computer program, and when the computer program is executed by a processor, each step of the data synchronization method described in any item of the first aspect is implemented.

[0052] As can be seen from the above technical solution, in this application, a monitoring environment for commodity services is built, and a relational database, a message middleware, and a parsing middleware are configured in the monitoring environment for commodity services; the relational database is used to manage and store commodity data, and at the same time, a data synchronization channel is established between the message middleware and each downstream system corresponding to the commodity data; the parsing middleware is used to monitor in real time whether the commodity data in the relational database has changed; when the parsing middleware monitors that the commodity data has changed, a change data packet is obtained; the change data packet is synchronized to the message middleware, so that the message middleware can distribute the change data packet to each of the downstream systems through the data synchronization channel. This application first builds a monitoring environment for commodity services, configures a relational database, a message middleware, and a parsing middleware in the monitoring environment, uses the relational database to manage and store commodity data, establishes a data synchronization channel between the message middleware and the downstream systems corresponding to the commodity data, and uses this data synchronization channel to perform data synchronization. Then, the parsing middleware is used to monitor the change situation of the commodity data in the relational database in real time, so that it is not necessary to rely on the computing power of the system itself, improving the monitoring efficiency, and synchronizing to the message middleware in the form of a data packet. The change data packet is efficiently sent to the downstream systems through the data synchronization channel by the message middleware, and it is not necessary for the commodity service system that originally stores the commodity data and the downstream systems to have equivalent computing power, thereby overall improving the efficiency of data synchronization. In this way, the commodity data can be synchronized and updated efficiently and accurately, ensuring normal promotion and sales, and improving the consumption experience of consumers. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to the provided drawings.

[0054] Figure 1 An alternative flowchart of a data synchronization method provided by an embodiment of this application;

[0055] Figure 2 An alternative flowchart structure diagram of a data synchronization method provided by an embodiment of this application;

[0056] Figure 3 A schematic structural diagram of a data synchronization device provided by an embodiment of this application;

[0057] Figure 4 A schematic structural diagram of a data synchronization device provided by an embodiment of this application. Detailed implementation manners

[0058] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0059] The present invention can be used in numerous general-purpose or special-purpose computing device environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, and so on.

[0060] An embodiment of the present invention provides a data synchronization method, which can be applied to various computer terminals or intelligent terminals, and the execution subject thereof can be a processor or a server of a computer terminal or an intelligent terminal. The method flow chart of the method is as Figure 1 shown, and specifically includes:

[0061] S1: Build a commodity service monitoring environment, and configure a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment.

[0062] The present application first builds a commodity service monitoring environment, and configures a relational database (MySQL), a message middleware (RabbitMQ), and a parsing middleware (Canal). Their versions can be specified to ensure the normal operation of the commodity service monitoring environment. MySQL can be selected as 8.0.22, RabbitMQ can be selected as 3.11.2, and Canal can be selected as 1.1.5.

[0063] MySql is a widely used open-source relational database management system that supports transactions, stored procedures, and a powerful query language (SQL). In the data change synchronization scenario, the Binlog (Binary Log) of MySQL is the core component, which records all update operations of the database, including the complete history of transaction submissions, and is used for master-slave replication, backup recovery, and real-time data capture. By listening to and parsing Binlog, efficient data synchronization and event-driven can be achieved.

[0064] RabbitMq is an open-source message middleware that supports multiple message protocols and is implemented based on AMQP (Advanced Message Queuing Protocol). It provides reliable message sending, queue management, and consumer models, and is suitable for scenarios with high concurrency and asynchronous communication. In a data synchronization architecture, RabbitMQ is usually used to distribute data change events after binlog parsing to multiple downstream systems.

[0065] Canal is an open-source middleware for parsing MySQL Binlog. It emulates the MySQL slave protocol and captures and parses database changes (inserts, updates, deletes, etc.) by subscribing to the Binlog of the master database. Canal efficiently and reliably pushes the incremental data of the database to downstream systems and is commonly used for data synchronization, real-time computing, and the implementation of event-driven architectures.

[0066] S2: Use the relational database to manage and store commodity data, and at the same time establish a data synchronization channel between the message middleware and each downstream system corresponding to the commodity data.

[0067] In this application, by using a relational database to manage and store commodity data and establishing a data synchronization channel between the message middleware and each downstream system corresponding to the commodity data, asynchronous synchronization can be achieved, avoiding direct dependencies between the system commodity service system and downstream systems, and reducing the coupling degree.

[0068] Moreover, this application does not directly synchronize data through the system. In the form of a channel, data can be quickly transmitted after data changes, significantly reducing latency.

[0069] S3: Use the parsing middleware to continuously monitor whether the commodity data in the relational database has changed.

[0070] This application uses parsing middleware to implement the monitoring function, which can capture all changes in commodity data in the relational data, including inserts, updates, deletes, etc.

[0071] S4: When the parsing middleware monitors that the commodity data has changed, obtain the change data packet.

[0072] The parsing middleware can capture each change in commodity data and obtain the change data packet in real time. It supports multiple data formats, facilitating the application and parsing of downstream systems.

[0073] Packaging the changed data in the form of data packets for synchronization can prevent the loss of changed data, reduce the number of network communications, and reduce network overhead.

[0074] S5: Synchronize the change data packet to the message middleware for the message middleware to distribute the change data packet to each downstream system through the data synchronization channel.

[0075] During the synchronization process, the change data packet can be compressed to reduce the amount of data transmitted and improve the transmission speed; the change data packet can also be encrypted to ensure the security of the changed data.

[0076] Optionally, one or more data synchronization channels can be established according to the quantity and type of downstream systems. For example, each downstream system can have an exclusive data synchronization channel to ensure data isolation, or downstream systems of the same type can share the same data synchronization channel to reduce waste of channel resources.

[0077] As can be seen from the above technical solution, this application builds a commodity service monitoring environment and configures a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment; uses the relational database to manage and store commodity data, and at the same time establishes a data synchronization channel between the message middleware and each downstream system corresponding to the commodity data; uses the parsing middleware to monitor in real time whether the commodity data in the relational database has changed; when the parsing middleware monitors that the commodity data has changed, obtain the change data packet; synchronize the change data packet to the message middleware for the message middleware to distribute the change data packet to each downstream system through the data synchronization channel. This application first builds a monitoring environment for the commodity service, configures a relational database, a message middleware, and a parsing middleware in the monitoring environment, uses the relational database to manage and store commodity data, establishes a data synchronization channel between the message middleware and the downstream system corresponding to the commodity data, and uses this data synchronization channel for data synchronization. Then, it monitors the change of commodity data in the relational database in real time through the parsing middleware, so that it does not need to rely on the computing power of the system itself, improves the monitoring efficiency, and synchronizes it to the message middleware in the form of a data packet. The message middleware efficiently sends the change data packet to the downstream system through the data synchronization channel, and does not require the commodity service system and the downstream system that originally store the commodity data to have equivalent computing power, thereby overall improving the efficiency of data synchronization, so that the commodity data can be synchronized and updated efficiently and accurately, ensuring normal promotion and sales, and improving the consumption experience of consumers.

[0078] In the method provided by the embodiment of the present invention, the process of configuring a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment is specifically described as follows:

[0079] Install a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment;

[0080] Enable the binary log function of the relational database;

[0081] Configure the global parameter file in the parsing middleware;

[0082] Call the instance configuration file in the parsing middleware, and configure the connection information of the relational database and the relevant information of the message middleware in the instance configuration file.

[0083] Specifically, during the configuration process, it is first necessary to enable the binlog binary log function, that is, enable binlog in the my.cnf file. Among them, the Binlog format can be set to row level, and a unique server ID can be configured for master-slave replication or binlog parsing. In addition, configure the message queue for starting the message middleware and the parsing middleware. The parsing middleware is responsible for following data monitoring and connecting to the message middleware.

[0084] It is also necessary to configure the canal.properties file (global parameter file) of the parsing middleware, and configure the instance.properties file (instance configuration file) to connect to the relational database.

[0085] Optionally, the client of the parsing middleware can be integrated in Spring Boot to achieve real-time monitoring of the relational database. Among them, it is necessary to add necessary dependencies through Maven, write Canal client code to monitor, connect to the parsing middleware server through CanalConnector, subscribe to all tables of the relational database, and continuously obtain changed data.

[0086] The process of obtaining the change data packet when the parsing middleware in the present application monitors that the commodity data has changed will be described in detail below.

[0087] When the parsing middleware monitors that the commodity data has changed, obtain each piece of changed data;

[0088] For each piece of the changed data, perform semantic parsing on the piece of changed data to identify the keywords of the piece of changed data;

[0089] Obtain the commodity name corresponding to the piece of changed data;

[0090] Construct a change entry from the commodity name and the keywords;

[0091] Combine the change entry with the changed data to form a standard change message;

[0092] Summarize each piece of the described standard change information in tabular form to obtain a change data packet.

[0093] Specifically, when it is monitored that the commodity data has changed, in order to make the changed data clearer, construct a more regular change data packet, and improve the subsequent analysis and synchronization of the change data packet, after obtaining the changed data in this application, it does not directly perform packaging. Instead, for each piece of changed data, a change entry is constructed from its corresponding commodity name and keyword to combine the change entry with the changed data itself to form standard changed data. The change entry can be regarded as a label marked on the changed data.

[0094] Then summarize the standard changed data in tabular form. Among them, the change entry can be used as the first column data of each row of the table, acting as a title, and then the changed data itself is filled in the columns behind that row, thus forming tabular data with multiple rows and multiple columns, and using this tabular data as a change data packet.

[0095] The following embodiments explain in detail the steps of synchronizing the change data packet to the message middleware in this application.

[0096] Decompose the change data packet to obtain each piece of standard change information;

[0097] Match each piece of the described standard change information with each preset information type to determine the information type to which each piece of the standard change information belongs;

[0098] Summarize separately each piece of standard change information with the same information type to obtain each change information set;

[0099] Synchronize each of the change information sets to the message middleware separately.

[0100] Among them, for the process of synchronizing each of the change information sets to the message middleware separately, it includes:

[0101] Determine separately the information type corresponding to each of the change information sets;

[0102] For each of the change information sets, if the information type corresponding to the change information set is a key type, then synchronize the change information set to the message middleware in a real-time synchronization manner;

[0103] If the information type corresponding to the change information set is a commodity description type, then synchronize the change information set to the message middleware in a batch synchronization manner;

[0104] If the information type corresponding to the change information set is of the historical record type, the change information set is synchronized to the message middleware in a delayed synchronization manner.

[0105] Specifically, considering that the standard change data also includes various different information types, if the standard change data of different information types are synchronized in different ways, the flexibility of data synchronization can be improved and resources can be saved. For example, the information type is divided into key type, product description type, and historical record type. Among them, the standard change information corresponding to the key type includes product price, product inventory, product on / off-shelf status, and limited-time promotion activities; the standard change information corresponding to the product description type includes product introduction, product classification, product attributes, product pictures, and merchant information; the standard change information corresponding to the historical record type includes product historical data, product evaluations and ratings, logistics information, and product statistical analysis data. Therefore, appropriate synchronization methods are selected according to different information types. For example, real-time synchronization is used for data with high real-time requirements, and batch synchronization is used for batch data, etc. This can support multi-scenario and diversified data synchronization, and can also dynamically adjust the synchronization strategy to improve flexibility and adapt to business changes.

[0106] In addition, for the process of synchronizing the change information set to the message middleware in a batch synchronization manner, the following operations can be performed:

[0107] Set a dynamic time window. During the window period of the dynamic time window, accumulate the change information set and other change information sets, and store them in a data packet. Among them, the data packet is set with a capacity limit;

[0108] Real-time monitor whether the total data volume of all change information sets in the data packet reaches the capacity limit;

[0109] When the dynamic time window ends or the data volume reaches the capacity limit, synchronize all the change information sets in the data packet to the message middleware.

[0110] Specifically, the batch synchronization method can reduce the number of network communications and network overhead, improve the transmission efficiency, and can also reduce the system load and optimize the resource utilization rate. However, it should be noted that the number of change information sets should not be too large, and time should also be noted. Therefore, a dynamic time window mechanism is set to ensure timeliness and prevent data overload.

[0111] For the change information set of the historical record type, it can be synchronized in a delayed manner to adapt to a weak network environment and reduce the occupation of real-time resources.

[0112] In an example, such as Figure 2As shown, taking the new product supply scenario as an example, when data changes corresponding to new product replenishment occur in the commodity service system, the relational database can monitor them in real time through the parsing middleware, package the changed data into a change data packet, synchronize it to the message queue of the message middleware, and then synchronize it to downstream systems, including the store replenishment system, intelligent replenishment system, supplier system, and so on.

[0113] corresponding to Figure 1 the method described above, the embodiment of the present invention also provides a data synchronization device for Figure 1 the specific implementation of the method in, the data synchronization device provided by the embodiment of the present invention can be in a computer terminal or various mobile devices, combined with Figure 3 to introduce the data synchronization device, as Figure 3 shown, the device may include:

[0114] Configuration module 10, for building a commodity service monitoring environment and configuring a relational database, message middleware, and parsing middleware in the commodity service monitoring environment;

[0115] Data synchronization channel establishment module 20, for using the relational database to manage and store commodity data, and at the same time establishing a data synchronization channel between the message middleware and each downstream system corresponding to the commodity data;

[0116] Change judgment module 30, for using the parsing middleware to monitor in real time whether the commodity data in the relational database has changed;

[0117] Change data packet acquisition module 40, for when the parsing middleware monitors that the commodity data has changed, acquiring the change data packet;

[0118] Data synchronization module 50, for synchronizing the change data packet to the message middleware for the message middleware to distribute the change data packet to each of the downstream systems through the data synchronization channel.

[0119] As can be seen from the above technical solution, in this application, a monitoring environment for commodity services is built, and a relational database, a message middleware, and a parsing middleware are configured in the monitoring environment for commodity services; the relational database is used to manage and store commodity data, and at the same time, a data synchronization channel is established between the message middleware and each downstream system corresponding to the commodity data; the parsing middleware is used to monitor in real time whether the commodity data in the relational database has changed; when the parsing middleware monitors that the commodity data has changed, a change data packet is obtained; the change data packet is synchronized to the message middleware for the message middleware to distribute the change data packet to each downstream system through the data synchronization channel. In this application, a monitoring environment is first built for commodity services, and a relational database, a message middleware, and a parsing middleware are configured in the monitoring environment. The relational database is used to manage and store commodity data, and a data synchronization channel is established between the message middleware and the downstream system corresponding to the commodity data. Data synchronization is performed using this data synchronization channel. Then, the change of commodity data in the relational database is monitored in real time through the parsing middleware, so that the computing power of the system itself does not need to be relied on, the monitoring efficiency is improved, and it is synchronized to the message middleware in the form of a data packet. The change data packet is efficiently sent to the downstream system through the data synchronization channel by the message middleware, and the commodity service system and the downstream system that originally stored the commodity data do not need to have equivalent computing power, thereby improving the overall data synchronization efficiency. In this way, the commodity data can be synchronously updated efficiently and accurately, ensuring normal promotion and sales and improving the consumption experience of consumers.

[0120] Furthermore, an embodiment of this application provides a data synchronization device. Optionally, Figure 4 The hardware structure block diagram of the data synchronization device is shown. Referring to Figure 4 , the hardware structure of the data synchronization device may include: at least one processor 01, at least one communication interface 02, at least one memory 03, and at least one communication bus 04.

[0121] In the embodiment of this application, the number of the processor 01, the communication interface 02, the memory 03, and the communication bus 04 is at least one, and the processor 01, the communication interface 02, and the memory 03 complete communication with each other through the communication bus 04.

[0122] The processor 01 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.

[0123] The memory 03 may include a high-speed RAM memory and may also include a non-volatile memory, such as at least one disk memory.

[0124] Among them, the memory stores a program, and the processor can call the program stored in the memory. The program is used to execute the following data synchronization method, including:

[0125] Build a commodity service monitoring environment, and configure a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment;

[0126] Use the relational database to manage and store commodity data, and at the same time establish a data synchronization channel between the message middleware and each downstream system corresponding to the commodity data;

[0127] Use the parsing middleware to monitor in real time whether the commodity data in the relational database has changed;

[0128] When the parsing middleware monitors that the commodity data has changed, obtain the change data packet;

[0129] Synchronize the change data packet to the message middleware for the message middleware to distribute the change data packet to each of the downstream systems through the data synchronization channel.

[0130] Optionally, the refined functions and extended functions of the program can refer to the description of the data synchronization method in the method embodiment.

[0131] The embodiment of the present application also provides a storage medium. The storage medium can store a program suitable for being executed by a processor. When the program runs, it controls the device where the storage medium is located to execute the following data synchronization method, including:

[0132] Build a commodity service monitoring environment, and configure a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment;

[0133] Use the relational database to manage and store commodity data, and at the same time establish a data synchronization channel between the message middleware and each downstream system corresponding to the commodity data;

[0134] Use the parsing middleware to monitor in real time whether the commodity data in the relational database has changed;

[0135] When the parsing middleware monitors that the commodity data has changed, obtain the change data packet;

[0136] Synchronize the change data packet to the message middleware for the message middleware to distribute the change data packet to each of the downstream systems through the data synchronization channel.

[0137] Specifically, the storage medium may be a computer-readable storage medium, and the computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM.

[0138] Optionally, the refinement function and the extension function of the program may refer to the description of the data synchronization method in the method embodiments.

[0139] In addition, in each of the embodiments of the present disclosure, the functional modules may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a live broadcast device, or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present disclosure.

[0140] Finally, it should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0141] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts among the various embodiments may be referred to each other.

[0142] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data synchronization method, characterized in that: include: Building a commodity service monitoring environment, and configuring a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment; The relational database is used to manage and store commodity data, and a data synchronization channel is established between the message middleware and various downstream systems corresponding to the commodity data; Using the analysis middleware to monitor in real time whether the commodity data in the relational database is changed; When the parsing middleware detects that the product data has changed, it obtains the changed data packet; The changed data packet is synchronized to the message middleware, so that the message middleware distributes the changed data packet to each of the downstream systems through the data synchronization channel.

2. The method according to claim 1, characterized in that The configuration of the relational database, the message middleware and the parsing middleware in the commodity service monitoring environment includes: Installing a relational database, a message middleware and a parsing middleware in the commodity service monitoring environment; Enable the binary log function of the relational database; Configure the global parameter file in the parsing middleware; The instance configuration file in the parsing middleware is called, and the connection information of the relational database and the related information of the message middleware are configured in the instance configuration file.

3. The method according to claim 1, characterized in that When the parsing middleware monitors that the commodity data has changed, it obtains the changed data packet, including: When the analysis middleware monitors changes in product data, it obtains each piece of changed data; For each piece of the change data, semantic analysis is performed on the change data to identify keywords of the change data; Get the product name corresponding to the changed data; Constructing a change entry from the product name and keywords; Combining the change entry with the change data to form a standard change information; The various pieces of standard change information are summarized in a table format to obtain a change data package.

4. The method according to claim 1, characterized in that The step of synchronizing the changed data packet to the message middleware includes: Decomposing the change data packet to obtain each piece of standard change information; Matching each piece of the standard change information with each piece of preset information type to determine the information type to which each piece of the standard change information belongs; Summarize the pieces of standard change information of the same information type to obtain each change information set; Each of the change information sets is synchronized to the message middleware respectively.

5. The method according to claim 4, characterized in that The step of synchronizing each of the change information sets to the message middleware includes: Respectively determine the information type corresponding to each of the change information sets; For each of the change information sets, if the information type corresponding to the change information set is a key type, the change information set is synchronized to the message middleware in a real-time synchronization manner; If the information type corresponding to the change information set is a product description type, the change information set is synchronized to the message middleware in batch synchronization; If the information type corresponding to the change information set is a historical record type, the change information set is synchronized to the message middleware in a delayed synchronization manner.

6. The method according to claim 5, characterized in that The step of synchronizing the change information set to the message middleware in a batch synchronization manner includes: A dynamic time window is set, and the change information set and other change information sets are accumulated within the window period of the dynamic time window, and stored in a data packet; wherein the data packet is set with a capacity upper limit; Real-time monitoring of whether the total data volume of all change information sets in the data packet reaches the capacity upper limit; When the dynamic time window ends or the data volume reaches the upper capacity limit, all the change information sets in the data packet are synchronized to the message middleware.

7. The method according to any one of claims 4 to 6, characterized in that: The information types include key type, product description type and historical record type; The standard change information corresponding to the key types includes product price, product inventory, product shelf status and limited-time promotion activities; The standard change information corresponding to the product description type includes product introduction, product category, product attributes, product pictures and merchant information; The standard change information corresponding to the historical record type includes product historical data, product evaluation and rating, logistics information, and product statistical analysis data.

8. A data synchronization device, characterized in that: include: A configuration module, used to build a commodity service monitoring environment, and configure a relational database, a message middleware, and a parsing middleware in the commodity service monitoring environment; A data synchronization channel establishment module is used to manage and store commodity data using the relational database, and to establish a data synchronization channel between the message middleware and various downstream systems corresponding to the commodity data; A change judgment module, used to monitor in real time whether the commodity data in the relational database has changed using the analysis middleware; A changed data packet acquisition module, used to acquire a changed data packet when the parsing middleware monitors that the commodity data has changed; A data synchronization module is used to synchronize the changed data packet to the message middleware, so that the message middleware can distribute the changed data packet to each of the downstream systems through the data synchronization channel.

9. A data synchronization device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the data synchronization method as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the data synchronization method according to any one of claims 1 to 7 is implemented.

Images