Data synchronization method, device, equipment and storage medium

Abstract

The present application discloses a data synchronization method, device, equipment and storage medium. The method obtains target data, and for each data receiving end, detects the network transmission status between it and the receiving end, thereby determining whether the data receiving end uses the real-time synchronization mode or the batch synchronization mode. For the data receiving end using the real-time synchronization mode, the target data is directly sent to improve the efficiency of transmitting the target data to the data receiving end; for the data receiving end using the batch synchronization mode, the target data is pre-processed by the pre-stored business processing logic corresponding to the data receiving end, and then the pre-processed target data is sent to the data receiving end in batches, thereby improving the processing efficiency of the target data. The present application can adjust the data synchronization mode in combination with the network transmission status corresponding to the data receiving end, maximize the utilization rate of network transmission resources, and improve the efficiency of data transmission and processing. The technical solution of the present application can be widely used in the field of data synchronization technology.

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization, and in particular to a data synchronization method, apparatus, device and storage medium. Background Art

[0002] At present, with the rapid development of information technology, related applications have gradually been integrated into people's lives, providing people with a variety of services. These applications may involve various types of data transmission and processing. In some scenarios, different data receivers need to maintain data consistency. If the received data is inconsistent, it may affect the performance of the application. At this time, data synchronization technology is needed to synchronously distribute the same data to different data receivers.

[0003] In the related art, when data synchronization is performed, in order to improve the efficiency of data transmission and processing, real-time synchronization is generally performed. For example, the data is placed in a message queue, and different data receiving ends can directly read and process the data from the message queue. Since each data receiving end reads data from the same message queue, data consistency is guaranteed. However, in actual applications, it is found that some data receiving ends may have poor network transmission status and cannot obtain data in time for processing, resulting in low utilization of network transmission resources and affecting the efficiency of data processing. Summary of the invention

[0004] The embodiments of the present application provide a data synchronization method, apparatus, device and storage medium, which can improve the utilization rate of network transmission resources and improve the efficiency of data transmission and processing.

[0005] One aspect of an embodiment of the present application provides a data synchronization method, which is used for a data sending end, wherein the data sending end performs data synchronization transmission to multiple data receiving ends, and the data sending end pre-stores the service processing logic of each of the data receiving ends; the method includes:

[0006] Get target data;

[0007] Detecting the network transmission status between each of the data receiving ends, and determining the data synchronization mode corresponding to each of the data receiving ends according to the network transmission status; wherein the data synchronization mode includes a real-time synchronization mode and a batch synchronization mode;

[0008] If it is determined that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode, sending the target data to the data receiving end;

[0009] If it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode, the pre-stored business processing logic of the data receiving end is queried, the target data is processed according to the business processing logic, and the processed target data is sent to the data receiving end in batches.

[0010] On the other hand, an embodiment of the present application provides a data synchronization device, which is used for a data sending end, wherein the data sending end performs data synchronization transmission to multiple data receiving ends, and the data sending end pre-stores the service processing logic of each of the data receiving ends; the device includes:

[0011] An acquisition unit, used for acquiring target data;

[0012] A detection unit, used to detect the network transmission status between each of the data receiving terminals, and determine the data synchronization mode corresponding to each of the data receiving terminals according to the network transmission status; wherein the data synchronization mode includes a real-time synchronization mode and a batch synchronization mode;

[0013] A first processing unit, configured to send the target data to the data receiving end if it is determined that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode;

[0014] The second processing unit is used to query the pre-stored business processing logic of the data receiving end if it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode, process the target data according to the business processing logic, and send the processed target data to the data receiving end in batches.

[0015] Optionally, in some embodiments, the device further includes a conversion unit, and the conversion unit is specifically configured to:

[0016] According to the data structure of the linked list, convert the target data into a plurality of log data;

[0017] According to the data sorting in the target data, a log index corresponding to each piece of the log data is determined.

[0018] Optionally, in some embodiments, the first processing unit is specifically configured to:

[0019] Acquire a system identifier of the data receiving end;

[0020] According to the log index, a number of the log data are extracted as the data to be synchronized in the current batch, and the first log index corresponding to the log data at the end position in the data to be synchronized is queried;

[0021] The system identifier, the data to be synchronized, and the first log index are sent to the data receiving end.

[0022] Optionally, in some embodiments, the first processing unit is specifically configured to:

[0023] Query the second log index of the most recent feedback from the data receiving end; wherein the second log index is used to represent the log index corresponding to the log data at the end position in a batch of data to be synchronized that has been synchronized by the data receiving end;

[0024] Starting from the next log data of the log data corresponding to the second log index, the first number of log data is extracted as the data to be synchronized in the current batch.

[0025] Optionally, in some embodiments, the first processing unit is specifically configured to:

[0026] Detecting a second number of data receiving terminals that are performing data synchronization at the current time node, and a third number of data synchronization requests received in a first preset time period before the current time node;

[0027] Determine, according to the second number and the third number, the first number used when extracting the data to be synchronized of the current batch;

[0028] The first number is negatively correlated with the second number, and the first number is negatively correlated with the third number.

[0029] Optionally, in some embodiments, the first processing unit is further configured to:

[0030] Recording the first time node of sending the current batch of data to be synchronized to the data receiving end;

[0031] If a second log index corresponding to the current batch fed back by the data receiving end is received within a second preset period after the first time node, extracting the next batch of data to be synchronized according to the second log index corresponding to the current batch;

[0032] If the second log index corresponding to the current batch fed back by the data receiving end is not received within a second preset time period after the first time node, the data to be synchronized in the current batch is resent to the data receiving end.

[0033] Optionally, in some embodiments, the second processing unit is specifically configured to:

[0034] Acquire a system identifier of the data receiving end;

[0035] Processing the log data according to the business processing logic to obtain processing data corresponding to the log data;

[0036] When the amount of the newly added processed data reaches a preset capacity threshold, or the time since the last sending of the processed data reaches a preset time threshold, query the third log index corresponding to the last log data of the current preprocessing;

[0037] The system identifier, the processed data and the third log index are sent to the data receiving end.

[0038] Optionally, in some embodiments, the detection unit is specifically used for:

[0039] Detecting network environment parameters between the data receiving end and the data receiving end;

[0040] Determining a network quality score corresponding to the data receiving end according to the network environment parameters;

[0041] If the network quality score is greater than a preset score threshold, determining that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode;

[0042] If the network quality score is less than or equal to the preset score threshold, it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode.

[0043] Optionally, in some embodiments, the data sending end is a distributed source data system, the distributed source data system includes a master node and multiple slave nodes, the master node is used to synchronously transmit data to multiple data receiving ends; the acquisition unit is specifically used to:

[0044] Acquire original data through the master node;

[0045] The original data is converted by the master node to obtain standard data in a universal data exchange format;

[0046] The master node sends the standard data to each of the slave nodes, and the slave nodes reach a consensus on the standard data, and the consensus-based standard data is determined as the target data.

[0047] Optionally, in some embodiments, the acquiring unit is specifically configured to:

[0048] Performing hash processing on the standard data by the master node to obtain a first hash value;

[0049] Sending the standard data and the first hash value to each of the slave nodes through the master node;

[0050] Performing hash processing on the standard data by the slave node to obtain a second hash value, and verifying the consistency of the first hash value and the second hash value to obtain a consistency verification result corresponding to the standard data;

[0051] When the consistency check results of more than half of the slave nodes are passed, it is determined that the standard data has completed consensus.

[0052] Optionally, in some embodiments, the data sending end is a distributed source data system, the distributed source data system includes a master node and multiple slave nodes, the master node is used to synchronously transmit data to multiple data receiving ends; the device also includes a switching unit, the switching unit is specifically used to:

[0053] Synchronize the data in the master node to each of the slave nodes;

[0054] Detecting the operating status of the distributed source data system;

[0055] When it is determined that the distributed source data system needs to switch the master node, determining a new master node from the slave nodes;

[0056] Detect whether there is an unfinished data synchronization task in the old master node; wherein the unfinished data synchronization task is a data synchronization task for which the data to be synchronized has been sent but the second log index corresponding to the feedback from the data receiving end has not been received;

[0057] If there are unfinished data synchronization tasks in the old master node, stop sending new data to be synchronized through the old master node until the unfinished data synchronization task is completed or times out, and switch to the new master node for data synchronization transmission.

[0058] On the other hand, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0059] The memory is used to store computer programs;

[0060] The processor executes the computer program to implement the aforementioned data synchronization method.

[0061] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the aforementioned data synchronization method.

[0062] On the other hand, an embodiment of the present application also provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the aforementioned data synchronization method.

[0063] The embodiments of the present application include at least the following beneficial effects: the present application provides a data synchronization method, device, equipment and storage medium. The method of the present application can be applied to a data sending end, which is used to perform data synchronization transmission to multiple data receiving ends, and the business processing logic of each data receiving end is pre-stored. The method obtains the target data, and for each data receiving end, detects the network transmission status between it, so as to determine whether the data receiving end uses the real-time synchronization mode or the batch synchronization mode. For the data receiving end using the real-time synchronization mode, the target data is directly sent to improve the efficiency of the target data transmission to the data receiving end; for the data receiving end using the batch synchronization mode, the target data is pre-processed by the pre-stored business processing logic corresponding to the data receiving end, and then the pre-processed target data is sent to the data receiving end in batches, so as to improve the processing efficiency of the target data. The present application can adjust the data synchronization mode in combination with the network transmission status corresponding to the data receiving end, so as to maximize the utilization rate of network transmission resources and improve the efficiency of data transmission and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0065] Figure 1 A schematic diagram of a data synchronization method used in the related art;

[0066] Figure 2 A system architecture diagram for a data synchronization method provided in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of a system architecture of a streaming media service provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of the relationship between a data sending end and a data receiving end provided in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of a data synchronization method provided in an embodiment of the present application;

[0070] Figure 6A schematic diagram of converting target data into log data provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of extracting data to be synchronized provided in an embodiment of the present application;

[0072] Figure 8 A schematic diagram of the application principle of an RSS system provided in an embodiment of the present application;

[0073] Fig. 9 A schematic diagram of various data structures transmitted in an embodiment of the present application;

[0074] Fig.10 A schematic diagram of a principle of data synchronization provided in an embodiment of the present application;

[0075] Fig.11 A schematic diagram of the principle of a distributed source data system crash rollback provided in an embodiment of the present application;

[0076] Fig.12 A structural block diagram of a data synchronization device provided in an embodiment of the present application;

[0077] Fig.13 This is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0079] It can be understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept.

[0080] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0081] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0082] 1) ETL (Extract, Transform, Load) tools, a key technology for data integration and data warehouse construction, the application process of ETL tools involves three main steps: extraction, transformation, and loading. ETL tools can help extract data from multiple data sources, perform necessary transformations and cleansing, and then load the data into the target data warehouse or data lake.

[0083] 2) Kafka, a distributed stream processing platform, is mainly used to build real-time data pipelines and streaming applications. It is designed to process large amounts of data with high throughput and low latency.

[0084] 3) Pulsar, a distributed message queue system designed to handle large-scale messaging and stream processing tasks.

[0085] 4) GDEF (General Data Exchange Format), a general data exchange format, aims to provide a standardized way to represent and exchange various types of data. This format is often used for data transmission between different systems to ensure data consistency and readability.

[0086] 5) Raft, a distributed consensus algorithm for managing replicated logs. It aims to solve the complexity of understanding and implementing early consensus algorithms, with the goal of providing higher understandability and better practical applicability.

[0087] At present, with the rapid development of information technology, related applications have gradually been integrated into people's lives, providing people with a variety of services. These applications may involve various types of data transmission and processing. In some scenarios, different data receivers need to maintain data consistency. If the received data is inconsistent, it may affect the performance of the application.

[0088] For example, in the scenario of financial transactions, such as bank transfers and stock trading, the data consistency needs to be maintained between the various terminal devices involved in the transaction data (such as the terminal devices used by traders and the terminal devices of the bank system), otherwise it may lead to financial losses or abnormal transaction records. For example, in a medical information system, data such as medical records, diagnosis results, and treatment plans need to be consistent between the hospital's information system, doctors, and the terminal devices used by patients, otherwise it may lead to misdiagnosis or failure to understand the condition in time, affecting the medical experience. For another example, in social network services, data such as account information, friend relationships, and interactive messages need to maintain data consistency between terminal devices of different objects, otherwise asymmetric information or erroneous information may be transmitted, which will affect the interactive experience.

[0089] Therefore, in the related art, data synchronization technology is often used for the above-mentioned scenarios where data consistency needs to be maintained. For example, in some schemes, ETL tools are used for data synchronization. ETL tools can extract data from the data source, perform necessary conversions and cleaning, and then load the data into the target data warehouse or data lake. For the scenario in this application where data consistency of multiple data receiving ends needs to be maintained, the data that needs to be synchronized can be directly obtained from the data source, and after conversion and cleaning, the data can be loaded into multiple data receiving ends. This implementation method synchronizes data in batches, and the data after conversion and cleaning is sent to the data receiving end only when a certain amount of data is reached. The real-time performance is poor, which may affect the efficiency of the data receiving end in processing data, and network transmission resources cannot be well utilized.

[0090] In order to improve the efficiency of data transmission and processing, some solutions will adopt real-time synchronization strategies. For example, data can be placed in a message queue, where the message queue can be a Kafka queue, a Pulsar queue, etc. For example, please refer to Figure 1 , Figure 1 FIG. 1 is a schematic diagram showing a data synchronization method used in the related art. Figure 1 As shown, in the related art, when synchronizing data, the data can be placed in a message queue, and different data receiving ends can directly read and process the data from the message queue. Since each data receiving end reads data from the same message queue, data consistency is guaranteed. However, in actual applications, it is found that some data receiving ends may have poor network transmission status and cannot obtain data for processing in time, resulting in low utilization of network transmission resources and affecting the efficiency of data processing.

[0091] In view of this, a data synchronization method, device, equipment and storage medium are provided in the embodiments of the present application. The method of the present application can be applied to a data sending end, which is used to perform data synchronization transmission to multiple data receiving ends, and the business processing logic of each data receiving end is pre-stored. The method obtains the target data, and for each data receiving end, detects the network transmission status between it, so as to determine whether the data receiving end uses the real-time synchronization mode or the batch synchronization mode. For the data receiving end using the real-time synchronization mode, the target data is directly sent to improve the efficiency of transmitting the target data to the data receiving end; for the data receiving end using the batch synchronization mode, the target data is pre-processed by the pre-stored business processing logic corresponding to the data receiving end, and then the pre-processed target data is sent to the data receiving end in batches, so as to improve the processing efficiency of the target data. The present application can adjust the data synchronization mode in combination with the network transmission status corresponding to the data receiving end, so as to maximize the utilization rate of network transmission resources and improve the efficiency of data transmission and processing.

[0092] System architecture and scenario description of the application examples

[0093] Figure 2 It is a system architecture diagram used in a data synchronization method provided in an embodiment of the present application, which includes a terminal device 240, the Internet 230, a gateway 220, a backend server 210, etc.

[0094] In the embodiment of the present application, the terminal device 240 can install and run related applications, such as video playback applications, shopping applications, etc. The terminal device 240 can include various forms such as desktop computers, laptop computers, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, home theater terminals, and dedicated terminals. In addition, it can be a single device or a collection of multiple devices. The terminal device 240 can communicate with the Internet 230 in a wired or wireless manner to exchange data.

[0095] The backend server 210 refers to a computer system that can provide certain services to the terminal device 240. Compared with the ordinary terminal device 240, the backend server 210 has higher requirements in terms of stability, security, performance, etc. The backend server 210 can be a high-performance computer in the network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc.

[0096] The gateway 220 is also called an internetwork connector or a protocol converter. The gateway realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway can also provide filtering and security functions. The message sent by the terminal device 240 to the backend server 210 must be sent to the corresponding backend server 210 through the gateway 220. The message sent by the backend server 210 to the terminal device 240 must also be sent to the corresponding terminal device 240 through the gateway 220.

[0097] The data synchronization method provided in the embodiment of the present application can be executed locally on the terminal device 240 or the background server 210, or it can be executed based on the interaction between the terminal device 240 and the background server 210. The background server 210 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0098] The data synchronization method provided in the embodiment of the present application can be executed in various scenarios, which is exemplarily described below.

[0099] 1. Scenarios of shopping applications

[0100] The method provided in the embodiments of the present application can be applied in the scenario of shopping applications.

[0101] The method in the embodiment of the present application can be applied in the scenario of shopping applications. For example, in shopping applications, it is often necessary to ensure that the data related to the inventory of goods is consistent between multiple servers to avoid overselling; or to ensure that the account information (such as address, payment information) is consistent between multiple servers to prevent order abnormalities.

[0102] The data synchronization method in the embodiment of the present application can be applied to the above-mentioned shopping application. In this way, when the shopping application is used on different terminal devices, the transaction can be better guaranteed to proceed smoothly, which is conducive to improving the shopping experience.

[0103] 2. Streaming media application scenarios

[0104] The method provided in the embodiments of the present application can be applied in the scenario of streaming media application.

[0105] Currently, with the continuous development of information technology, many streaming media applications have emerged. Streaming media is a technology that continuously transmits audio and video content on the network, allowing terminal devices to start playing media files before the data is completely downloaded. Streaming media scenarios usually involve real-time or near real-time content transmission, which can include online video, audio live broadcast, Internet TV, distance education courses, game live broadcast, video conferencing, etc.

[0106] The data synchronization method provided in the embodiment of the present application can be applied in the streaming media scenario. Figure 3 , Figure 3 A schematic diagram of the system architecture of a streaming media service provided in an embodiment of the present application is shown. Figure 3 As shown, streaming content providers can use CDN (Content Distribution Network) to distribute streaming content, that is, cache relevant business data on the nodes of each content distribution network. When people want to obtain relevant streaming resources, the nearest node will respond to the request of the corresponding object and feed back the streaming resources to the relevant terminal device for playback and display.

[0107] In order to ensure real-time synchronous update of streaming media content in the content distribution network, the data synchronization method in the embodiment of the present application can be adopted to improve the efficiency of data transmission and processing as much as possible and ensure the browsing experience of streaming media services.

[0108] General description of the embodiments of the present application

[0109] Before introducing and explaining the data synchronization method provided in the embodiment of the present application, the software and hardware entities involved in the data synchronization method in the embodiment of the present application are first introduced and explained.

[0110] The data synchronization method in the embodiment of the present application can be applied to a data sending end, and the data sending end can perform data synchronization transmission to multiple data receiving ends. Here, multiple data receiving ends can refer to any number of two or more data receiving ends, and the present application does not limit this. In the embodiment of the present application, for each data receiving end, after they receive the target data that needs to be synchronized, they can perform corresponding business processing according to the target data. For example, taking the scenario of e-commerce as an example, there are multiple sellers on the e-commerce platform, and each seller can upload their own product information to the platform system, and the platform system can display the information of each product. When a shopper logs in to the e-commerce platform to shop, the platform system will generate a transaction data after placing an order, and this transaction data will be synchronized to the terminal device used by the shopper and the seller. At this time, the platform system can be used as a data sending end, and the terminal device used by the shopper and the seller can be used as a data receiving end. When the transaction data is synchronized to the terminal device used by the shopper, the order data can be obtained after further processing. Similarly, when the transaction data is synchronized to the terminal device used by the seller, the revenue data can be obtained after further processing. Therefore, for each data receiving end, there is generally a business processing logic for the target data, and the business processing logic adopted by different data receiving ends is often different.

[0111] In the embodiment of the present application, the business processing logic for the target data in the data receiving end can be integrated into the data sending end. That is, the business processing logic of the data receiving end is placed in front, so that the data sending end can also process the target data according to the specific business processing logic of the data receiving end. When the network transmission status of the data receiving end is poor, the data sending end can first process part of the tasks on behalf of the data receiving end, thereby improving the efficiency of data processing.

[0112] For example, please refer to Figure 4 , Figure 4 A schematic diagram showing the relationship between a data sending end and a data receiving end provided in an embodiment of the present application is shown. Figure 4 In the example, for a data transmitting end, there are three corresponding data receiving ends, which are respectively recorded as a first data receiving end, a second data receiving end, and a third data receiving end. The first data receiving end, the second data receiving end, and the third data receiving end can be any device type, and their device types can be the same or different.

[0113] In particular, the underlying systems of the first data receiving end, the second data receiving end and the third data receiving end may be the same or different. When the underlying systems of the first data receiving end, the second data receiving end and the third data receiving end are different, the three belong to heterogeneous systems.

[0114] At the data sending end, the business processing logic of the first data receiving end, the business processing logic of the second data receiving end, and the business processing logic of the third data receiving end are pre-stored. In the embodiment of the present application, the pre-stored business processing logic of the first data receiving end is recorded as the first business processing logic, the pre-stored business processing logic of the second data receiving end is recorded as the second business processing logic, and the pre-stored business processing logic of the third data receiving end is recorded as the third business processing logic.

[0115] It should be noted that in the embodiment of the present application, the business processing logic of the data receiving end pre-stored by the data sending end may be only part of the business processing logic of the data receiving end. For example, after the data receiving end receives the target data, it will perform a business processing flow of multiple steps. In the embodiment of the present application, the business processing logic of the first several steps can be encapsulated and pre-stored in the data sending end. In this way, it can support the data sending end to process the tasks of the data sending end on its behalf, and it will not make the data sending end pre-store too much program code and appear bloated.

[0116] As described above, in the embodiment of the present application, the number of data receiving terminals can be any number of two or more than two, and is not fixed to Figure 4 The situation shown. And it can be understood that for the data sending end, the number of its corresponding data receiving ends can be flexible and variable. For example, in some scenarios, it is possible that a certain (or some) data receiving end no longer needs to receive and process the target data. At this time, these data receiving ends can be removed from the current data synchronization task, and the number of data receiving ends corresponding to the data sending end is reduced; in other scenarios, it is also possible that a certain (or some) terminal does not originally belong to the data receiving end, but currently also needs to synchronously receive and process the target data. At this time, these terminals can be added to the current data synchronization task as data receiving ends corresponding to the data sending end. The number of data receiving ends will increase, and this application does not impose any restrictions on this.

[0117] It should be noted that in the embodiments of the present application, the data transmitting end may be a single device or a distributed system including multiple devices, each of which is dispersedly arranged. The same is true for the data receiving end, and the present application does not limit this.

[0118] Please refer to Figure 5 , Figure 5 A schematic diagram of a data synchronization method provided in an embodiment of the present application is shown. In an embodiment of the present application, Figure 5 The process shown can be applied to the aforementioned data sending end. Figure 5 As shown, a data synchronization method according to an embodiment of the present application includes but is not limited to the following steps:

[0119] Step 510: Obtain target data;

[0120] Step 520: Detect the network transmission status between each data receiving end, and determine the data synchronization mode corresponding to each data receiving end according to the network transmission status; wherein the data synchronization mode includes a real-time synchronization mode and a batch synchronization mode;

[0121] Step 530: If it is determined that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode, the target data is sent to the data receiving end;

[0122] Step 540: If it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode, query the pre-stored business processing logic of the data receiving end, process the target data according to the business processing logic, and send the processed target data to the data receiving end in batches.

[0123] In an embodiment of the present application, a data synchronization method is provided. The method can adjust the data synchronization mode in combination with the network transmission status corresponding to the data receiving end, thereby maximizing the utilization rate of network transmission resources and improving the efficiency of data transmission and processing.

[0124] In step 510, the target data is obtained, and the target data refers to the data that needs to be synchronized at each data receiving end. In the embodiment of the present application, there is no restriction on the data type of the target data, which can be flexibly set according to the specific application. For example, in some embodiments, the target data can be structured data, such as numerical data, including integers, floating point numbers, etc., which can be used to represent information such as age, price or quantity, and text data, i.e., a string of characters, such as name, address or description. In some embodiments, the target data can be unstructured data, such as image data, such as photos, video frames, etc., or audio data, such as voice data, music data, etc. In some embodiments, the target data can also be semi-structured data or multimodal data, which will not be elaborated in this application.

[0125] In the embodiments of the present application, the specific content of the target data can be set according to the actual application scenario of the data synchronization method. Exemplarily, in some embodiments, when the data synchronization method in the present application is applied in the scenario of e-commerce, the target data may include shopper behavior data (such as browsing history, purchase history and search information, etc.), commodity data (such as inventory quantity, supplier information and price data, etc.) or order data (such as order status, delivery information, payment records, etc.). In some embodiments, when the data synchronization method in the present application is applied in the scenario of the financial industry, the target data may include transaction data (such as transaction records of financial products such as stocks, bonds, and foreign exchange), account data (such as identity information, credit score, account balance, etc.), and market data (such as market index, interest rate changes, economic indicators, etc.). Of course, it can be understood that the above is only used for an exemplary introduction to the content of the target data in the embodiments of the present application, and does not mean to limit the actual application of the present application.

[0126] In step 510, when acquiring the target data, the specific acquisition method can be selected according to the actual needs and the location of the target data. In some embodiments, the target data can be acquired by querying a database, for example, by extracting the target data from a database through a specific query statement (such as an SQL statement) or an application programming interface. In some embodiments, the target data can be acquired by reading related files, for example, by acquiring the target data from a log file. In some embodiments, the target data can also be captured from a website through a crawler tool, or the target data can be acquired using a third-party data service, which is not limited in this application.

[0127] In the embodiment of the present application, the action of obtaining the target data can be performed continuously or intermittently. For example, the target data can be continuously obtained from the data source using a real-time two-way communication protocol, or a query can be sent once every period of time to obtain the target data from the data source. Of course, in order to maximize the efficiency of data transmission synchronization, the target data can be obtained by continuous execution and real-time acquisition.

[0128] It should be noted that in the embodiments of the present application, the target data will be synchronously transmitted to multiple data receiving ends, and the underlying systems used by these data receiving ends may be heterogeneous systems, and the data formats supported and processed by the heterogeneous systems may be different. Therefore, in some embodiments, when multiple data receiving ends use heterogeneous systems, the target data can be pre-processed into common data that can be compatible and used by various heterogeneous systems.

[0129] In step 520, the network transmission status between the data receiving end and the data receiving end can be detected, and the data synchronization mode corresponding to the data receiving end can be determined according to the network transmission status. Specifically, in the embodiments of the present application, when detecting the network transmission status between the data receiving end and the data receiving end, there are multiple ways to use. Exemplarily, in some embodiments, relevant commands can be used to query some indicator data of the network transmission between the data receiving end and the data receiving end. For example, the Ping command can be used to view the response time and packet loss rate corresponding to the data receiving end. The lower the response time and packet loss rate, the better the network transmission status; relatively, the higher the response time and packet loss rate, the worse the network transmission status. In some embodiments, a network monitoring tool or a network packet capture tool can be used to capture and analyze the status of the data packet transmitted between the data receiving end and the data receiving end. For example, indicators such as network bandwidth, traffic, throughput in a predetermined time period, and delay can be monitored, and the network transmission status between the data receiving end and the data receiving end can be judged according to these indicators.

[0130] Of course, it can be understood that in the embodiments of the present application, the specific method of judging the network transmission status and the related data indicators are not limited to the above-mentioned situation. The technicians in this field can flexibly select the judgment strategy according to the actual needs, and the present application does not impose any restrictions on this.

[0131] In the embodiment of the present application, when the network transmission state is detected, the detection result can be divided into two categories, for example, the detection result of the network transmission state is set as two categories of "excellent" and "poor". When specifically determining which category the network transmission state belongs to, one or more detected indicator data can be used. Exemplarily, for example, the response time corresponding to the data receiving end can be detected, and a threshold value for judgment can be set. If the response time exceeds the set threshold, the detection result of the network transmission state corresponding to the data receiving end can be determined as a "poor" category; if the response time does not exceed the set threshold, the detection result of the network transmission state corresponding to the data receiving end can be determined as a "good" category. In some embodiments, it can also be judged by multiple indicators, and the judgment logic of each indicator is similar to the judgment logic of the above-mentioned "response time". If most of the indicators (for example, more than half) are determined as the "good" category, then the final detection result of the network transmission state is determined as the "good" category; relatively, if only some indicators (such as less than half) are determined as the "good" category, then the final detection result of the network transmission state is determined as the "poor" category, and this application does not limit this.

[0132] In step 520, the data synchronization mode corresponding to the data receiving end can be further determined according to the determined network transmission status. In the embodiment of the present application, for all data receiving ends, the data synchronization modes that they can use include real-time synchronization mode and batch synchronization mode, wherein the real-time synchronization mode refers to the transmission of target data to the data receiving end immediately after the data sending end obtains the target data. This mode can quickly feedback to the data receiving end when the target data arrives at the data sending end, which can ensure the immediacy of data transmission. The batch synchronization mode means that after the data sending end obtains the target data, it does not directly transmit the target data to the data receiving end, but transmits the target data in batches after a period of time according to a pre-set method. In this mode, after the target data arrives at the data sending end, it may take a while to be synchronized to the data receiving end.

[0133] It is easy to understand that in the embodiment of the present application, in the real-time synchronization mode, the target data is transmitted as it arrives, which usually requires higher network transmission resource support, that is, when the network transmission status between the data sending end and the data receiving end is good, the real-time synchronization mode is used, and the target data can be quickly sent to the data receiving end. If the network transmission status between the data sending end and the data receiving end is poor, even if the real-time synchronization mode is used, after the data sending end sends the target data, it is difficult for the data receiving end to receive it quickly due to the limited network transmission. For the data sending end, there is a waste of network transmission resources. Therefore, in the embodiment of the present application, when determining the data synchronization mode corresponding to the data receiving end, if the network transmission status between the data sending end and the data receiving end is good, for example, the previous network transmission status detection result is determined to be a "good" category, then it can be determined that the corresponding data synchronization mode adopts the real-time synchronization mode.

[0134] On the other hand, if the network transmission status between the data sending end and the data receiving end is poor, for example, if the previous network transmission status detection result is determined to be "poor", even if the real-time synchronization mode is used, real-time synchronization may not be achieved. Instead, the network transmission resources of the data sending end are occupied, resulting in the disadvantage of low utilization of network transmission resources. Therefore, in this case, it can be determined that the data synchronization mode corresponding to the data receiving end adopts the batch synchronization mode. It is easy to understand that compared with the real-time synchronization mode, the batch synchronization mode has a lower demand for network transmission resources and a higher utilization rate. It can synchronize data through batch transmission, which is more suitable for data receiving ends with poor network transmission status.

[0135] In step 530, if it is determined that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode, the target data can be directly sent to the data receiving end.

[0136] Specifically, in an embodiment of the present application, the data transmission protocol adopted by the data receiving end can be determined in advance based on the device type of the data receiving end, and then when the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode, the corresponding data transmission protocol can be directly used to send the target data to the data receiving end.

[0137] In step 540, if it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode, the pre-stored business processing logic of the data receiving end can be queried, the target data is processed by the business processing logic, and then the processed target data is cached. When the set sending conditions are met, the processed target data is sent to the data receiving end in batches.

[0138] As described above, in an embodiment of the present application, the business processing logic of each data receiving end is integrated in the data sending end, and the business processing logic can correspond to the data receiving end one by one. In an embodiment of the present application, unique identification information can be pre-assigned to each data receiving end, recorded as a system identifier, and the system identifier corresponding to each data receiving end is different, and the system identifier can include at least one of numbers or characters. When pre-storing the business processing logic, the business processing logic and the system identifier corresponding to each data receiving end can be bound. In this way, when performing data synchronization, if the data synchronization mode corresponding to a certain data receiving end is a batch synchronization mode, the corresponding business processing logic can be determined based on the system identifier corresponding to the data receiving end. Then, the target data is processed using the business processing logic.

[0139] In step 540, the processed target data is sent to the data receiving end in batches. Here, the processed target data can be sent periodically, and the period of sending the processed target data to the data receiving end can be set according to actual needs. For example, in some embodiments, a fixed sending period can be set, such as sending once every 5 seconds. In the embodiments of the present application, there is no restriction on the length of the sending period. In some embodiments, the sending period can also be associated with the network transmission state corresponding to the data receiving end. For example, when the network transmission state corresponding to the data receiving end is in a poor situation, the corresponding data synchronization mode is a batch synchronization mode, and the corresponding sending period can be further determined according to the index data related to the network transmission state. The implementation principle is: the worse the network transmission state corresponding to the data receiving end, the longer the sending period can be. The specific functional relationship between the index data related to the network transmission state and the sending period is not limited by the present application. In some embodiments, the processed target data can also be sent to the data receiving end when the processed target data reaches a certain amount of data.

[0140] It should be pointed out that when executing the data synchronization method in the embodiment of the present application, step 520 can be executed continuously and in real time, that is, for each data receiving end, the corresponding network transmission state can be continuously monitored, so as to realize the switching of the data synchronization mode with the network transmission state. For example, in some cases, a data receiving end may be originally in the real-time synchronization mode, but the current monitoring of its network transmission state has changed, from a better situation to a worse situation. At this time, the data transmission mode corresponding to the data receiving end can be switched to the batch synchronization mode. In other cases, a data receiving end may be originally in the batch synchronization mode, but the current monitoring of its network transmission state has changed, from a worse situation to a better situation. At this time, the data transmission mode corresponding to the data receiving end can be switched to the real-time synchronization mode. In this way, it can better adapt to the changes in the network transmission state of each data receiving end, which is conducive to improving the efficiency of data synchronization transmission and the utilization rate of network transmission resources. Of course, in the embodiment of the present application, step 520 can also be executed in intervals, for example, the network transmission state corresponding to each data receiving end is detected once every period of time, so as to update their corresponding data synchronization mode, and the present application does not limit this.

[0141] In particular, in an embodiment of the present application, for the process of switching the data transmission mode corresponding to the data receiving end from the real-time synchronization mode to the batch synchronization mode, in certain scenarios, the data receiving end may fluctuate in a short period of time, causing the network transmission state to be temporarily in a poor condition. This situation has a small impact on real-time synchronous transmission and does not need to be switched. Therefore, in an embodiment of the present application, if the network transmission state of the data receiving end deteriorates from a better situation to a worse situation, the duration after the data receiving end changes to a poor network transmission state can be monitored. If the duration reaches a certain duration threshold, the data transmission mode corresponding to the data receiving end is switched to the batch synchronization mode, otherwise the real-time synchronization mode can be maintained.

[0142] Furthermore, in an embodiment of the present application, when switching the data transmission mode corresponding to the data receiving end, in order to prevent data disorder, the switching can be performed after the current synchronization task of the previous data transmission mode is completed.

[0143] It can be understood that the data synchronization method provided in the embodiment of the present application can be applied to a data sending end, which is used to synchronously transmit data to multiple data receiving ends, and the business processing logic of each data receiving end is pre-stored. The method obtains the target data, and for each data receiving end, detects the network transmission status between it, so as to determine whether the data receiving end uses the real-time synchronization mode or the batch synchronization mode. For the data receiving end using the real-time synchronization mode, the target data is directly sent to improve the efficiency of transmitting the target data to the data receiving end; for the data receiving end using the batch synchronization mode, the target data is pre-processed by the pre-stored business processing logic corresponding to the data receiving end, and then the pre-processed target data is sent to the data receiving end in batches, so as to improve the processing efficiency of the target data. The present application can adjust the data synchronization mode in combination with the network transmission status corresponding to the data receiving end, so as to maximize the utilization rate of network transmission resources and improve the efficiency of data transmission and processing.

[0144] Specifically, in some embodiments, after acquiring the target data, the method in the embodiment of the present application further includes:

[0145] According to the data structure of the linked list, the target data is converted into multiple log data;

[0146] According to the data sorting in the target data, the log index corresponding to each log data is determined.

[0147] In the embodiment of the present application, the target data may include more data content. In order to facilitate the orderly and complete transmission of the target data to the data receiving end, in the embodiment of the present application, the target data can be converted into log data according to the data structure of the linked list. It is understandable that the target data may include multiple business-related records. For example, in some embodiments, the target data may be data recording interactive operations, each of which may include the following fields: identification, timestamp, operation account, operation type, detailed information, etc. Each record can be converted into a corresponding log data. The log data includes all the contents of the record, which can be a simple text line data. In this way, the target data can be split and converted into multiple log data, and the specific number of log data corresponding to it is not limited by this application.

[0148] In an embodiment of the present application, when converting the target data into log data, the conversion can be performed according to the data structure of a linked list. A linked list is a linear data structure in which each element (node) contains a link to the next element in the list, and each element generally includes two parts: a data field and a pointer field. For the application of the present application, the data field stores specific log data, while the pointer field points to the next log data. Moreover, when organizing the log data according to the data structure of the linked list, the log index corresponding to each log data can be determined according to the data sorting corresponding to the log data recorded in the target data. In an embodiment of the present application, the log index is the unique identification information assigned to each log data, and different log data have different corresponding log indexes. Exemplarily, the log index may include at least one of a number or a character.

[0149] It can be understood that in the embodiments of the present application, when the log index corresponding to each log data is determined according to the data sorting in the target data, in some embodiments, for the case where there is a timestamp for each data record in the target data, the corresponding log data can be sorted according to the timestamp, and then the log index of each log data can be determined in an ascending manner.

[0150] For example, please refer to Figure 6 , Figure 6 A schematic diagram of converting target data into log data provided in an embodiment of the present application is shown. In an embodiment of the present application, for the target data, the log data obtained by the conversion can be arranged according to the original order of the data content in the target data, and then a log index can be assigned to each log data. Figure 6 As shown, the log index corresponding to the log data can be numbered with numbers, starting from 1 and increasing in sequence, for example Figure 6 The log indexes corresponding to the log data in are 1, 2, 3...N (N is a positive integer). Since the amount of target data may be large, the log indexes corresponding to the log data can be sorted one by one according to the above numbers, and this application does not limit the specific values ​​of the log indexes.

[0151] It can be understood that in the embodiments of the present application, by converting the target data into log data, the data can be conveniently indexed and marked according to the data sorting. When it is subsequently transmitted to the data receiving end, the data sorting can be determined based on the log index, so that the target data can be synchronized to the data receiving end in an orderly and complete manner for processing.

[0152] Specifically, in some embodiments, sending target data to a data receiving end includes:

[0153] Get the system identifier of the data receiving end;

[0154] According to the log index, a number of log data are extracted as the data to be synchronized in the current batch, and the first log index corresponding to the log data at the end position in the data to be synchronized is queried;

[0155] The system identifier, the data to be synchronized and the first log index are sent to the data receiving end.

[0156] In the embodiments of the present application, when sending target data to the data receiving end, in some embodiments, real-time synchronous transmission can be performed in small batches. Specifically, some data to be synchronized as the current batch can be extracted from the log data according to the log index. Here, when extracting the data to be synchronized, in order to ensure that the data can be transmitted to the data receiving end in a complete and orderly manner, the log data can be extracted according to the continuous log index.

[0157] For example, please refer to Figure 7 , Figure 7 FIG. 1 shows a schematic diagram of extracting data to be synchronized provided in an embodiment of the present application. Figure 7 As shown, the log index corresponding to the log data obtained by converting the target data includes some log indexes with values ​​of 187-192. Assuming that you need to extract the data to be synchronized, you can first determine the log index corresponding to the first log data to be extracted, for example, start extracting from the log data with the corresponding log index of 188, and then continue to extract the log data with the corresponding log index of 189 and the log data with the corresponding log index of 190, until the number of log data corresponding to the current batch is reached, and use these extracted log data as the data to be synchronized for the current batch. Figure 7 As shown, assuming that 4 log data are extracted in the current batch to constitute the data to be synchronized, the extracted log data are log data corresponding to log indexes 188, 189, 190 and 191 respectively.

[0158] In the embodiment of the present application, after extracting the data to be synchronized, the log index corresponding to the log data at the end of the data to be synchronized can be queried. In the embodiment of the present application, it is recorded as the first log index. Figure 7 , Figure 7 Among the data to be synchronized in the current batch extracted, the log index corresponding to the log data at the end is 191. Therefore, the first log index corresponding to the current batch is 191.

[0159] Furthermore, in an embodiment of the present application, a system identifier of the data receiving end (ie, the identification information of the aforementioned data receiving end) may also be obtained, and then the system identifier, the data to be synchronized, and the first log index may be sent to the data receiving end.

[0160] For the data receiving end, after receiving the system identifier, the data to be synchronized and the first log index, it can perform data synchronization based on these data. Specifically, the system identifier can be used by the data receiving end to verify whether the received data is applicable to the data receiving end. It can be matched with the local system identifier. If the two are consistent, then the processing can continue; if the two are inconsistent, it means that the data receiving end has received the data that was sent incorrectly, and at this time, the synchronization of the currently received data to be synchronized can be refused.

[0161] For the data receiving end, during the data synchronization process, a log index can also be recorded in real time. The log index is the log index corresponding to the log data at the end position in a batch of data to be synchronized that the data receiving end completes synchronization. In the embodiment of the present application, it is recorded as the second log index. The second log index can be updated according to the data to be synchronized in the batch every time a batch of data to be synchronized is synchronized. And the second log index after each update can be fed back to the data sending end to facilitate the data sending end to understand the data synchronization process of the data receiving end. In the embodiment of the present application, for the data receiving end, after receiving the system identifier, the data to be synchronized and the first log index, it can also compare the first log index and the second log index. If the first log index is greater than the second log index, it means that the data to be synchronized currently received has not completed synchronization, and the data to be synchronized can be used normally. Relatively speaking, if the first log index is less than or equal to the second log index, it means that the data to be synchronized currently received has completed synchronization before, and at this time, the synchronization of the data to be synchronized currently received can be refused.

[0162] Specifically, in some embodiments, according to the log index, a number of log data are extracted as the data to be synchronized in the current batch, including:

[0163] Query the second log index of the most recent feedback from the data receiving end; wherein the second log index is used to represent the log index corresponding to the log data at the end position in a batch of data to be synchronized that has been synchronized by the data receiving end;

[0164] Starting from the next log data of the log data corresponding to the second log index, the first number of log data is extracted as the data to be synchronized in the current batch.

[0165] As described above, in an embodiment of the present application, the data receiving end can record the second log index in real time during the data synchronization process. The second log index is the log index corresponding to the log data at the end position in a batch of data to be synchronized that the data receiving end has completed synchronization. The log index can be used to record the data synchronization process of the data receiving end. After the data receiving end completes the synchronization task of a batch of data to be synchronized, it can update the currently recorded second log index and send it to the data receiving end. In this way, it is also convenient for the data receiving end to determine each batch of data to be synchronized.

[0166] Specifically, in the embodiment of the present application, when the data receiving end extracts the data to be synchronized in the current batch, the second log index fed back by the data receiving end most recently can be queried, so as to understand which log data the data receiving end has recently synchronized. Then, starting from the next log data of the log data corresponding to the second log index fed back most recently, the first number of log data can be extracted as the data to be synchronized in the current batch.

[0167] Exemplarily, it is assumed that the log index corresponding to the log data obtained by the target data conversion includes some log indexes with values ​​of 187-220. When the data sending end determines the current batch of data to be synchronized corresponding to a certain data receiving end, it is found that the second log index of the most recent feedback from the data receiving end is 196. Then, starting from the log data with the corresponding log index of 197, the log data of the first number can be extracted as the current batch of data to be synchronized corresponding to the data receiving end. Here, the first number can be a pre-set fixed value, for example, it can be set to 4, then the extracted data to be synchronized includes log data with corresponding log indexes of 197, 198, 199, and 200. Of course, the above is only used to exemplify the principle of extracting data to be synchronized in the embodiment of the present application, and does not mean to limit its specific implementation.

[0168] It should be noted that, in the embodiment of the present application, for each data receiving end, their respective synchronization efficiencies may differ, and therefore, the second log index fed back by each data receiving end may be different. In the embodiment of the present application, the second log index fed back by each data receiving end may be recorded and maintained at the data receiving end. When the second log index fed back by the data receiving end is received, the previous second log index is updated according to its corresponding system identifier; when the data to be synchronized is sent to the data receiving end, it is also extracted according to its corresponding second log index.

[0169] It can be understood that in the embodiment of the present application, extracting the data to be synchronized through the second log index can better combine the synchronization process of the data receiving end to synchronize the data, which can reduce the probability of repeated data transmission, improve the utilization rate of network transmission resources, and help improve the efficiency of data transmission and processing.

[0170] Specifically, in some embodiments, the first number is determined by the following steps:

[0171] Detecting a second number of data receiving terminals that are performing data synchronization at the current time node, and a third number of data synchronization requests received in a first preset time period before the current time node;

[0172] According to the second number and the third number, determine the first number used when extracting the data to be synchronized in the current batch;

[0173] Among them, the first number is negatively correlated with the second number, and the first number is negatively correlated with the third number.

[0174] In the embodiments of the present application, when extracting the first number of log data as the data to be synchronized for the current batch, as described above, in some embodiments, the first number can be a pre-set fixed value, and the present application does not limit its size; in other embodiments, the first number can also be determined based on the pressure of the current data sending end in performing the data synchronization task. It is easy to understand that if the current data sending end is under less pressure to perform the data synchronization task, then the data to be synchronized can be transmitted in each batch less, and the number of transmissions can be more, which can facilitate the rapid synchronization of the target data to the data receiving end; relatively, if the current data sending end is under greater pressure to perform the data synchronization task, then the data to be synchronized can be transmitted in each batch more, and the number of transmissions can be less, which can alleviate the data sending pressure of the data sending end.

[0175] Specifically, in an embodiment of the present application, when evaluating the pressure of the data synchronization task of the data sending end, the number of data receiving ends that are performing data synchronization at the current time node can be detected, and recorded as the second number. The second number can be used to measure the pressure at the current time node; at the same time, in an embodiment of the present application, the pressure of the data synchronization task in the recent period of time can also be considered. For example, the number of data synchronization requests received within the first preset time period before the current time node can be detected and recorded as the third number. The data synchronization request here can be transmitted from an upstream data source. Whenever new target data is sent to the data sending end, a data synchronization request will be triggered to enable the data sending end to perform the data synchronization task for the target data. For the length of the first preset time period set in the embodiment of the present application, its length can be determined according to actual needs, and this application does not impose any restrictions.

[0176] In the embodiment of the present application, the larger the second number is, the more data receiving ends the current data sending end needs to connect to, and the greater the pressure of the data synchronization task; similarly, the larger the third number is, the more target data the data sending end needs to send in the recent period of time, and similarly, the greater the pressure of the data synchronization task. Therefore, in the embodiment of the present application, the size of the first number can be determined according to the second number and the third number, and the implementation principle is: the larger the second number and the third number are, the smaller the determined first number is; the smaller the second number and the third number are, the larger the determined first number is. That is, the first number is negatively correlated with the second number, and is also negatively correlated with the third number. The first number can be 1 or any integer greater than 1. This application does not limit the specific functional relationship between them.

[0177] It can be understood that in the embodiment of the present application, the first number is flexibly determined based on the second number and the third number, so that the data to be synchronized in each batch transmission can be easily adjusted, and the efficiency of data transmission can be improved when the data synchronization task pressure at the data sending end is relatively small; when the data synchronization task pressure at the data sending end is relatively large, the frequency of transmission can be appropriately reduced, thereby alleviating the data sending pressure at the data sending end.

[0178] Specifically, in some embodiments, the method in the embodiments of the present application further includes:

[0179] Record the first time node when the current batch of data to be synchronized is sent to the data receiving end;

[0180] If a second log index corresponding to the current batch fed back by the data receiving end is received within a second preset period after the first time node, extract the next batch of data to be synchronized according to the second log index corresponding to the current batch;

[0181] If the second log index corresponding to the current batch fed back by the data receiving end is not received within a second preset time period after the first time node, the data to be synchronized in the current batch is resent to the data receiving end.

[0182] As described above, in an embodiment of the present application, the data receiving end can record a second log index in real time during data synchronization. The second log index is the log index corresponding to the log data at the end position in a batch of data to be synchronized that the data receiving end has completed synchronization. The log index can be used to record the data synchronization process of the data receiving end.

[0183] In an embodiment of the present application, for each batch of data to be synchronized, after the data transmitting end sends the data to the data receiving end, the time node of the current batch of data to be synchronized can be recorded, and in an embodiment of the present application, it is recorded as the first time node. Then, it is possible to monitor in real time whether the second log index corresponding to the current batch is received. Here, when monitoring the reception of the second log index corresponding to the current batch, the received second log index can be matched with the first log index of the current batch. If it is found that the second log index that is the same as the first log index of the current batch is received, it means that the second log index corresponding to the current batch is received.

[0184] In an embodiment of the present application, after the current batch of data to be synchronized is sent to the data receiving end, under normal circumstances, the data receiving end will quickly receive and complete the data synchronization of the current batch of data to be synchronized. Therefore, the data sending end can receive the second log index corresponding to the current batch in a short time. If the data sending end has not received the second log index corresponding to the current batch, it may be that a problem occurred during the data transmission process, and the current batch of data to be synchronized was not sent to the data receiving end normally, or it may be that a fault occurred at the data receiving end and the data synchronization task could not be completed normally. In an embodiment of the present application, a preset time period can be set, recorded as the second preset time period, and the time length of the second preset time period can be slightly shorter. This application does not limit its specific time length.

[0185] For the set second preset time period, in the embodiment of the present application, the duration can be accumulated starting from the first time node. If the data sending end receives the second log index corresponding to the current batch after the first time node before the accumulated duration reaches the second preset time period, it means that the data receiving end has normally received and completed the data synchronization of the current batch of data to be synchronized. At this time, the next batch of data to be synchronized can be continuously extracted according to the second log index of the current batch. The specific extraction logic has been introduced in the above embodiment and will not be repeated here. Relatively speaking, if the accumulated duration has reached the second preset time period, but the second log index corresponding to the current batch has not been received, it means that there may be problems with the data transmission or data synchronization of the current batch of data to be synchronized. At this time, the current batch of data to be synchronized can be resent to the data receiving end to ensure the reliability and stability of data transmission.

[0186] It should be noted that in the embodiment of the present application, in the real-time transmission mode, the upper limit of the number of retransmissions can be pre-set. For each batch of data to be synchronized, if the corresponding second log index is not received after multiple retransmissions, it means that the network transmission status between the data receiving end and the data receiving end may be abnormal, or the data receiving end has a fault. At this time, in order to reduce the waste of network transmission resources, the data synchronization mode corresponding to the data receiving end can be adjusted to the batch synchronization mode. In the embodiment of the present application, the upper limit of the number of retransmissions can be flexibly set according to actual needs.

[0187] Specifically, in some embodiments, processing the target data according to the business processing logic and sending the processed target data in batches to the data receiving end includes:

[0188] Get the system identifier of the data receiving end;

[0189] Process the log data according to the business processing logic to obtain the processing data corresponding to the log data;

[0190] When the amount of newly processed data reaches a preset capacity threshold, or the time since the last time the processed data was sent reaches a preset time threshold, query the third log index corresponding to the last log data of the current preprocessing;

[0191] The system identifier, the processing data and the third log index are sent to the data receiving end.

[0192] In the embodiment of the present application, for the batch synchronization mode, the target data can also be transmitted in the form of log data. Specifically, similar to the previous implementation, in the embodiment of the present application, in the batch synchronization mode, the system identifier of the data receiving end can also be obtained, so that the data receiving end can verify the received data before synchronization, thereby improving the reliability of data synchronization.

[0193] When sending the processed target data to the data receiving end, for each log data, the business processing logic of the data receiving end can be used to process it, obtain the corresponding processed data and cache it. Then, the newly added processed data can be transmitted in a regular or quantitative manner. For example, a time threshold can be set in advance. If the time from the last time the processed data is sent reaches the preset time threshold, the processed data can be sent to the data receiving end. In other words, a capacity threshold can be set in advance. When the amount of newly added (relative to the last time it was sent) processed data reaches the preset capacity threshold, the processed data can also be sent to the data receiving end.

[0194] It should be noted that, in an embodiment of the present application, when sending processed data, the third log index corresponding to the last log data of the current preprocessing can be queried, and the processed data, the system identifier and the third log index can be sent together to the data receiving end. Similarly, in an embodiment of the present application, for the batch transmission mode, the data receiving end can also maintain an index, recorded as the fourth log index, and the fourth log index can represent the log index corresponding to the processed data at the end position in a batch of processed data that the data receiving end completes synchronization. In addition, the fourth log index can be fed back to the data sending end. In this way, the data sending end can determine which processed data to send (which can be all unsynchronized processed data) based on the fourth log index fed back by the data receiving end. In addition, at the data receiving end, the relationship between the third log index and the fourth log index can also be compared. If it is found that the third log index is less than or equal to the fourth log index, it means that it has been synchronized, and the currently received processed data can be synchronized without using it.

[0195] In the embodiments of the present application, when sending processed data, in some embodiments, the processed data can be packaged into snapshot data of a specific format to improve the transmission efficiency of the data. It can be understood that in the embodiments of the present application, the processed data is generated by processing in the order of the log data, so the obtained processed data has integrity, which can be convenient for the data receiving end to use, and is conducive to improving the synchronization and processing efficiency of the data.

[0196] Specifically, in some embodiments, detecting the network transmission status between each data receiving end and determining the data synchronization mode corresponding to each data receiving end according to the network transmission status includes:

[0197] Detect network environment parameters between the data receiving end and the data receiving end;

[0198] Determine the network quality score corresponding to the data receiving end according to the network environment parameters;

[0199] If the network quality score is greater than a preset score threshold, determining that the data synchronization mode corresponding to the data receiving end is a real-time synchronization mode;

[0200] If the network quality score is less than or equal to a preset score threshold, it is determined that the data synchronization mode corresponding to the data receiving end is a batch synchronization mode.

[0201] In the embodiment of the present application, when the data synchronization mode corresponding to the data receiving end is determined according to the network transmission state between the data receiving end and the data receiving end, in some embodiments, a variety of indicator data can be used in combination. Specifically, in the embodiment of the present application, the network environment parameters between the data receiving end and the data sending end can be detected. Exemplarily, here, the network environment parameters can include but are not limited to the retransmission rate retran_ratio of the session connection between the data sending end and the data receiving end, the number of timeout retransmission data packets retransmit_timeout_packets, the number of TLP (Transaction Layer Packet, Transaction Layer Packet) retransmission data packets, the round-trip time (Round-Trip Time, RTT) of the network transmission, the maximum window max_cwnd, the packet loss rate loss_rate, and the throughput goodput and other parameters, wherein the round-trip time can include the minimum round-trip time min_rtt, the smooth round-trip time srtt, the maximum round-trip time max_rtt and other types. These network environment parameters can be determined according to the value at the current time node, or the value of the predetermined time period taken from the current time node forward, and the present application does not limit this.

[0202] According to the network environment parameters, the network quality score corresponding to the data receiving end can be determined. It can be understood that the size of the network quality score can be positively correlated with the quality of the network environment of the session connection between the data sending end and the data receiving end. If the network environment of the session connection is highly volatile and the transmission effect is poor, the network quality score corresponding to the data receiving end can be determined to be a relatively low value. If the network environment of the session connection is less volatile and the transmission effect is better, the network quality score corresponding to the data receiving end can be determined to be a relatively high value. In the embodiment of the present application, there is no limitation on the specific functional relationship between the network environment parameters and the network quality score. Specifically, a variety of different types of parameters can be combined to weightedly determine the network quality score corresponding to the data receiving end.

[0203] After obtaining the network quality score corresponding to the data receiving end, the data synchronization mode adopted by the data receiving end can be determined according to the network quality score. For example, a score threshold can be set in advance. If the network quality score is greater than the preset score threshold, it means that the network environment is good, and the data synchronization mode corresponding to the data receiving end can be determined to be the real-time synchronization mode. Conversely, if the network quality score is less than or equal to the preset score threshold, it means that the network environment is poor, and the data synchronization mode corresponding to the data receiving end can be determined to be the batch synchronization mode.

[0204] It can be understood that in the embodiment of the present application, the network environment parameters between the data sending end and the data receiving end are comprehensively considered to determine the data synchronization mode corresponding to the data receiving end, which can improve the utilization rate of network transmission resources and more efficiently utilize the limited data transmission capacity of the data sending end. The data synchronization mode is more applicable to the data receiving end.

[0205] Specifically, in some embodiments, the data sending end is a distributed source data system, which includes a master node and multiple slave nodes, and the master node is used to synchronously transmit data to multiple data receiving ends;

[0206] Get target data, including:

[0207] Get the original data through the master node;

[0208] The original data is converted through the master node to obtain standard data in a universal data exchange format;

[0209] The master node sends the standard data to each slave node, and the slave nodes reach a consensus on the standard data, and the consensus-based standard data is determined as the target data.

[0210] In an embodiment of the present application, in some scenarios, the data sending end may adopt a distributed source data system, which may include multiple nodes, specifically a master node and multiple slave nodes, wherein the master node can be used to synchronously transmit data to multiple data receiving ends, and the slave node can be used as a backup node. When the master node fails, it can switch to the slave node to run, thereby improving the reliability and stability of data synchronization.

[0211] For the above data sending end, the distributed source data system is used, and the slave node needs to back up the relevant data. For example, for the target data, it can first obtain the original data through the master node, and the data type of the original data can be arbitrary. Then, the master node can convert the format of the original data to obtain standard data in a universal data exchange format. The standard data can be applied to various systems to ensure the consistency and readability of the data.

[0212] In the embodiment of the present application, after obtaining the standard data, the standard data can be distributed to each slave node for backup and consensus. When the slave nodes reach a consensus on the standard data, the standard data can be used as target data and synchronously transmitted to each data receiving end.

[0213] Specifically, in some embodiments, the master node sends the standard data to each slave node, and the slave nodes reach a consensus on the standard data, including:

[0214] The master node performs hash processing on the standard data to obtain a first hash value;

[0215] Sending the standard data and the first hash value to each slave node through the master node;

[0216] The second hash value is obtained by performing hash processing on the standard data from the node, and the consistency of the first hash value and the second hash value is verified to obtain a consistency verification result corresponding to the standard data;

[0217] When the consistency check results of more than half of the slave nodes pass, the standard data is determined to complete the consensus.

[0218] In the embodiment of the present application, when consensus is reached on the standard data, the master node can perform hash processing on the standard data, calculate a hash value, and record it as the first hash value. Then, the master node can send the standard data and the first hash value to each slave node, and the slave node can serve as a consensus node to verify the consistency of the standard data.

[0219] Specifically, when the slave node verifies the consistency of the standard data, the standard data can also be hashed to calculate a hash value, which is recorded as the second hash value. Then, the first hash value and the second hash value can be compared. If they are the same, the consistency verification is passed, and the slave node generates and feeds back the corresponding consensus signature. The master node can receive and accumulate the number of consensus signatures returned by each slave node. When the total number reaches a certain amount (generally more than half of the number of slave nodes), it can be determined that the consensus of the standard data is completed. In the embodiment of the present application, there is no restriction on the type of hash algorithm used.

[0220] It is understandable that in the embodiments of the present application, by processing data synchronization tasks through a distributed source data system, data can be backed up through a consensus mechanism to prevent single point failures, thereby improving the reliability and stability of data synchronization.

[0221] Specifically, in some embodiments, the data sending end is a distributed source data system, the distributed source data system includes a master node and multiple slave nodes, the master node is used to synchronously transmit data to multiple data receiving ends; the method in the embodiment of the present application also includes:

[0222] Synchronize the data in the master node to each slave node;

[0223] Detect the operating status of distributed source data systems;

[0224] When it is determined that the distributed source data system needs to switch the master node, a new master node is determined from the slave nodes;

[0225] Detect whether there are unfinished data synchronization tasks in the old master node; wherein the unfinished data synchronization tasks are data synchronization tasks for which the data to be synchronized has been sent but the second log index corresponding to the feedback from the data receiving end has not been received;

[0226] If there are unfinished data synchronization tasks in the old master node, stop sending new data to be synchronized through the old master node until the unfinished data synchronization task is completed or times out, and then switch to the new master node for data synchronization transmission.

[0227] In the embodiment of the present application, the data sending end adopts the implementation mode of the distributed source data system, and the data in the master node can be synchronized to each slave node, such as the target data (i.e. the standard data after the above consensus), the log index fed back by the data receiving end, etc. During the execution of the data synchronization task, the running status of the distributed source data system can be detected. If it is found that the current master node fails, or the master node needs to be switched due to factors such as load balancing, one can be selected from the slave nodes as the new master node.

[0228] It should be noted that since the master node switching of the distributed source data system may occur at any time, in order to better coordinate the data synchronization tasks during the switching process, in the embodiment of the present application, after determining the new master node, if the old master node can operate normally, it is possible to further detect whether there are any completed data synchronization tasks in the old master node. Here, taking the real-time transmission mode as an example, the unfinished data synchronization tasks may include the data synchronization tasks for which the data to be synchronized has been sent but the second log index corresponding to the feedback from the data receiving end has not been received (for the batch transmission mode, the unfinished data synchronization tasks may include the data synchronization tasks for which the processed data has been sent but the fourth log index corresponding to the feedback from the data receiving end has not been received). If it exists, the master node may not be switched temporarily, but the sending of new data to be synchronized (and processing data) through the old master node may be stopped. Until all unfinished data synchronization tasks in the old master node are completed or timed out, the switching action may be performed. After the switching, the new master node may be used normally for data synchronization transmission.

[0229] In this way, it is possible to avoid the situation where the new and old master nodes send data to be synchronized (and process data) at the same time, which can reduce the probability of data transmission confusion and improve the utilization of network transmission resources. Of course, it is understandable that if the old master node cannot operate normally due to factors such as failure, or it has no unfinished data synchronization tasks, it can directly switch to the new master node for data synchronization transmission.

[0230] Below, in conjunction with specific application examples, a data synchronization method provided in an embodiment of the present application is introduced and explained.

[0231] The data synchronization method in the embodiment of the present application can be applied to a distributed source data system built based on the Raft algorithm. In the embodiment of the present application, the distributed source data system can be recorded as an RSS system (Raft Source System).

[0232] For example, please refer to Figure 8 , Figure 8 The following is a schematic diagram showing the application principle of an RSS system provided in an embodiment of the present application. In an embodiment of the present application, the distributed source data system may include a master node and several backup nodes (i.e., slave nodes). Figure 8 The number of standby nodes in the system is two, and it can also be any number of one or more than two, which is not limited by the present application. The Raft protocol can be run between the master node and the standby node. The master node mainly receives and synchronizes the load data, while the standby node can reach consensus and back up the data, and switch to be used as the new master node when appropriate.

[0233] Figure 8 In the embodiment, the distributed source data system serves as a data sending end, and its corresponding data receiving end may be some downstream systems. In the embodiment of the present application, these systems may be recorded as target systems. Figure 8 There are three data receiving ends, namely, target system A, target system B and target system C. In the embodiment of the present application, for each data receiving end, different synchronization modes are provided according to the network transmission status between them and the distributed source data system, namely, real-time synchronization mode and batch synchronization mode.

[0234] After receiving the data sent by the distributed source data system, each target system will check whether the data is legal, for example, whether the system identifier matches itself and whether the push progress is consistent with the current synchronization process. If it is legal, the data will be received and processed.

[0235] For example, taking the real-time synchronization mode as an example, when the data of the upstream system is transmitted to the distributed source data system, first, the data transmitted by the upstream system will be received by the master node. The master node can convert the format to obtain standard data, and then send the standard data to each standby node for consensus. Based on the Raft protocol, it can ensure that more than half of the standby nodes put the standard data into their own logs. The data is stored in the master node and the standby node in log format. After the standard data consensus is reached, it can be used as the target data for real-time synchronization mode transmission and sent to the target system. The target system can parse the target data into data of the type defined by its own system, and then process it. Figure 8 In this example, target system A and target system B use real-time synchronization mode.

[0236] In batch synchronization mode, the business processing logic corresponding to each target system can be encapsulated as an execution script and injected into the master node and standby node of the distributed source data system. The data sent by the upstream system can be directly processed by the business processing logic on the master node without converting it into standard data. The data processed by the master node can be packaged and stored in a scheduled or quantitative manner and sent to the target system as snapshot data. Figure 8 In the example, target system C uses batch synchronization mode.

[0237] In an embodiment of the present application, the target system can reply to the synchronization progress of the distributed source data system itself. As in the aforementioned embodiment, the second log index and the fourth log index are used to inform the current synchronization process. In an embodiment of the present application, the distributed source data system can maintain a synchronization progress table to store these indexes. Once feedback from a target system is received, the synchronization progress table will be updated and the information will be sent to the backup node. The synchronization progress of each target system is independent, and in the distributed source data system, all data after the minimum progress can be retained.

[0238] Reference Fig. 9 , Fig. 9 A schematic diagram of various data structures transmitted in an embodiment of the present application is shown. In the embodiment of the present application, the data structure transmitted in the real-time synchronization mode is recorded as a synchronization data structure, the data structure transmitted in the batch synchronization mode is recorded as a snapshot structure, and the data structure fed back by the target system to the distributed source data system is recorded as a reply data structure. This is introduced one by one below.

[0239] Reference Fig. 9 In real-time synchronization mode, each time the master node in the distributed source data system sends data to be synchronized to the target system, its overall data structure may include a target system identifier (Target-ID), a general data set (GDEF-Set), a log index (Last-Index), and a source node identifier (From-RSS-ID). Among them, the target system identifier can be used by the target system to query whether the sent data matches the local identifier. If it does not match, the data can be rejected. The general data set (i.e., the data to be synchronized) can be extracted and merged from the log data, which can be used for data synchronization of the target system. The log index (i.e., the first log index) is used to identify the index of the log data at the end of the general data set, which can represent the push progress of the general data set. After receiving the data, the target system can check whether it is greater than the log index in the last batch of data sent. If not, the data can be rejected, and the distributed source data system can be replied with the largest log index ever received. The source node identifier can identify the node that sends data in the distributed source data system.

[0240] In batch synchronization mode, each time the master node in the distributed source data system sends snapshot data (i.e., processed data) to the target system, its overall data structure may include a target system identifier (Target-ID), snapshot data (Snap), a log execution index (Exec-Index), and a source node identifier (From-RSS-ID). The meanings of the target system identifier and the source node identifier are the same as those mentioned above and are not repeated here. Snapshot data is generated by processing data, and the log execution index (i.e., the third log index) is used to identify the index corresponding to the processed data at the end of the snapshot data, which can represent the push progress of the snapshot data.

[0241] The reply data structure fed back by the target system to the distributed source data system includes the master node identifier (To-RSS-ID) and the synchronized progress (Finish-Index). The master node identifier is used to reply to the specified node. If multiple data to be synchronized or snapshot data are received, multiple replies are required. The master node identifier may change when the master node is switched. The synchronized progress is used to reply to the index corresponding to the currently synchronized log data or processed data. When the target system uses the real-time synchronization mode, the index of the synchronized log data (i.e., the second log index) is fed back. When the target system uses the batch synchronization mode, the index corresponding to the synchronized processed data (i.e., the fourth log index) is fed back. For the distributed source data system, when it receives the synchronized progress, it can use the largest index as the updated synchronized progress.

[0242] Reference Fig.10 , Fig.10 FIG. 1 shows a schematic diagram of a principle of data synchronization provided in an embodiment of the present application. Fig.10 As shown, after the data is sent to the distributed source data system through the upstream system, the corresponding log data will be generated. The log data is a linked list structure that is continuously appended at the end. From the starting index position, it continues to extend backward. In the embodiment of the present application, a consensus log index (Commit-Index) can be marked, which can record the location of the last log data that has completed the consensus.

[0243] In the embodiment of the present application, in the real-time synchronization mode, the master node maintains two indexes for each target system, namely the first log index (the aforementioned log index (Last-Index)) and the second log index (the synchronized progress (Finish-Index) in the aforementioned real-time synchronization mode). The initial values ​​are all the starting index 0, and the generation process of the synchronization data structure is as follows:

[0244] Step 1a): Move the first log index corresponding to the target system to a certain threshold, which can be an index several bits after the second log index and needs to be less than or equal to the consensus log index;

[0245] Step 2a): Get all log data from the second log index to the first log index interval (left open and right closed), and read them one by one to obtain a common data set.

[0246] Step 3a): Assemble the target system identifier, common data set, log index and source node identifier to obtain the synchronization data structure.

[0247] like Fig.10 The target system A in the sync process determines the location of its current second log index, and then determines the location of the first log index according to the set number, and then starts from the next log data corresponding to the second log index, and ends at the position corresponding to the first log index, extracts these log data as the current batch of data to be synchronized, and sends them to the target system A. The number set here can be determined according to the data transmission pressure of the master node. It should be noted that different target systems may be in different synchronization processes. Therefore, a piece of log data may generate several different synchronization data structures, which is not limited in this application.

[0248] Step 4a): According to the second log index replied by the target system, the old second log index is updated.

[0249] Normally, the new second log index will be the first log index of the previous batch of synchronized data structures. Fig.10 For the target system B, when the master node receives the new second log index, it will update its position, thereby continuing to generate the synchronization data structure according to the new second log index.

[0250] It can be seen that in the embodiment of the present application, in the real-time synchronization mode, the data of the target system is a process of continuously advancing the synchronization progress. The above step 2a) can ensure that the data retrieved from the log data of each target system is in order. In step 4a), the second log index will not exceed the first log index of the previous batch of synchronization data structure when updating, and there will be no data gaps, ensuring the integrity of the retrieved data. In the real-time synchronization mode, only the order of the data is concerned and not the content of the data. The target system still needs to parse and process the data after receiving it.

[0251] In the embodiment of the present application, in the batch synchronization mode, the master node also maintains two indexes for each target system, namely the third log index (the aforementioned log execution index (Exec-Index)) and the fourth log index (the synchronized progress (Finish-Index) in the aforementioned batch synchronization mode). The initial values ​​are all the starting index 0, and the generation process of the snapshot structure is as follows:

[0252] Step 1b): The target system first injects the execution script and synchronization configuration into the distributed source data system. The synchronization configuration specifies the time and size of the transmission snapshot structure, and the execution script is responsible for processing data on behalf of the target system.

[0253] Step 2b): When new log data is agreed upon, the consensus log index will increase, and the execution script will immediately process the log data (including parsing and execution of business logic). The execution result is temporarily stored in the master node (and the backup node), and the third log index is pushed forward. In the embodiment of the present application, the business processing logic (i.e., the execution script) is pre-stored in the distributed source data system, which will complete the data processing on behalf of the target system. Finally, it is only necessary to synchronize the execution result to the target system through a snapshot, which can improve the efficiency of data processing in the target system.

[0254] Step 3b): When the configuration threshold (such as a preset capacity threshold or a preset duration threshold) is triggered, the temporarily stored processing data is taken and packaged into snapshot data in a specific format to generate a snapshot structure. The data in this snapshot data structure is equivalent to the data set within the interval from the fourth log index to the third log index (left open and right closed).

[0255] Step 4b): The synchronization schedule processing logic after the target system responds is similar to step 4a) of the real-time synchronization mode. The fourth log index will be updated to the position of the third log index, and then continue with the above step 2b).

[0256] like Fig.10 For the target system C, the master node can generate a snapshot structure through the above process and send it to the target system C.

[0257] In the embodiment of the present application, the batch synchronization mode is a process of advancing the synchronization progress in stages. The advancement span of the fourth log index is relatively large, but the advancement of the third log index in the above step 2b) is real-time and orderly. Therefore, the batch synchronization mode will not have data gaps, the data is complete, and the efficiency of the target system in processing data can be improved.

[0258] In the embodiment of the present application, a distributed source data system is used to maintain data consistency in abnormal situations. Specifically, abnormal scenarios mainly include:

[0259] 1. Network failure:

[0260] The data synchronized by the distributed source data system cannot reach the target system or the reply from the target system cannot reach the distributed source data system. For each target system, the distributed source data system will start a timer when sending synchronized data. If a reply from the target system is received within the timer, the system will continue to execute normally. Otherwise, the distributed source data system will repeat the sending steps and the related indexes will not change until the retry succeeds.

[0261] 2. Master node switching:

[0262] The distributed source data system switches the master node to another node due to network reasons or load balancing. The master node switching may occur at any time, which may cause the two nodes to send synchronization data to the target system at the same time. Therefore, the new master node can wait for a while to allow the old master node to complete the data being synchronized. When the old master node finds a new master node, it will no longer send new synchronization data, but will only complete the task being synchronized. After completion, it will no longer change the index of each target system and notify the new master node of the latest synchronization schedule. At this time, the new master node continues to perform data synchronization according to the latest index. However, due to network failures and other reasons, the old master node may not be able to complete the ongoing synchronization task. At this time, the old master node will no longer retry and will directly notify the new master node.

[0263] 3. Target system downtime:

[0264] After the target system crashes, it cannot reply to the progress of the distributed source data system synchronization. At this time, the target system may have only synchronized part of the data. The solution is the same as the network failure. The distributed source data system will retry periodically after not receiving a reply. When the target system restarts, it can still receive the data that was not processed last time, and the integrity and consistency of the data will not be affected.

[0265] 4. Distributed source data system downtime:

[0266] Each node in the distributed source data system will periodically persist the synchronization schedule. If the distributed source data system is restored, it will read the relevant indexes of each target system recorded in the disk and continue to perform data synchronization tasks. Fig.11 , Fig.11A schematic diagram of the principle of a distributed source data system crash rollback provided in an embodiment of the present application is shown. In the embodiment of the present application, since persistent storage is performed periodically, when a crash occurs, the persistent second log index that may be restored from the disk will be slightly smaller than the second log index before the crash. At this time, the second log index will roll back to the persistent second log index. The target system may receive a section of processed synchronization data, but due to the orderliness of the log data, although the data will be repeated, there will be no gaps, and the target system can eventually synchronize to complete and consistent data.

[0267] Reference Fig.12 In an embodiment of the present application, a data synchronization device is also provided, which can be used at a data sending end, and the data synchronization device includes:

[0268] An acquisition unit 1210, configured to acquire target data;

[0269] The detection unit 1220 is used to detect the network transmission status between each data receiving end and determine the data synchronization mode corresponding to each data receiving end according to the network transmission status; wherein the data synchronization mode includes a real-time synchronization mode and a batch synchronization mode;

[0270] The first processing unit 1230 is configured to send target data to the data receiving end if it is determined that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode;

[0271] The second processing unit 1240 is used to query the pre-stored business processing logic of the data receiving end if it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode, process the target data according to the business processing logic, and send the processed target data to the data receiving end in batches.

[0272] Optionally, in some embodiments, the apparatus in the embodiments of the present application further includes a conversion unit, and the conversion unit is specifically used to:

[0273] According to the data structure of the linked list, the target data is converted into multiple log data;

[0274] According to the data sorting in the target data, the log index corresponding to each log data is determined.

[0275] Optionally, in some embodiments, the first processing unit is specifically configured to:

[0276] Get the system identifier of the data receiving end;

[0277] According to the log index, a number of log data are extracted as the data to be synchronized in the current batch, and the first log index corresponding to the log data at the end position in the data to be synchronized is queried;

[0278] The system identifier, the data to be synchronized and the first log index are sent to the data receiving end.

[0279] Optionally, in some embodiments, the first processing unit is specifically configured to:

[0280] Query the second log index of the most recent feedback from the data receiving end; wherein the second log index is used to represent the log index corresponding to the log data at the end position in a batch of data to be synchronized that has been synchronized by the data receiving end;

[0281] Starting from the next log data of the log data corresponding to the second log index, the first number of log data is extracted as the data to be synchronized in the current batch.

[0282] Optionally, in some embodiments, the first processing unit is specifically configured to:

[0283] Detecting a second number of data receiving terminals that are performing data synchronization at the current time node, and a third number of data synchronization requests received in a first preset time period before the current time node;

[0284] According to the second number and the third number, determine the first number used when extracting the data to be synchronized in the current batch;

[0285] Among them, the first number is negatively correlated with the second number, and the first number is negatively correlated with the third number.

[0286] Optionally, in some embodiments, the first processing unit is further configured to:

[0287] Record the first time node when the current batch of data to be synchronized is sent to the data receiving end;

[0288] If a second log index corresponding to the current batch fed back by the data receiving end is received within a second preset period after the first time node, extract the next batch of data to be synchronized according to the second log index corresponding to the current batch;

[0289] If the second log index corresponding to the current batch fed back by the data receiving end is not received within a second preset time period after the first time node, the data to be synchronized in the current batch is resent to the data receiving end.

[0290] Optionally, in some embodiments, the second processing unit is specifically configured to:

[0291] Get the system identifier of the data receiving end;

[0292] Process the log data according to the business processing logic to obtain the processing data corresponding to the log data;

[0293] When the amount of newly processed data reaches a preset capacity threshold, or the time since the last time the processed data was sent reaches a preset time threshold, query the third log index corresponding to the last log data of the current preprocessing;

[0294] The system identifier, the processing data and the third log index are sent to the data receiving end.

[0295] Optionally, in some embodiments, the detection unit is specifically used for:

[0296] Detect network environment parameters between the data receiving end and the data receiving end;

[0297] Determine the network quality score corresponding to the data receiving end according to the network environment parameters;

[0298] If the network quality score is greater than a preset score threshold, determining that the data synchronization mode corresponding to the data receiving end is a real-time synchronization mode;

[0299] If the network quality score is less than or equal to a preset score threshold, it is determined that the data synchronization mode corresponding to the data receiving end is a batch synchronization mode.

[0300] Optionally, in some embodiments, the data sending end is a distributed source data system, the distributed source data system includes a master node and multiple slave nodes, the master node is used to synchronously transmit data to multiple data receiving ends; the acquisition unit is specifically used to:

[0301] Get the original data through the master node;

[0302] The original data is converted through the master node to obtain standard data in a universal data exchange format;

[0303] The master node sends the standard data to each slave node, and the slave nodes reach a consensus on the standard data, and the consensus-based standard data is determined as the target data.

[0304] Optionally, in some embodiments, the acquiring unit is specifically configured to:

[0305] The master node performs hash processing on the standard data to obtain a first hash value;

[0306] Sending the standard data and the first hash value to each slave node through the master node;

[0307] The second hash value is obtained by performing hash processing on the standard data from the node, and the consistency of the first hash value and the second hash value is verified to obtain a consistency verification result corresponding to the standard data;

[0308] When the consistency check results of more than half of the slave nodes pass, the standard data is determined to complete the consensus.

[0309] Optionally, in some embodiments, the data sending end is a distributed source data system, the distributed source data system includes a master node and multiple slave nodes, the master node is used to synchronously transmit data to multiple data receiving ends; the device in the embodiment of the present application also includes a switching unit, the switching unit is specifically used to:

[0310] Synchronize the data in the master node to each slave node;

[0311] Detect the operating status of distributed source data systems;

[0312] When it is determined that the distributed source data system needs to switch the master node, a new master node is determined from the slave nodes;

[0313] Detect whether there are unfinished data synchronization tasks in the old master node; wherein the unfinished data synchronization tasks are data synchronization tasks for which the data to be synchronized has been sent but the second log index corresponding to the feedback from the data receiving end has not been received;

[0314] If there are unfinished data synchronization tasks in the old master node, stop sending new data to be synchronized through the old master node until the unfinished data synchronization task is completed or times out, and then switch to the new master node for data synchronization transmission.

[0315] It is understandable that if Figure 5 The contents of the data synchronization method embodiment shown in the figure are applicable to the data synchronization device embodiment. The functions specifically implemented by the data synchronization device embodiment are similar to those in the embodiment shown in the figure. Figure 5 The data synchronization method embodiment shown is the same as that shown in FIG. Figure 5 The beneficial effects achieved by the data synchronization method embodiment shown are also the same.

[0316] The present application also discloses an electronic device, including:

[0317] at least one processor;

[0318] at least one memory for storing at least one program;

[0319] When at least one program is executed by at least one processor, the at least one processor implements Figure 5 An embodiment of a data synchronization method is shown.

[0320] The electronic device of the embodiment of the present application may be a terminal device, a computer device or a server device.

[0321] For example, refer to Fig.13 , Fig.13 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Taking the electronic device as a terminal device as an example, Fig.13In the embodiment, the electronic device 1300 may include an RF (Radio Frequency) circuit 1310, a memory 1320 including one or more computer-readable storage media, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a short-range wireless transmission module 1370, a processor 1380 including one or more processing cores, and a power supply 1390. Those skilled in the art will appreciate that Fig.13 The device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0322] The RF circuit 1310 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 1380 for processing; in addition, the data related to the uplink is sent to the base station. Generally, the RF circuit 1310 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subject identity module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. In addition, the RF circuit 1310 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0323] The memory 1320 can be used to store software programs and modules (or units). The processor 1380 executes various functional applications and data processing by running the software programs and modules (or units) stored in the memory 1320. The memory 1320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function), etc.; the data storage area may store data created according to the use of the electronic device 1300 (such as audio data, a phone book), etc. In addition, the memory 1320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1320 may also include a memory controller to provide the processor 1380 and the input unit 1330 with access to the memory 1320. Although Fig.13 The RF circuit 1310 is shown, but it is understandable that it is not an essential component of the electronic device 1300 and can be omitted as needed without changing the essence of the invention.

[0324] The input unit 1330 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to object setting and function control. Specifically, the input unit 1330 may include a touch-sensitive surface 1331 and other input devices 1332. The touch-sensitive surface 1331, also known as a touch display screen or touch pad, can collect touch operations of an object on or near it (such as operations of an object using a finger, stylus or any other suitable object or accessory on or near the touch-sensitive surface 1331), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 1331 may include a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the object, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 1380, and can receive instructions sent by the processor 1380 and execute them. In addition, the touch-sensitive surface 1331 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 1331, the input unit 1330 may further include other input devices 1332. Specifically, the other input devices 1332 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0325] The display unit 1340 can be used to display information input by an object or information provided to an object and to control various graphic object interfaces of the electronic device 1300. These graphic object interfaces can be composed of graphics, texts, icons, videos and any combination thereof. The display unit 1340 may include a display panel 1341. Optionally, the display panel 1341 may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like. Further, the touch-sensitive surface 1331 may be covered on the display panel 1341. When the touch-sensitive surface 1331 detects a touch operation on or near it, it is transmitted to the processor 1380 to determine the type of touch event. Subsequently, the processor 1380 provides a corresponding visual output on the display panel 1341 according to the type of touch event. Although in Fig.13 In the embodiment, the touch-sensitive surface 1331 and the display panel 1341 are implemented as two independent components to implement input and output functions, but in some embodiments, the touch-sensitive surface 1331 and the display panel 1341 can be integrated to implement input and output functions.

[0326] The electronic device 1300 may also include at least one sensor 1350, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1341 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1341 or the backlight when the electronic device 1300 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 1300, they will not be repeated here.

[0327] The audio circuit 1360, the speaker 1361, and the microphone 1362 can provide an audio interface between the object and the electronic device 1300. The audio circuit 1360 can transmit the electrical signal converted from the received audio data to the speaker 1361, which is converted into a sound signal for output; on the other hand, the microphone 1362 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1360 and converted into audio data, and then the audio data is output to the processor 1380 for processing, and then sent to another electronic device through the RF circuit 1310, or the audio data is output to the memory 1320 for further processing. The audio circuit 1360 may also include an earplug jack to provide communication between an external headset and the electronic device 1300.

[0328] The short-range wireless transmission module 1370 may be a WIFI (wireless fidelity) module, a Bluetooth module, an infrared module, etc. The electronic device 1300 may transmit information with wireless transmission modules provided on other devices through the short-range wireless transmission module 1370 .

[0329] The processor 1380 is the control center of the electronic device 1300. It uses various interfaces and lines to connect various parts of the entire device. By running or executing software programs or modules stored in the memory 1320, and calling data stored in the memory 1320, it executes various functions of the electronic device 1300 and processes data, thereby controlling the device as a whole. Optionally, the processor 1380 may include one or more processing cores; optionally, the processor 1380 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, object interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1380.

[0330] The electronic device 1300 also includes a power supply 1390 (such as a battery) for supplying power to various components. Optionally, the power supply 1390 can be logically connected to the processor 1380 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 1390 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0331] Although not shown, the electronic device 1300 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0332] The present application also discloses a computer-readable storage medium in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the following Figure 5 An embodiment of a data synchronization method is shown.

[0333] Understandably, Figure 5 The contents of the data synchronization method embodiment shown in the figure are applicable to the computer-readable storage medium embodiment. The functions specifically implemented by the computer-readable storage medium embodiment are similar to those of Figure 5 The data synchronization method embodiment shown is the same as that of Figure 5 The beneficial effects achieved by the data synchronization method embodiment shown are also the same.

[0334] The embodiment of the present application further discloses a computer program product or a computer program, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in the above-mentioned computer-readable storage medium; Fig.13 The processor of the electronic device shown can read the computer instruction from the above-mentioned computer readable storage medium, and the processor executes the computer instruction so that the computer device executes Figure 5 An embodiment of a data synchronization method is shown.

[0335] Understandably, Figure 5 The contents of the data synchronization method embodiment shown in the figure are all applicable to the computer program product or computer program embodiment, and the functions specifically implemented by the computer program product or computer program embodiment are the same as those in the embodiment of the present invention. Figure 5 The data synchronization method embodiment shown is the same as that of Figure 5 The beneficial effects achieved by the data synchronization method embodiment shown are also the same.

[0336] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0337] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0338] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0339] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0340] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus, or device and execute instructions), or in conjunction with such instruction execution systems, apparatuses, or devices. For purposes of this specification, a "computer-readable storage medium" may be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution systems, apparatuses, or devices.

[0341] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0342] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0343] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0344] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A data synchronization method, characterized in that: Used for a data sending end, the data sending end performs data synchronous transmission to multiple data receiving ends, and the data sending end pre-stores the service processing logic of each of the data receiving ends, and the service processing logic of each of the data receiving ends is different; the method includes: Get target data; Detecting the network transmission status between each of the data receiving terminals, and determining the data synchronization mode corresponding to each of the data receiving terminals according to the network transmission status; wherein the data synchronization mode includes a real-time synchronization mode and a batch synchronization mode; If it is determined that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode, sending the target data to the data receiving end; If it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode, query the pre-stored business processing logic of the data receiving end, process the target data according to the business processing logic, and send the processed target data to the data receiving end in batches; After acquiring the target data, the method further includes: According to the data structure of the linked list, convert the target data into a plurality of log data; Determine the log index corresponding to each piece of the log data according to the data sorting in the target data; The sending the target data to the data receiving end comprises: Acquire a system identifier of the data receiving end; According to the log index, a number of the log data are extracted as the data to be synchronized in the current batch, and a first log index corresponding to the log data at the end position in the data to be synchronized is queried; Sending the system identifier, the data to be synchronized, and the first log index to the data receiving end; The step of extracting a plurality of log data as the data to be synchronized in the current batch according to the log index includes: Query the second log index of the most recent feedback from the data receiving end; wherein the second log index is used to represent the log index corresponding to the log data at the end position in a batch of data to be synchronized that has been synchronized by the data receiving end; Starting from the next log data of the log data corresponding to the second log index, the first number of log data is extracted as the data to be synchronized in the current batch.

2. The data synchronization method according to claim 1, characterized in that: The first number is determined by the following steps: Detecting a second number of data receiving terminals that are performing data synchronization at the current time node, and a third number of data synchronization requests received in a first preset time period before the current time node; Determine, according to the second number and the third number, the first number used when extracting the data to be synchronized of the current batch; The first number is negatively correlated with the second number, and the first number is negatively correlated with the third number.

3. The data synchronization method according to claim 1, characterized in that: The method further comprises: Recording the first time node of sending the current batch of data to be synchronized to the data receiving end; If a second log index corresponding to the current batch fed back by the data receiving end is received within a second preset period after the first time node, extracting the next batch of data to be synchronized according to the second log index corresponding to the current batch; If the second log index corresponding to the current batch fed back by the data receiving end is not received within a second preset time period after the first time node, the data to be synchronized in the current batch is resent to the data receiving end.

4. The data synchronization method according to claim 1, characterized in that: The processing of the target data according to the business processing logic and sending the processed target data in batches to the data receiving end includes: Acquire a system identifier of the data receiving end; Processing the log data according to the business processing logic to obtain processing data corresponding to the log data; When the amount of the newly added processed data reaches a preset capacity threshold, or the time since the last sending of the processed data reaches a preset time threshold, query the third log index corresponding to the last log data of the current preprocessing; The system identifier, the processed data and the third log index are sent to the data receiving end.

5. The data synchronization method according to any one of claims 1 to 4, characterized in that: The detecting of the network transmission status between the data receiving terminals and determining the data synchronization mode corresponding to each of the data receiving terminals according to the network transmission status includes: Detecting network environment parameters between the data receiving end and the data receiving end; Determining a network quality score corresponding to the data receiving end according to the network environment parameters; If the network quality score is greater than a preset score threshold, determining that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode; If the network quality score is less than or equal to the preset score threshold, it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode.

6. The data synchronization method according to claim 1, characterized in that: The data sending end is a distributed source data system, the distributed source data system includes a master node and multiple slave nodes, the master node is used to synchronously transmit data to multiple data receiving ends; The obtaining of target data comprises: Acquire original data through the master node; The original data is converted by the master node to obtain standard data in a universal data exchange format; The master node sends the standard data to each of the slave nodes, and the slave nodes reach a consensus on the standard data, and the consensus-based standard data is determined as the target data.

7. The data synchronization method according to claim 6, characterized in that: The step of sending the standard data to each of the slave nodes through the master node, and reaching consensus on the standard data through the slave nodes, includes: Performing hash processing on the standard data by the master node to obtain a first hash value; Sending the standard data and the first hash value to each of the slave nodes through the master node; Performing hash processing on the standard data by the slave node to obtain a second hash value, and verifying the consistency of the first hash value and the second hash value to obtain a consistency verification result corresponding to the standard data; When the consistency check results of more than half of the slave nodes are passed, it is determined that the standard data has completed consensus.

8. The data synchronization method according to claim 1, characterized in that: The data sending end is a distributed source data system, the distributed source data system includes a master node and multiple slave nodes, the master node is used to synchronously transmit data to multiple data receiving ends; The method further comprises: Synchronize the data in the master node to each of the slave nodes; Detecting the operating status of the distributed source data system; When it is determined that the distributed source data system needs to switch the master node, determining a new master node from the slave nodes; Detect whether there is an unfinished data synchronization task in the old master node; wherein the unfinished data synchronization task is a data synchronization task for which the data to be synchronized has been sent but the second log index corresponding to the feedback from the data receiving end has not been received; If there are unfinished data synchronization tasks in the old master node, stop sending new data to be synchronized through the old master node until the unfinished data synchronization task is completed or times out, and switch to the new master node for data synchronization transmission.

9. A data synchronization device, characterized in that: Used for a data sending end, the data sending end performs data synchronous transmission to multiple data receiving ends, and the data sending end pre-stores the service processing logic of each of the data receiving ends, and the service processing logic of each of the data receiving ends is different; the device includes: An acquisition unit, used for acquiring target data; A detection unit, used to detect the network transmission status between each of the data receiving terminals, and determine the data synchronization mode corresponding to each of the data receiving terminals according to the network transmission status; wherein the data synchronization mode includes a real-time synchronization mode and a batch synchronization mode; A first processing unit, configured to send the target data to the data receiving end if it is determined that the data synchronization mode corresponding to the data receiving end is the real-time synchronization mode; A second processing unit is configured to query the pre-stored service processing logic of the data receiving end if it is determined that the data synchronization mode corresponding to the data receiving end is the batch synchronization mode, process the target data according to the service processing logic, and send the processed target data to the data receiving end in batches; The device further comprises a conversion unit, which is specifically configured to: According to the data structure of the linked list, convert the target data into a plurality of log data; Determine the log index corresponding to each piece of the log data according to the data sorting in the target data; The first processing unit is specifically configured to: Acquire a system identifier of the data receiving end; According to the log index, a number of the log data are extracted as the data to be synchronized in the current batch, and a first log index corresponding to the log data at the end position in the data to be synchronized is queried; Sending the system identifier, the data to be synchronized, and the first log index to the data receiving end; The first processing unit is specifically configured to: Query the second log index of the most recent feedback from the data receiving end; wherein the second log index is used to represent the log index corresponding to the log data at the end position in a batch of data to be synchronized that has been synchronized by the data receiving end; Starting from the next log data of the log data corresponding to the second log index, the first number of log data is extracted as the data to be synchronized in the current batch.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, a data synchronization method as described in any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a data synchronization method as described in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a data synchronization method as described in any one of claims 1 to 8 is implemented.

Images