Data synchronization method, device, computer equipment and storage medium for multiple data sources

Abstract

The present application relates to a data synchronization method, apparatus, computer device, and storage medium for multiple data sources. The method includes: sending service data of multiple data types through different data channels, so that a data platform receiving each piece of service data stores each piece of service data in a data pool, and reads the service data from the data pool according to the service priority for data synchronization to obtain a synchronization result; after waiting for a preset query interval, sending a first query request to the data platform; the synchronization result queried based on the first query request returned by the data platform; updating the data synchronization identifier of the service data based on the synchronization result, where the data synchronization identifier indicates whether the service data is successfully synchronized. By using this method, efficient data synchronization operations can be achieved when multiple data sources synchronize a large amount of data simultaneously.

Description

Technical Field

[0001] The present application relates to the field of communication technologies, and in particular, to a method and apparatus for multi-data source data synchronization, a computer device, and a storage medium. Background Art

[0002] With the development of network communication technologies, information interaction has become increasingly important. Data synchronization can improve data accuracy and ensure consistency between each data source and its different endpoints.

[0003] In traditional data synchronization solutions, when a data source sends a data synchronization request to a data platform, it needs to wait for the data platform to perform logical processing on the synchronized data, and then the data platform returns the synchronization result to the data source. When multiple data sources send data synchronization requests to the data platform, the data platform needs to allocate memory for logical processing for each request. The more data sources there are, the more memory is consumed. Moreover, after data synchronization is completed, each data source sends a synchronization result query request to the data platform, further increasing the data processing pressure on the data platform and resulting in low data synchronization efficiency. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method and apparatus for multi-data source data synchronization, a computer device, and a storage medium that can perform efficient data synchronization.

[0005] A method for multi-data source data synchronization, the method comprising:

[0006] Sending business data of multiple data types through different data channels, so that a data platform receiving each piece of the business data stores each piece of the business data in a data pool, and reads the business data from the data pool according to business priorities for data synchronization to obtain a synchronization result;

[0007] After a preset query interval, sending a first query request to the data platform;

[0008] Receiving the synchronization result queried based on the first query request returned by the data platform;

[0009] Updating a data synchronization identifier of the business data based on the synchronization result, where the data synchronization identifier indicates whether the business data is successfully synchronized.

[0010] In one embodiment, after sending the business data of multiple data types through different data channels, the method further comprises:

[0011] Modifying the data sending status of the business data to sent;

[0012] Receive the result information sent by the data platform indicating whether the service data has been received;

[0013] Reject adjusting the sent status of the modified data based on the result information.

[0014] In one embodiment, the storing each piece of the service data in a data pool and reading the service data from the data pool according to the service priority for data synchronization to obtain a synchronization result includes:

[0015] Store each piece of the service data in the data pool;

[0016] Determine the time interval for data processing according to the processing efficiency of processing the service data;

[0017] Read a preset number of service data from the data pool based on the time interval;

[0018] Synchronize the service data to the database to obtain a synchronization result.

[0019] In one embodiment, the method is applied to the data source side; the first query request carries a data source side identifier;

[0020] The receiving the synchronization result queried based on the synchronization result query request returned by the data platform includes:

[0021] Receive the synchronization result of the target service data queried by the data platform based on the data source side identifier; the target service data is the service data sent by the data source side, and the data platform does not send the synchronization result of the target service data to the data source side after completing data synchronization.

[0022] In one embodiment, the method further includes:

[0023] After sending the first query request to the data platform, listen in real time for the synchronization result returned by the data platform;

[0024] If the synchronization result requested by the first query request is not received after a preset duration, send a second query request carrying the data source side identifier and the service data identifier to the data platform;

[0025] Receive the synchronization result obtained by the data platform based on the data source side identifier and the service data identifier carried by the second query request;

[0026] Wherein, the service data identifier is the data identifier of the service data for which the synchronization result requested by the first query request is not received.

[0027] In one embodiment, the receiving of the synchronization result queried based on the first query request from the data platform includes:

[0028] Receiving the synchronization result queried by the data platform from the stored synchronization result table based on the first query request;

[0029] Wherein, the synchronization result includes synchronization success, or synchronization failure and the reason for failure.

[0030] In one embodiment, the updating of the data synchronization identifier of the service data based on the synchronization result includes:

[0031] When the data synchronization identifier indicates that the service data synchronization fails, modifying the data sending status of the service data to not sent, and modifying the data synchronization identifier of the service data to synchronization failure;

[0032] When the data synchronization identifier indicates that the service data synchronization succeeds, updating the data synchronization identifier of the service data to synchronization success.

[0033] A data synchronization system for multiple data sources, the system includes:

[0034] At least two data sources, used for sending service data of multiple data types through different data channels;

[0035] A data platform, used for storing the received service data in a data pool, and reading the service data from the data pool according to the service priority for data synchronization to obtain a synchronization result;

[0036] The at least two data sources are further used for, after waiting for a preset query interval, sending a first query request to the data platform;

[0037] The data platform is further used for querying the synchronization result based on the first query request;

[0038] The at least two data sources are further used for receiving the synchronization result, and updating the data synchronization identifier of the service data based on the synchronization result, where the data synchronization identifier indicates whether the service data is synchronized successfully.

[0039] A data synchronization method for multiple data sources, the method includes:

[0040] Receiving service data sent by a data source, and storing the service data in a data pool;

[0041] After waiting for a preset reading interval, reading the service data according to the service priority for data synchronization to obtain a synchronization result;

[0042] Receive a first query request sent by the data source and return a synchronization result queried based on the first query request.

[0043] In one embodiment, storing each piece of the service data in a data pool and reading the service data from the data pool according to service priorities includes: storing each piece of the service data in the data pool; determining a time interval for data processing according to the processing efficiency of processing the service data; reading a preset number of service data from the data pool based on the time interval; synchronizing the service data to a database to obtain a synchronization result.

[0044] A data synchronization device for multiple data sources, the device includes:

[0045] A sending module, configured to send service data of multiple data types through different data channels, so that a data platform receiving each piece of the service data stores each piece of the service data in a data pool, and reads the service data from the data pool according to service priorities for data synchronization to obtain a synchronization result;

[0046] A query module, configured to send a first query request to the data platform after waiting for a preset query interval;

[0047] A receiving module, configured to receive a synchronization result queried based on the first query request returned by the data platform;

[0048] An update module, configured to update a data synchronization identifier of the service data based on the synchronization result, where the data synchronization identifier indicates whether the service data is synchronized successfully.

[0049] In one embodiment, the update module is further configured to modify a data sending status of the service data to sent; receive result information sent by the data platform indicating whether the service data is received; and reject adjusting the modified data sending status based on the result information.

[0050] In one embodiment, the device further includes:

[0051] A control module, configured to instruct the data platform to store each piece of the service data in a data pool; determine a time interval for data processing according to the processing efficiency of processing the service data; and read a preset number of service data from the data pool based on the time interval.

[0052] In one embodiment, the method is applied to a data source side; the first query request carries a data source side identifier;

[0053] The receiving module is further configured to receive the synchronization result of the target service data queried based on the data source end identifier returned by the data platform; the target service data is the service data sent by the data source end, and the data platform does not send the synchronization result of the target service data to the data source end after completing data synchronization.

[0054] In one embodiment, the receiving module is further configured to, after sending a first query request to the data platform, listen for the synchronization result returned by the data platform in real time; if the synchronization result requested by the first query request is not received after a preset duration, send a second query request carrying the data source end identifier and the service data identifier to the data platform; receive the synchronization result obtained by the data platform based on the data source end identifier and the service data identifier carried by the second query request; wherein, the service data identifier is the data identifier of the service data for which the synchronization result requested by the first query request is not received.

[0055] In one embodiment, the receiving module is further configured to receive the synchronization result queried by the data platform from the stored synchronization result table based on the first query request; wherein, the synchronization result includes synchronization success, or synchronization failure and the reason for failure.

[0056] In one embodiment, the updating module is further configured to, when the data synchronization identifier indicates that the service data synchronization fails, modify the data sending status of the service data to not sent, and modify the data synchronization identifier of the service data to synchronization failure; when the data synchronization identifier indicates that the service data synchronization succeeds, update the data synchronization identifier of the service data to synchronization success.

[0057] A multi-data-source data synchronization device, the device includes:

[0058] A receiving module, configured to receive service data of multiple data types sent by a data source end through different data channels;

[0059] A reading module, configured to store each piece of the service data in a data pool, and read the service data from the data pool according to the service priority for data synchronization to obtain a synchronization result;

[0060] The receiving module is further configured to receive a first query request sent by the data source end after waiting for a preset query interval;

[0061] A query module, configured to query the synchronization result based on the first query request, and return the synchronization result to the data source end, so that the data source end updates the data synchronization identifier of the service data based on the synchronization result, and the data synchronization identifier indicates whether the service data is synchronized successfully.

[0062] In one of the embodiments, the reading module is also used to store each of the business data in a data pool; determine the time interval for data processing based on the processing efficiency of processing the business data; read a preset amount of business data from the data pool based on the time interval; and synchronize the business data to a database to obtain a synchronization result.

[0063] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned method for synchronizing data from multiple data sources.

[0064] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the steps of the above-mentioned method for synchronizing data from multiple data sources.

[0065] The above-mentioned data synchronization method, device, computer equipment and storage medium of multiple data sources send business data of multiple data types through different data channels, so that the data platform receiving each business data stores each business data in the data pool, and reads the business data from the data pool according to the business priority for data synchronization to obtain the synchronization result; after waiting for the preset query interval, send a first query request to the data platform; receive the synchronization result returned by the data platform based on the first query request query; update the data synchronization identifier of the business data based on the synchronization result, and the data synchronization identifier indicates whether the business data is synchronized successfully. Sending business data through appropriate data channels respectively can improve the efficiency of data transmission; the data platform receives business data through the data pool and processes business data regularly, which can reasonably use memory and avoid memory leakage, and sends the synchronization result query request to the data platform after waiting for the time interval, which staggers the synchronization peak of the data platform, reduces the pressure of data processing, and improves the efficiency of business data processing. Therefore, when multiple data sources synchronize large quantities of data at the same time, efficient data synchronization operations can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A diagram of an application environment of a data synchronization method for multiple data sources in one embodiment;

[0067] Figure 2 A schematic diagram of a flow chart of a method for synchronizing data from multiple data sources in one embodiment;

[0068] Figure 3 A schematic diagram of a large batch data operation synchronized with multiple data sources at the same time in one embodiment;

[0069] Figure 4 is a schematic diagram of using different data channels in one embodiment;

[0070] Figure 5 Schematic diagram of sending a query request in an embodiment;

[0071] Figure 6 Schematic flow diagram of the data platform reading service data in an embodiment;

[0072] Figure 7 Schematic flow diagram of the data synchronization method for multiple data sources in another embodiment;

[0073] Figure 8 Structural block diagram of the data synchronization device for multiple data sources in an embodiment;

[0074] Figure 9 Structural block diagram of the data synchronization device for multiple data sources in another embodiment;

[0075] Figure 10 Structural block diagram of the data synchronization device for multiple data sources in another embodiment;

[0076] Figure 11 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0077] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0078] The data synchronization method for multiple data sources provided by the present application can be applied to an application environment as shown in Figure 1 In this application environment, it includes a data source end 102 and a data platform 104.

[0079] The data source end 102 sends service data of multiple data types through different data channels, so that the data platform 104 that receives each service data stores each service data in a data pool, and reads the service data from the data pool according to the service priority for data synchronization to obtain a synchronization result; after waiting for a preset query interval, a first query request is sent to the data platform 104; the data source end 102 receives the synchronization result queried based on the first query request returned by the data platform; the data synchronization identifier of the service data is updated based on the synchronization result, and the data synchronization identifier indicates whether the service data is synchronized successfully.

[0080] Among them, the data source end 102 may be a terminal, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.

[0081] The data platform 104 can be an independent physical server or a service node in a blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP).

[0082] In addition, the data platform 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0083] The data source end 102 and the data platform 104 can be connected through communication connection methods such as Bluetooth, USB (Universal Serial Bus), or network. This application does not limit this here.

[0084] In one embodiment, as Figure 2 shown, a multi-data-source data synchronization method is provided. Taking the data source end in Figure 1 as an example, the method includes the following steps:

[0085] S202, sending business data of multiple data types through different data channels, so that the data platform receiving each business data stores each business data in a data pool, and reads the business data from the data pool according to the business priority for data synchronization to obtain a synchronization result.

[0086] Among them, the types of business data can be text type, digital type, time type, etc. The business data can specifically be manifested as commodity information, including commodity identifiers, commodity prices, commodity inventory quantities, and storage addresses of commodity pictures, etc.

[0087] In addition, the priority can be divided according to the urgency of the business data. For example, the priority can be divided according to time sequence. The business data stored in the data pool carries a timestamp, which records the time when the business data requests synchronization. The earlier the request synchronization time on the business data, the higher the priority for the data platform to synchronize the business data.

[0088] In addition, as Figure 3 shown, there can be multiple data source ends, and multiple data source ends can simultaneously send a large amount of business data to the data platform.

[0089] In one embodiment, after the data source end sends business data of multiple data types through different data channels, it modifies the data sending status of the business data to "sent"; receives result information sent by the data receiving platform indicating whether the business data has been received; and refuses to adjust the modified data sending status based on the result information.

[0090] For example, when a large number of business data to be synchronized are lost during the data packet transmission in the data channel, after receiving some of the business data, the data platform will send result information to the data source end indicating that it has not received all the business data to be synchronized. After receiving this result information, the data source end will not process this result information nor correspondingly modify the sending status of the business data to be synchronized.

[0091] In one embodiment, as Figure 4 shown, the data source end classifies business data according to different data types, determines the transmission channel according to the category of the business data, and transmits data to the data platform. For example, lightweight data such as text data uses the HTTP (Hyper Text Transfer Protocol) protocol, and larger data such as file data uses the TCP (Transmission Control Protocol) protocol.

[0092] S204: After waiting for a preset query interval, send a first query request to the data platform.

[0093] Among them, the preset query interval can be several seconds, several minutes, or other interval times. The first query request carries the data source end identifier, which can be a digital identifier, a letter identifier, a text identifier, or a mixed identifier of numbers and letters, etc. The data platform can retrieve the synchronization result of all the business data requested to be synchronized by this data source end based on this data source end identifier. For example, if the data source end presets a query interval of 30 minutes, 30 minutes after sending business data of multiple data types through different data channels, the data source end will send a first query request carrying this data source end identifier to the data platform.

[0094] In one embodiment, as Figure 5 shown, the data source end can send the first query request to the data platform multiple times. When the data source end has not received the synchronization result of all the business data requested to be synchronized, it can send the first query request to the data platform at preset time intervals.

[0095] For example, the preset sending time interval at the data source end is 30 minutes. After the data source end sends the first query request to the data platform for the first time and only receives the synchronization results of some of the service data for which the request synchronization is requested, the data source end will send the first query request again 30 minutes after receiving the synchronization results.

[0096] S206. Receive the synchronization results queried based on the first query request returned by the data platform.

[0097] In one embodiment, the data source end receives the synchronization results of the target service data queried based on the data source end identifier from the data platform; the target service data is the service data sent by the data source end, and the data platform does not send the synchronization results of the target service data to the data source end after completing the data synchronization.

[0098] In one embodiment, after the data platform sends the synchronization results of the service data, it will update the sending status in the service data synchronization results and change the sending status to sent. When the data source end sends the first query request again, the synchronization results of the service data with the sending status of sent will not be sent again.

[0099] For example, the data source end sends three pieces of service data to be synchronized. When the data platform receives the first query request from the data source end, the data platform will retrieve the data source end identifier in the first query request. At this time, it is found that only the first and second pieces of service data have been processed. The data platform will send the synchronization results of these two pieces of processed service data to the data source end and change the sending status in the synchronization results of these two pieces of service data to sent. When the data platform receives the first query request from this data source end again, the data platform will retrieve again based on this data source end identifier. When it is found that the third piece of service data has been processed, the data platform will only send the synchronization results of the third piece of service data, and the service data with the previously sent synchronization results will not be sent again.

[0100] In one embodiment, the data source end listens for the synchronization results returned by the data platform in real time; if the synchronization results requested by the first query request are not received after a preset duration, a second query request carrying the data source end identifier and the service data identifier is sent to the data platform; receive the synchronization results obtained based on the data source end identifier and the service data identifier carried in the second query request returned by the data platform. Among them, the service data identifier is the data identifier of the service data for which the synchronization results requested by the first query request have not been received, and the data identifier of the service data can be a digital identifier, a letter identifier, a text identifier, or a mixed identifier of numbers and letters, etc.

[0101] For example, the data source is preset with a duration of 24 hours. When the data source end receives the synchronization result returned by the data platform in response to the first query request and has not processed the synchronization result yet, a power outage occurs at the data source end, resulting in the loss of the synchronization result. The data source end that restores power will think that it has not received the synchronization result, and thus continuously send the first query request to the data platform. When 24 hours have passed since the data source end first sent the first query request and it still has not received the synchronization results of all business data, the data source end will send a second query request carrying the data source end identifier and the business data identifier.

[0102] In one embodiment, the data source end receives the synchronization result queried by the data platform from the stored synchronization result table based on the first query request; wherein, the synchronization result includes synchronization success, or synchronization failure and the reason for failure.

[0103] S208, update the data synchronization identifier of the business data based on the synchronization result, and the data synchronization identifier indicates whether the business data is synchronized successfully.

[0104] In one embodiment, when the data synchronization identifier indicates that the business data synchronization fails, the data source end modifies the data sending status of the business data to not sent, and modifies the data synchronization identifier of the business data to synchronization failure; when the data synchronization identifier indicates that the business data synchronization succeeds, the data source end updates the data synchronization identifier of the business data to synchronization success.

[0105] In one embodiment, when the data synchronization identifier indicates that the business data synchronization fails, the data source end will update the business data with synchronization failure according to the reason for failure in the returned synchronization result. After the update is completed, the business data is sent again. For example, when the reason for failure is that the format of the business data to be synchronized sent by the data source end does not meet the requirements of the data platform, the data source end will modify the format of the business data to be synchronized according to this reason for failure. After the modification is completed, the business data is sent to the data platform again.

[0106] In the above-mentioned data synchronization method for multiple data sources, business data of multiple data types are sent through different data channels, so that the data platform receiving each business data stores each business data in the data pool, and reads the business data from the data pool according to the business priority for data synchronization to obtain the synchronization result; after waiting for the preset query interval, a first query request is sent to the data platform; the synchronization result based on the first query request query returned by the data platform is received; based on the synchronization result, the data synchronization identifier of the business data is updated, and the data synchronization identifier indicates whether the business data is successfully synchronized. The business data is sent through appropriate data channels respectively, which can improve the efficiency of data transmission; the data platform receives the business data through the data pool, processes the business data regularly, can reasonably use the memory, avoid memory leakage, and waits for the time interval before sending the synchronization result query request to the data platform, which staggers the synchronization peak of the data platform, reduces the pressure of data processing, and improves the efficiency of business data processing. Therefore, when multiple data sources synchronize large quantities of data at the same time, efficient data synchronization operations can be achieved.

[0107] In one embodiment, Figure 6 As shown, S202, the method includes:

[0108] S602, the data platform stores each business data in a data pool.

[0109] The data pool may be a message queue, which may use message systems such as RabbitMQ, ActiveMQ, and Kafka. Message queues implement data transfer between processes in a distributed system. This data transfer refers to the communication between the data source and the data platform by sending business data in messages, rather than by directly calling each other's processes.

[0110] S604, determining a time interval for data processing according to the processing efficiency of the business data.

[0111] The processing efficiency refers to the maximum number of business data that the data platform can process within a period of time without the system crashing. The data platform reads business data from the data pool at intervals.

[0112] S606, reading a preset amount of business data from the data pool based on a time interval, synchronizing the business data to a database, and obtaining a synchronization result.

[0113] In one embodiment, based on the time interval and the processing efficiency of the data platform for business data, the data platform presets the number of business data to be read. For example, if the processing efficiency of the data platform for business data is at most 1000 business data per minute, then the data platform can set the time interval to 1 minute and the number of business data read each time to 1000. The data platform will read 1000 business data from the data pool each time. After an interval of 1 minute, the data platform will read another 1000 business data from the data pool.

[0114] In this embodiment, by setting the time interval for data processing and reading a preset number of business data from the data pool, the processing efficiency of the data platform for business data can be controlled, ensuring that the data platform operates efficiently under appropriate pressure.

[0115] In one embodiment, as Figure 7 shown, a method for data synchronization of multiple data sources is provided. This embodiment mainly takes the application of this method to the data platform 104 in the above Figure 1 as an example for illustration. Referring to Figure 7 , the method for data synchronization of multiple data sources specifically includes the following steps:

[0116] S702, Receive business data of multiple data types sent by the data source end through different data channels.

[0117] S704, Store each piece of business data in the data pool, and read the business data from the data pool according to the business priority for data synchronization to obtain a synchronization result.

[0118] In one embodiment, store each piece of the business data in the data pool; determine the time interval for data processing according to the processing efficiency of processing the business data; read a preset number of business data from the data pool based on the time interval; synchronize the business data to the database to obtain a synchronization result.

[0119] S706, Receive a first query request sent by the data source end after waiting for a preset query interval.

[0120] S708, Query the synchronization result based on the first query request, and return the synchronization result to the data source end, so that the data source end updates the data synchronization identifier of the business data based on the synchronization result. The data synchronization identifier indicates whether the business data is synchronized successfully.

[0121] The specific steps of S701 - S708 can refer to Figure 2 the steps S202 - S208 in

[0122] The above data synchronization method for multiple data sources receives service data of multiple data types sent through different data channels at the data source end, stores each piece of service data in a data pool, which can reduce the logical steps for processing service data; and reads service data from the data pool according to service priorities for data synchronization to obtain a synchronization result; it can reasonably utilize memory and avoid memory leaks. Therefore, when multiple data sources synchronize a large amount of data simultaneously, efficient data synchronization operations can be achieved.

[0123] It should be understood that although Figure 2 、 6 each step in the flowcharts of Figure 2 、 6 、7 is shown in sequence according to the arrow indications, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover,

[0124] In one embodiment, as Figure 8 shown, a data synchronization device for multiple data sources is provided. This device can be a software module, a hardware module, or a combination of both to form a part of a computer device. Specifically, the device includes: a sending module 802, a query module 804, a receiving module 806, and an updating module 808, where:

[0125] The sending module 802 is used to send service data of multiple data types through different data channels, so that the data platform receiving each piece of service data stores each piece of service data in a data pool, and reads service data from the data pool according to service priorities for data synchronization to obtain a synchronization result.

[0126] The query module 804 is used to send a first query request to the data platform after waiting for a preset query interval.

[0127] The receiving module 806 is used to receive the synchronization result queried based on the first query request returned by the data platform.

[0128] The update module 808 is used to update the data synchronization identifier of the business data based on the synchronization result, and the data synchronization identifier indicates whether the business data is successfully synchronized. In the above embodiment, by sending business data of multiple data types through different data channels, the data platform receiving each business data stores each business data in the data pool, and reads the business data from the data pool according to the business priority for data synchronization, and obtains the synchronization result; after waiting for the preset query interval, the first query request is sent to the data platform; the synchronization result based on the query of the first query request returned by the data platform is received; based on the synchronization result, the data synchronization identifier of the business data is updated, and the data synchronization identifier indicates whether the business data is successfully synchronized. The business data is sent through appropriate data channels respectively, which can improve the efficiency of data transmission; the data platform receives the business data through the data pool, processes the business data regularly, can reasonably use the memory, avoid memory leakage, and waits for the time interval before sending the synchronization result query request to the data platform, which staggers the synchronization peak of the data platform, reduces the pressure of data processing, and improves the efficiency of business data processing. Therefore, when multiple data sources synchronize large quantities of data at the same time, efficient data synchronization operation can be achieved.

[0129] In one of the embodiments, the update module 808 is further used to modify the data sending status of the business data to sent; receive result information sent by the data platform to indicate whether the business data is received; and refuse to adjust the modified data sending status based on the result information.

[0130] In one embodiment, if Figure 9 As shown, the device also includes:

[0131] Control module 810 is used to instruct the data platform to store various business data in the data pool; determine the time interval for data processing according to the processing efficiency of business data; read a preset amount of business data from the data pool based on the time interval; synchronize the business data to the database to obtain synchronization results.

[0132] In one of the embodiments, the first query request carries a data source identifier; the receiving module 706 is also used to receive the synchronization result of the target business data returned by the data platform based on the query of the data source identifier; the target business data is the business data sent by the data source, and the data platform does not send the synchronization result of the target business data to the data source after completing the data synchronization.

[0133] In one embodiment, the receiving module 806 is further configured to, after sending the first query request to the data platform, listen in real time for the synchronization result returned by the data platform; if the synchronization result requested by the first query request is not received after a preset duration, send a second query request carrying the data source end identifier and the service data identifier to the data platform; and receive the synchronization result obtained by the data platform based on the data source end identifier and the service data identifier carried in the second query request; wherein, the service data identifier is the data identifier of the service data for which the synchronization result requested by the first query request has not been received.

[0134] In one embodiment, the receiving module 806 is further configured to receive the synchronization result queried by the data platform from the stored synchronization result table based on the first query request; wherein, the synchronization result includes synchronization success, or synchronization failure and the reason for failure.

[0135] In one embodiment, the updating module 808 is further configured to, when the data synchronization identifier indicates that the service data synchronization fails, modify the data sending status of the service data to not sent, and modify the data synchronization identifier of the service data to synchronization failure; when the data synchronization identifier indicates that the service data synchronization succeeds, update the data synchronization identifier of the service data to synchronization success.

[0136] As Figure 10 shown, in one embodiment, a multi-data-source data synchronization device is provided. The device can be a software module, a hardware module, or a combination of both to form a part of a computer device. The device specifically includes: a receiving module 1002, a reading module 1004, and a query module 1006, wherein:

[0137] The receiving module 1002 is configured to receive service data of multiple data types sent by the data source end through different data channels;

[0138] The reading module 1004 is configured to store each service data in a data pool, and read the service data from the data pool according to the service priority for data synchronization to obtain a synchronization result;

[0139] The receiving module 1002 is further configured to receive a first query request sent by the data source end after waiting for a preset query interval;

[0140] The query module 1006 is configured to query the synchronization result based on the first query request, and return the synchronization result to the data source end, so that the data source end updates the data synchronization identifier of the service data based on the synchronization result, and the data synchronization identifier indicates whether the service data synchronization is successful.

[0141] In one embodiment, the reading module 1004 is further configured to store each piece of service data in a data pool; determine a time interval for data processing according to the processing efficiency of processing the service data; read a preset number of service data from the data pool based on the time interval; and synchronize the service data to a database to obtain a synchronization result.

[0142] The above data synchronization device for multiple data sources can reduce the logical steps for processing service data by receiving the service data sent by the data sources and storing the service data in the data pool; after waiting for a preset reading interval, read the service data according to the service priority for data synchronization to obtain a synchronization result; and can reasonably utilize the memory and avoid memory leakage. Therefore, when multiple data sources synchronize a large amount of data simultaneously, efficient data synchronization operations can be achieved.

[0143] For the specific limitations of the data synchronization device for multiple data sources, reference can be made to the limitations on the data synchronization method for multiple data sources in the above text, which will not be elaborated here. Each module in the above data synchronization device for multiple data sources can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0144] In one embodiment, a computer device is provided. The computer device can be a data source end, and its internal structure diagram can be as Figure 11 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external data source end in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. The computer program, when executed by the processor, implements a data synchronization method for multiple data sources. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0145] Those skilled in the art can understand that Figure 11The structure shown is only a block diagram of some of the structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component layout.

[0146] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0147] In one embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0148] In one embodiment, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device implements the steps in the above method embodiments.

[0149] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application may include at least one of non-volatile and volatile memories. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0150] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0151] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A data synchronization method for multiple data sources, characterized in that, The method includes: Sending business data of multiple data types through different data channels, so that a data platform receiving each piece of the business data stores each piece of the business data in a data pool, and reads the business data from the data pool according to business priorities for data synchronization to obtain a synchronization result; the business priorities are divided according to the urgency of the business data; After waiting for a preset query interval, sending a first query request to the data platform; Receiving the synchronization result queried by the data platform from a stored synchronization result table based on the first query request; wherein, the synchronization result includes synchronization success, or synchronization failure and the reason for failure; Updating the data synchronization identifier of the business data based on the synchronization result, and the data synchronization identifier indicates whether the business data is synchronized successfully; Wherein, in the case that the synchronization result includes synchronization failure and the reason for failure, the method further includes: Updating the business data with synchronization failure according to the reason for failure, and after the update is completed, sending the business data again.

2. The method according to claim 1, wherein After sending the business data of multiple data types through different data channels, it further includes: Modifying the data sending status of the business data to sent; Receiving result information sent by the data platform indicating whether the business data is received; Refusing to adjust the modified data sending status based on the result information.

3. The method according to claim 1, characterized in that, The storing each piece of the business data in a data pool, and reading the business data from the data pool according to business priorities for data synchronization to obtain a synchronization result includes: Storing each piece of the business data in a data pool; Determining a data processing time interval according to the processing efficiency of processing the business data; Reading a preset number of pieces of business data from the data pool based on the time interval; Synchronizing the business data to a database to obtain a synchronization result.

4. The method according to claim 1, wherein The method is applied to the data source side; The first query request carries a data source side identifier; The receiving the synchronization result queried by the data platform based on the synchronization result query request includes: Receiving the synchronization result of target business data queried by the data platform based on the data source side identifier; The target business data is the business data sent by the data source side, and the data platform does not send the synchronization result of the target business data to the data source side after completing data synchronization.

5. The method according to claim 4, wherein The method further includes: After sending the first query request to the data platform, monitoring in real time the synchronization result returned by the data platform; If the synchronization result requested by the first query request is not received after a preset duration, sending a second query request carrying the data source side identifier and the business data identifier to the data platform; Receiving the synchronization result obtained by the data platform based on the data source side identifier and the business data identifier carried by the second query request; Wherein, the business data identifier is the data identifier of the business data for which the synchronization result requested by the first query request is not received.

6. The method according to claim 1, characterized in that, The types of the business data include text type, digital type, and time type.

7. The method according to claim 1, characterized in that, Updating the data synchronization identifier of the service data based on the synchronization result includes: When the data synchronization identifier indicates that the synchronization of the service data fails, modify the data sending status of the service data to not sent, and modify the data synchronization identifier of the service data to synchronization failed; When the data synchronization identifier indicates that the synchronization of the service data is successful, update the data synchronization identifier of the service data to synchronization successful.

8. A data synchronization method for multiple data sources, characterized in that, The method includes: Receiving service data of multiple data types sent by a data source end through different data channels; Storing each piece of the service data in a data pool, and reading service data from the data pool according to service priorities for data synchronization to obtain a synchronization result; the service priorities are divided according to the urgency of the service data; Receiving a first query request sent by the data source end after waiting for a preset query interval; Querying the synchronization result based on the first query request, and returning the synchronization result to the data source end, so that the data source end updates the data synchronization identifier of the service data based on the synchronization result, where the data synchronization identifier indicates whether the synchronization of the service data is successful; wherein, the synchronization result includes synchronization successful, or synchronization failed and the reason for failure; Wherein, in the case that the synchronization result includes synchronization failed and the reason for failure, the method further includes: Receiving again the service data updated from the data source end for the service data with synchronization failed.

9. The method according to claim 8, wherein Storing each piece of the service data in the data pool, and reading service data from the data pool according to service priorities for data synchronization to obtain a synchronization result includes: Storing each piece of the service data in the data pool; Determining a data processing time interval according to the processing efficiency of processing the service data; Reading a preset number of service data from the data pool based on the time interval; Synchronizing the service data to a database to obtain a synchronization result.

10. A data synchronization device for multiple data sources, characterized in that, The device includes: A sending module, configured to send service data of multiple data types through different data channels, so that a data platform receiving each piece of the service data stores each piece of the service data in a data pool, and reads service data from the data pool according to service priorities for data synchronization to obtain a synchronization result; the service priorities are divided according to the urgency of the service data; A query module, configured to send a first query request to the data platform after waiting for a preset query interval; A receiving module, configured to receive the synchronization result queried by the data platform from a stored synchronization result table based on the first query request; wherein, the synchronization result includes synchronization successful, or synchronization failed and the reason for failure; An updating module, configured to update the data synchronization identifier of the service data based on the synchronization result, where the data synchronization identifier indicates whether the synchronization of the service data is successful; Wherein, in the case that the synchronization result includes synchronization failed and the reason for failure, the receiving module is further configured to update the service data with synchronization failed according to the reason for failure, and after the update is completed, send the service data again.

11. A data synchronization device for multiple data sources, characterized in that, The device includes: A receiving module, configured to receive service data of multiple data types sent by a data source end through different data channels; A reading module, configured to store each piece of the service data in a data pool, and read the service data from the data pool according to service priorities for data synchronization to obtain a synchronization result; the service priorities are divided according to the urgency of the service data; The receiving module is further configured to receive a first query request sent by the data source end after waiting for a preset query interval; A query module, configured to query the synchronization result based on the first query request, and return the synchronization result to the data source end, so that the data source end updates a data synchronization identifier of the service data based on the synchronization result, where the data synchronization identifier indicates whether the service data is synchronized successfully; wherein, the synchronization result includes synchronization success, or synchronization failure and a failure reason; Wherein, when the synchronization result includes synchronization failure and a failure reason, the receiving module receives again the service data after the service data with synchronization failure is updated from the data source end.

12. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product, characterized in that, The computer program product includes computer instructions, and when the computer instructions are executed, the steps of the method according to any one of claims 1 to 9 are implemented.

Images