A method and apparatus for synchronizing external data

Abstract

This specification relates to the field of computer technology and provides a method and apparatus for synchronizing external data. The method includes: determining the data to be synchronized and the corresponding external data source to establish a data synchronization task; determining the real-time requirements of the data synchronization task based on business needs; obtaining a first weight of the external data source based on the real-time requirements; acquiring historical synchronization task execution information corresponding to the external data source to obtain the busyness and second weight of the external data source; determining a corresponding priority scoring method based on the busyness to obtain a priority score for the external data source; adjusting the bandwidth of the external data source based on the priority score to determine a network bandwidth configuration strategy; obtaining a corresponding data synchronization method based on the real-time requirements, and synchronizing the data to be synchronized to the target database using the corresponding data synchronization method according to the network bandwidth configuration strategy. Through the embodiments of this specification, it is possible to efficiently and conveniently synchronize external data to a local target database.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for synchronizing external data. Background Technology

[0002] In complex projects that interact with multiple external data sources, it is often necessary to synchronize a continuous stream of different types of data from external data sources to the local machine via relevant interfaces. These external data sources are complex and varied, and their formats are often diverse. Current technologies typically only allow for inter-database synchronization between databases of the same type. Furthermore, since inter-database synchronization is achieved through file export, this method consumes significant disk space, requiring regular disk cleanup to maintain system operation. For data synchronization scenarios involving multiple external data sources of different types, the complexity of the external data sources themselves leads to low synchronization efficiency. Therefore, there is an urgent need for an external data synchronization method that can efficiently and conveniently synchronize external data to the local machine. Summary of the Invention

[0003] In view of the current situation, this solution is proposed to overcome or at least partially solve the above problems.

[0004] On the one hand, the purpose of some embodiments of this specification is to provide an external data synchronization method, the method comprising:

[0005] Receive data synchronization requests;

[0006] The data to be synchronized and the corresponding external data source are determined according to the data synchronization request, and a data synchronization task is established based on the external data source and the data to be synchronized.

[0007] Determine the real-time requirements of the data synchronization task based on the preset business needs;

[0008] Based on the real-time requirements of the data synchronization task, the first weight of the external data source is obtained;

[0009] Obtain historical synchronization task execution information of the external data source within a preset time interval;

[0010] The busyness of the external data source is calculated based on the historical synchronization task execution information, and the second weight of the external data source is obtained based on the busyness.

[0011] The corresponding priority scoring method is determined based on the busyness level, and the priority score of the external data source is calculated using the corresponding priority scoring method based on the first weight and the second weight.

[0012] The bandwidth of the external data source is adjusted according to the priority score so that the bandwidth of the external data source is positively correlated with the priority score, thereby determining the network bandwidth configuration strategy corresponding to the external data source.

[0013] Based on the real-time requirements, the data synchronization method corresponding to the data synchronization task is matched, and according to the network bandwidth configuration strategy, the data to be synchronized is synchronized to the target database using the data synchronization method corresponding to the data synchronization task.

[0014] Furthermore, the real-time requirements of the data synchronization task are determined based on pre-defined business needs, including:

[0015] Based on the preset business requirements, a set of external data sources with real-time requirements and a set of data to be synchronized with real-time requirements are obtained.

[0016] Using the external data source set and the data to be synchronized set, determine whether the external data source and the data to be synchronized corresponding to the data synchronization task have real-time requirements, and obtain the judgment result.

[0017] The data synchronization tasks are divided according to the judgment results, and the real-time requirements of the data synchronization tasks are determined based on the division results.

[0018] Furthermore, the data synchronization tasks are divided according to the judgment results, and the real-time requirements of the data synchronization tasks are determined based on the division results, including:

[0019] If both the external data source and the data to be synchronized corresponding to the data synchronization task have real-time requirements, then the data synchronization task is classified as the first level.

[0020] If neither the external data source nor the data to be synchronized corresponding to the data synchronization task has a real-time requirement, then the data synchronization task is classified as the second level.

[0021] If the external data source corresponding to the data synchronization task has real-time requirements, but the data to be synchronized corresponding to the data synchronization task does not have real-time requirements, then the data synchronization task is classified as the third level.

[0022] If the external data source corresponding to the data synchronization task does not have real-time requirements, but the data to be synchronized corresponding to the data synchronization task has real-time requirements, then the data synchronization task is classified as the fourth level.

[0023] Using the level corresponding to the data synchronization task and the size of the data to be synchronized, the real-time requirements of the data synchronization task are obtained by matching from a preset real-time requirement matching table.

[0024] Furthermore, based on the real-time requirements of the data synchronization task, a first weight of the external data source is obtained, including:

[0025] Statistics on all data synchronization tasks corresponding to each external data source;

[0026] Determine the real-time requirements of each data synchronization task in all data synchronization tasks corresponding to each external data source, as well as the real-time requirements of each external data source itself;

[0027] The real-time requirements of each data synchronization task in all data synchronization tasks corresponding to each external data source, as well as the real-time requirements of each external data source itself, are used as linear weighting factors to perform linear weighting operations to obtain the first weight of the external data source.

[0028] Furthermore, the historical synchronization task execution information includes one or more of the following: the synchronization task volume of the external data source, the synchronization data volume, the number of updates, the update data volume, and the number of accesses.

[0029] Further, the busyness of the external data source is calculated based on the historical synchronization task execution information, and a second weight of the external data source is obtained based on the busyness, including:

[0030] The historical synchronization task execution information is subjected to nonlinear decay processing using a logarithmic function;

[0031] The historical synchronous task execution information after nonlinear decay processing is used as a linear weighting factor for linear weighting calculation to obtain the busyness of the external data source;

[0032] The busyness of the external data source is used as the second weight of the external data source.

[0033] Further, a corresponding priority scoring method is determined based on the busyness level, and the priority score of the external data source is calculated using the corresponding priority scoring method based on the first weight and the second weight, including:

[0034] Determine whether the busy level exceeds a preset threshold;

[0035] If the busyness does not exceed a preset threshold, the second weight is used as a gain parameter, and the first weight and the second weight are linearly added together to obtain the calculation result.

[0036] If the busyness exceeds a preset threshold, the second weight is used as a decay parameter, and the first weight and the second weight are linearly subtracted to obtain the calculation result.

[0037] The calculation results are optimized based on the business importance of the external data source to obtain a priority score for the external data source.

[0038] Furthermore, the calculation results are optimized based on the business importance of the external data source to obtain a priority score for the external data source, which further includes:

[0039] The business importance of the external data source is determined based on the business requirements and the first weight.

[0040] The priority score of the external data source is obtained by adding the importance of the business to the calculation result.

[0041] Furthermore, the bandwidth of the external data source is adjusted according to the priority score so that the bandwidth of the external data source is positively correlated with the priority score, thereby determining the network bandwidth configuration strategy corresponding to the external data source, including:

[0042] Obtain communication status information from external data sources; the communication status information includes one or more of bandwidth, latency, jitter, packet loss rate, and storage utilization.

[0043] The bandwidth of all external data sources is adjusted according to the priority score so that the bandwidth of the external data sources is positively correlated with the priority score, and the specified type of data in the communication status information is optimized according to the adjusted bandwidth.

[0044] The data transmission cost of each external data source is calculated using the optimized communication status information, and the data transmission costs of each external data source are added together to obtain the total data transmission cost.

[0045] Repeat the steps described above to adjust the bandwidth of all external data sources according to the priority score to obtain the total data transmission cost, thereby obtaining multiple total data transmission costs;

[0046] Select the minimum total data transmission cost from multiple total data transmission costs, and determine the network bandwidth configuration strategy based on the bandwidth of the external data source corresponding to the minimum total data transmission cost.

[0047] Furthermore, the specified type of data in the communication status information is optimized based on the adjusted bandwidth, including:

[0048] Determine whether the adjusted bandwidth is less than the preset minimum bandwidth;

[0049] If the adjusted bandwidth is not less than the preset minimum bandwidth, then there is no need to adjust the specified type of data in the communication status information; the specified type of data includes one or more of latency, jitter and packet loss rate.

[0050] If the adjusted bandwidth is less than the preset minimum bandwidth, then the value of the specified type of data is optimized using the preset optimization gradient.

[0051] Furthermore, the data synchronization methods include incremental synchronization, full synchronization, and a hybrid full-incremental synchronization.

[0052] On the other hand, some embodiments of this specification also provide an external data synchronization device, the device comprising:

[0053] The receiving module is used to receive data synchronization requests;

[0054] The determination module is used to determine the data to be synchronized and the corresponding external data source according to the data synchronization request, so as to establish a data synchronization task based on the external data source and the data to be synchronized;

[0055] The classification module is used to determine the real-time requirements of data synchronization tasks based on preset business needs;

[0056] The first sorting module is used to obtain the first weight of the external data source based on the real-time requirements of the data synchronization task;

[0057] The acquisition module is used to acquire historical synchronization task execution information of the external data source within a preset time interval;

[0058] The second sorting module is used to calculate the busyness of the external data source based on the historical synchronization task execution information, and obtain the second weight of the external data source;

[0059] The evaluation module is used to determine the corresponding priority scoring method based on the busyness, and to calculate the priority score of the external data source based on the first weight and the second weight using the corresponding priority scoring method.

[0060] The matching module is used to adjust the bandwidth of the external data source according to the priority score, so that the bandwidth of the external data source is positively correlated with the priority score, thereby determining the network bandwidth configuration strategy corresponding to the external data source.

[0061] The synchronization module is used to match the data synchronization method corresponding to the data synchronization task based on the real-time requirements, and to synchronize the data to be synchronized to the target database using the data synchronization method corresponding to the data synchronization task according to the network bandwidth configuration strategy.

[0062] On the other hand, some embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, which, when run by the processor, executes instructions for the methods described above.

[0063] On the other hand, some embodiments of this specification also provide a computer storage medium having a computer program stored thereon, which, when run by the processor of a computer device, executes instructions for the methods described above.

[0064] On the other hand, some embodiments of this specification also provide a computer program product, which includes a computer program that, when run by the processor of a computer device, executes instructions for the methods described above.

[0065] Some embodiments of this specification provide one or more technical solutions, which have at least the following technical effects:

[0066] The embodiments of this specification first automatically receive data synchronization requests, determine the data to be synchronized and the corresponding external data sources based on the data synchronization requests, establish data synchronization tasks, and determine the real-time requirements of the data synchronization tasks according to preset business needs. Based on these real-time requirements, a first weight of the external data source is obtained, which is then used to numerically measure the real-time performance of each data synchronization task. Simultaneously, historical synchronization task execution information of the external data sources within a preset time interval is obtained, and the busy level of each external data source is calculated based on this historical execution information to obtain a corresponding second weight. This second weight is then used to numerically measure the busy level of the external data sources. Finally, the external data is calculated based on the first and second weights. The priority score is determined by the busy level of the external data source. This is because the efficiency of data synchronization is affected when external data sources with different busy levels are synchronized. After calculating the priority score, the bandwidth of the external data source is allocated and adjusted according to the priority score so that the bandwidth of the external data source is positively correlated with the priority score. The network bandwidth configuration strategy corresponding to the external data source is determined, and the data synchronization method corresponding to the data synchronization task is matched according to the real-time requirements. This enables the automatic data synchronization process to be achieved efficiently and quickly when synchronizing the data to be synchronized from multiple external data sources to the target database using the corresponding data synchronization method and network bandwidth configuration strategy.

[0067] The above description is merely an overview of some embodiments of the technical solutions in this specification. In order to better understand the technical means of some embodiments of this specification and to implement them in accordance with the content of the specification, and to make the above and other objects, features and advantages of some embodiments of this specification more apparent and understandable, specific implementation methods of some embodiments of this specification are given below. Attached Figure Description

[0068] To more clearly illustrate some embodiments or technical solutions in the prior art of this specification, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. In the drawings:

[0069] Figure 1 A schematic diagram of an implementation system for an external data synchronization method in some embodiments of this specification is shown;

[0070] Figure 2 A flowchart of an external data synchronization method is shown in some embodiments of this specification;

[0071] Figure 3 This diagram illustrates the steps involved in determining the real-time requirements of a data synchronization task in some embodiments of this specification.

[0072] Figure 4 This is a schematic diagram illustrating the steps for obtaining the first weight of an external data source in some embodiments of this specification;

[0073] Figure 5 This is a schematic diagram illustrating the steps for obtaining the second weight of an external data source in some embodiments of this specification;

[0074] Figure 6 This is a schematic diagram illustrating the steps for obtaining priority scores from external data sources in some embodiments of this specification;

[0075] Figure 7 This is a schematic diagram illustrating the steps for determining a network bandwidth configuration strategy in some embodiments of this specification;

[0076] Figure 8 This is a schematic diagram of the structure of an external data synchronization device in some embodiments of this specification;

[0077] Figure 9 This is a schematic diagram of the computer device structure provided in some embodiments of this specification.

[0078] [Explanation of Labels in the Attached Image]

[0079] 101. Terminal;

[0080] 102. Server;

[0081] 801. Receiver module;

[0082] 802. Determine the module;

[0083] 803. Classification Module;

[0084] 804, First sorting module;

[0085] 805. Acquisition Module;

[0086] 806. Second sorting module;

[0087] 807. Evaluation Module;

[0088] 808. Matching module;

[0089] 809. Synchronization Module;

[0090] 902. Computer equipment;

[0091] 904, Processor;

[0092] 906. Memory;

[0093] 908. Drive mechanism;

[0094] 910. Input / output interfaces;

[0095] 912. Input devices;

[0096] 914. Output devices;

[0097] 916. Presentation equipment;

[0098] 918. Graphical User Interface;

[0099] 920. Network interface;

[0100] 922. Communication link;

[0101] 924. Communication bus. Detailed Implementation

[0102] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in some embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on some embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0103] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0104] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of relevant laws and regulations.

[0105] It should be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0106] like Figure 1 The diagram illustrates an implementation system for an external data synchronization method according to an embodiment of the present invention. The system may include a terminal 101 and a server 102. The terminal 101 and server 102 communicate via a network, which may include a Local Area Network (LAN), a Wide Area Network (WAN), the Internet, or a combination thereof, and is connected to a website, user equipment (e.g., computing devices), and a backend system. Staff can send a data synchronization request to the server 102 via the terminal 101. Upon receiving the data synchronization request, the server 102 determines the data to be synchronized and the corresponding external data source based on the request, thereby establishing a data synchronization task. Based on this task, the data to be synchronized from the external data source is then synchronized to the target database.

[0107] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, it may include multiple terminals 101, and this specification does not impose any restrictions.

[0108] Figure 2This is a flowchart illustrating an external data synchronization method provided by an embodiment of the present invention. This specification provides the operational steps of the method described in the embodiments or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods can be executed sequentially or in parallel according to the embodiments or accompanying drawings. Specifically, as shown... Figure 2 As shown, when applied to the server side described above, the method may include:

[0109] S201: Receive data synchronization request;

[0110] S202: Determine the data to be synchronized and the corresponding external data source according to the data synchronization request, and establish a data synchronization task based on the external data source and the data to be synchronized;

[0111] S203: Determine the real-time requirements of the data synchronization task based on the preset business needs;

[0112] S204: Based on the real-time requirements of the data synchronization task, obtain the first weight of the external data source;

[0113] S205: Obtain historical synchronization task execution information of the external data source within a preset time interval;

[0114] S206: Calculate the busyness of the external data source based on the historical synchronization task execution information, and obtain the second weight of the external data source based on the busyness;

[0115] S207: Determine the corresponding priority scoring method based on the busyness level, and calculate the priority score of the external data source using the corresponding priority scoring method based on the first weight and the second weight.

[0116] S208: Adjust the bandwidth of the external data source according to the priority score so that the bandwidth of the external data source is positively correlated with the priority score, thereby determining the network bandwidth configuration strategy corresponding to the external data source;

[0117] S209: Based on the real-time requirements, the data synchronization method corresponding to the data synchronization task is matched, and according to the network bandwidth configuration strategy, the data to be synchronized is synchronized to the target database using the data synchronization method corresponding to the data synchronization task.

[0118] The embodiments of this specification first automatically receive data synchronization requests, determine the data to be synchronized and the corresponding external data sources based on the data synchronization requests, establish data synchronization tasks, and determine the real-time requirements of the data synchronization tasks according to preset business needs. Based on these real-time requirements, a first weight of the external data source is obtained, which is then used to numerically measure the real-time performance of each data synchronization task. Simultaneously, historical synchronization task execution information of the external data sources within a preset time interval is obtained, and the busy level of each external data source is calculated based on this historical execution information to obtain a corresponding second weight. This second weight is then used to numerically measure the busy level of the external data sources. Finally, the external data is calculated based on the first and second weights. The priority score is determined by the busy level of the external data source. This is because the efficiency of data synchronization is affected when external data sources with different busy levels are synchronized. After calculating the priority score, the bandwidth of the external data source is allocated and adjusted according to the priority score so that the bandwidth of the external data source is positively correlated with the priority score. The network bandwidth configuration strategy corresponding to the external data source is determined, and the data synchronization method corresponding to the data synchronization task is matched according to the real-time requirements. This enables the automatic data synchronization process to be achieved efficiently and quickly when synchronizing the data to be synchronized from multiple external data sources to the target database using the corresponding data synchronization method and network bandwidth configuration strategy.

[0119] In some embodiments, the synchronization between the external data source and the target database can be database synchronization, file synchronization, or API call synchronization. The external data source stores the data to be synchronized that the target database needs. In addition to data synchronization, the external data source itself also has other tasks to perform, such as writing new data or modifying old data. When multiple external data sources need to synchronize data with the target database at the same time, and given the limited total bandwidth, it is necessary to formulate an appropriate network bandwidth configuration strategy and use an appropriate data synchronization method to accurately, efficiently, and orderly synchronize all the data to be synchronized to the target database.

[0120] See attached document Figure 3 In some embodiments, determining the real-time requirements of the data synchronization task based on preset business needs may include:

[0121] S301: Based on the preset business requirements, obtain the set of external data sources with real-time requirements and the set of data to be synchronized with real-time requirements;

[0122] S302: Using the external data source set and the data to be synchronized set, determine whether the external data source and the data to be synchronized corresponding to the data synchronization task have real-time requirements, and obtain the judgment result;

[0123] S303: Divide the data synchronization task according to the judgment result, and determine the real-time requirements of the data synchronization task according to the division result.

[0124] It can be understood that in some embodiments, the real-time requirements of external data sources and the real-time requirements of data to be synchronized are preset in advance according to the business requirements. Through the preset business requirements, the set of external data sources with real-time requirements and the set of data to be synchronized with real-time requirements can be obtained quickly. Then, using the set of external data sources and the set of data to be synchronized, it can be determined whether the external data source and the data to be synchronized corresponding to the data synchronization task to be executed have real-time requirements, and the judgment result can be obtained. Based on the judgment result, the data synchronization task can be quickly and accurately classified according to the real-time requirements.

[0125] Furthermore, in some embodiments, the data synchronization task is divided according to the judgment result, and the real-time requirements of the data synchronization task are determined according to the division result, including:

[0126] If both the external data source and the data to be synchronized corresponding to the data synchronization task have real-time requirements, then the data synchronization task is classified as the first level.

[0127] If neither the external data source nor the data to be synchronized corresponding to the data synchronization task has a real-time requirement, then the data synchronization task is classified as the second level.

[0128] If the external data source corresponding to the data synchronization task has real-time requirements, but the data to be synchronized corresponding to the data synchronization task does not have real-time requirements, then the data synchronization task is classified as the third level.

[0129] If the external data source corresponding to the data synchronization task does not have real-time requirements, but the data to be synchronized corresponding to the data synchronization task has real-time requirements, then the data synchronization task is classified as the fourth level.

[0130] Using the level corresponding to the data synchronization task and the size of the data to be synchronized, the real-time requirements of the data synchronization task are obtained by matching from a preset real-time requirement matching table.

[0131] In some embodiments, if both the external data source and the data to be synchronized corresponding to the current data synchronization task have real-time requirements, it indicates that the real-time requirements of the current data synchronization task are high, and the current data synchronization task is classified as Level 1. If neither the external data source nor the data to be synchronized corresponding to the current data synchronization task has real-time requirements, it indicates that the real-time requirements of the current data synchronization task are low, and the current data synchronization task is classified as Level 4. Correspondingly, when the external data source corresponding to the data synchronization task does not have real-time requirements, but the data to be synchronized corresponding to the data synchronization task has real-time requirements, the data synchronization task is classified as Level 4. When the external data source corresponding to the data synchronization task has real-time requirements, but the data to be synchronized corresponding to the data synchronization task does not have real-time requirements, the data synchronization task is classified as Level 3. Among these, Level 1 data synchronization tasks have the highest real-time requirements, and Level 4 data synchronization tasks have the lowest real-time requirements. The relative importance of Level 2 and Level 3 data synchronization tasks can be adjusted according to business needs. Generally, it can be considered that Level 2 data synchronization tasks have higher real-time requirements than Level 3 data synchronization tasks, thereby achieving rapid and refined differentiation of the real-time requirements of data synchronization tasks.

[0132] See attached document Figure 4 In some embodiments, obtaining a first weight for the external data source based on the real-time requirements of the data synchronization task may include:

[0133] S401: Count all data synchronization tasks corresponding to each external data source;

[0134] S402: Determine the real-time requirements of each data synchronization task in all data synchronization tasks corresponding to each external data source, as well as the real-time requirements of each external data source itself;

[0135] S403: Take the real-time requirements of each data synchronization task in all data synchronization tasks corresponding to each external data source, as well as the real-time requirements of each external data source itself, as linear weighting factors, and perform linear weighting operations to obtain the first weight of the external data source.

[0136] In some embodiments, after obtaining the real-time requirements of the data synchronization task, it is necessary to quantify these requirements. Using external data sources as a benchmark, the real-time requirements of each external data source are measured by a first weight. Specifically, firstly, all data synchronization tasks corresponding to each external data source need to be statistically analyzed. If, for an external data source X, there are more data synchronization tasks with high real-time requirements among all data synchronization tasks corresponding to X, then the real-time requirements of external data source X are higher. Furthermore, each external data source itself also has real-time requirements. Then, the real-time requirements of each data synchronization task among all data synchronization tasks corresponding to each external data source, along with the real-time requirements of each external data source itself, are used as linear weighting factors for linear weighting calculation to obtain the first weight of the external data source. Specifically, in some embodiments, when performing linear weighting calculation, the real-time requirements of a data synchronization task can be matched from a preset real-time requirement matching table based on the level of the data synchronization task and the size of the data to be synchronized, thereby achieving fast and accurate calculation of the first weight of the external data source.

[0137] Furthermore, in some embodiments, the historical synchronization task execution information includes one or more of the following: the synchronization task volume of the external data source, the synchronization data volume, the number of updates, the update data volume, and the number of accesses.

[0138] In some embodiments, historical synchronization task execution information often refers to data from the most recent time interval, thus ensuring the accuracy of subsequent calculations regarding the busyness of external data sources. Here, the synchronization task volume refers to the total number of data synchronization tasks, the synchronized data volume refers to the total amount of data synchronized to the target database, the update count refers to the number of times the external data source updates data, the update data volume refers to the total amount of data updated by the external data source, and the access count refers to the total number of times the external data source is accessed. In some embodiments, historical synchronization task execution information can also be other types of data, which is not limited herein.

[0139] See attached document Figure 5 In some embodiments, calculating the busyness of the external data source based on the historical synchronization task execution information and obtaining a second weight of the external data source based on the busyness may include:

[0140] S501: The historical synchronization task execution information is subjected to nonlinear decay processing using a logarithmic function;

[0141] S502: The historical synchronous task execution information after nonlinear decay processing is used as a linear weighting factor for linear weighting operation to obtain the busyness of the external data source;

[0142] S503: Use the busyness of the external data source as the second weight of the external data source.

[0143] In some embodiments, due to the large magnitude of the historical synchronization task execution information, a non-linear decay process is first applied to this information using a logarithmic function. Typically, the natural logarithm function is used to take the natural logarithm of the data in the historical synchronization task execution information to achieve this non-linear decay. Then, the non-linearly decayed historical synchronization task execution information is used as a linear weighting factor for linear weighting to obtain the busyness of the external data source. This busyness measures the degree of activity of the external data source in data synchronization, access, and updates, thus using the busyness as the second weight of the external data source to achieve fast and accurate calculation of the second weight.

[0144] Furthermore, in some embodiments, a corresponding priority scoring method is determined based on the busyness level, and the priority score of the external data source is calculated using the corresponding priority scoring method based on the first weight and the second weight, including:

[0145] Determine whether the busy level exceeds a preset threshold;

[0146] If the busyness does not exceed a preset threshold, the second weight is used as a gain parameter, and the first weight and the second weight are linearly added together to obtain the calculation result.

[0147] If the busyness exceeds a preset threshold, the second weight is used as a decay parameter, and the first weight and the second weight are linearly subtracted to obtain the calculation result.

[0148] The calculation results are optimized based on the business importance of the external data source to obtain a priority score for the external data source.

[0149] In some embodiments, after obtaining the first weight and the second weight, it is necessary to determine an appropriate priority score. First, check if the busyness exceeds a preset threshold. If the busyness does not exceed the preset threshold, it means that the external data source has free time to perform data synchronization tasks. In this case, the second weight corresponding to the data source is used as a gain parameter, and the first weight and the second weight are linearly added to increase the priority score of the external data source. If the busyness does not exceed the preset threshold, it means that the external data source does not have free time to perform data synchronization tasks. Therefore, the first weight and the second weight are linearly subtracted to decrease the priority score of the external data source, thereby achieving fast and automatic calculation of the priority score of the external data source.

[0150] Furthermore, in some embodiments, after obtaining the calculation result, refer to the appendix. Figure 6In some embodiments, the calculation results are optimized based on the business importance of the external data source to obtain a priority score for the external data source, which may further include:

[0151] S601: Determine the business importance of the external data source based on the business requirements and the first weight;

[0152] S602: Add the importance of the business to the calculation result to obtain the priority score of the external data source.

[0153] In some embodiments, it can be understood that the set of external data sources with real-time requirements and the set of data to be synchronized can be obtained through business requirements. The first weight is used to quantitatively evaluate the real-time performance of the external data sources. Based on this, the business importance of the external data sources can be determined according to the business requirements and the level of the first weight. The business importance is regarded as an adjustment parameter and added to the calculation result to quickly and accurately obtain the priority score of the external data sources, so as to ensure the effectiveness of the subsequent determination of network bandwidth configuration.

[0154] See attached document Figure 7 In some embodiments, adjusting the bandwidth of the external data source based on the priority score to make the bandwidth of the external data source positively correlated with the priority score, thereby determining the network bandwidth configuration strategy corresponding to the external data source, may include:

[0155] S701: Obtain communication status information from external data sources;

[0156] The communication status information includes one or more of the following: bandwidth, latency, jitter and packet loss rate, and storage utilization.

[0157] S702: Adjust the bandwidth of all external data sources according to the priority score so that the bandwidth of the external data sources is positively correlated with the priority score, and optimize the specified type of data in the communication status information according to the adjusted bandwidth;

[0158] S703: Calculate the data transmission cost of each external data source using the optimized communication status information, and add up the data transmission costs of each external data source to obtain the total data transmission cost;

[0159] S704: Repeat the steps above to adjust the bandwidth of all external data sources according to the priority score to obtain the total data transmission cost, thereby obtaining multiple total data transmission costs;

[0160] S705: Select the minimum total data transmission cost from multiple total data transmission costs, and determine the network bandwidth configuration strategy based on the bandwidth of the external data source corresponding to the minimum total data transmission cost.

[0161] This can be understood as follows: In some embodiments, when adjusting the bandwidth of external data sources according to the priority score, the communication status information of the external data sources is first obtained. The communication status information can be used to assess the current communication connectivity of the external data sources. Then, the bandwidth of all external data sources is adjusted according to the priority score so that the bandwidth of the external data sources is positively correlated with the priority score. Furthermore, the specified type of data in the communication status information is optimized based on the adjusted bandwidth. In some embodiments, optimizing the specified type of data in the communication status information based on the adjusted bandwidth may include:

[0162] Determine whether the adjusted bandwidth is less than the preset minimum bandwidth;

[0163] If the adjusted bandwidth is not less than the preset minimum bandwidth, then there is no need to adjust the specified type of data in the communication status information; the specified type of data includes one or more of latency, jitter and packet loss rate.

[0164] If the adjusted bandwidth is less than the preset minimum bandwidth, then the value of the specified type of data is optimized using the preset optimization gradient.

[0165] In some embodiments, if the adjusted bandwidth is not less than the preset minimum bandwidth, it means that the bandwidth adjustment has not had a significant impact on the specified type of data. In this case, the adjusted bandwidth is still compatible with information such as latency, jitter, and packet loss rate, and there is no need to adjust the specified type of data in the communication status information. However, if the adjusted bandwidth is less than the preset minimum bandwidth, the bandwidth will affect the specified type of data, such as increased latency, increased jitter rate, and increased packet loss rate. Therefore, it is necessary to optimize the value of the specified type of data so that the specified type of data in the communication status information can also change accordingly with the bandwidth adjustment, ensuring the accuracy of subsequent calculation of data transmission costs.

[0166] Furthermore, in some embodiments, the data transmission cost of each external data source can be calculated using the optimized communication state information according to the following formula:

[0167]

[0168] Where time represents latency, bandwidth represents bandwidth, jitter represents jitter rate, drop represents packet loss rate, a, b, c, d, and f represent parameters for adjusting latency, bandwidth, jitter rate, packet loss rate, and bandwidth inverse effect, respectively, k represents the exponential parameter, and e represents the natural constant.

[0169] Next, the data transmission costs of each external data source are summed to obtain the total data transmission cost. Since it is uncertain whether the current network bandwidth allocation strategy is optimal, the steps described above, adjusting the bandwidth of all external data sources based on the priority score, need to be repeated to obtain the total data transmission cost, resulting in multiple total data transmission costs. This can be understood as treating the total data transmission cost as an objective function value, with the optimization objective being to minimize this objective function value, i.e., minimizing the total data transmission cost. Based on the bandwidth of the external data source corresponding to the minimum total data transmission cost, a network bandwidth configuration strategy is determined to ensure the effectiveness of the network bandwidth configuration strategy, facilitating the accurate and efficient synchronization of the data to be synchronized to the target database.

[0170] Furthermore, in some embodiments, the data synchronization method includes incremental synchronization, full synchronization, and a hybrid full-incremental synchronization.

[0171] In some embodiments, a preset data synchronization method matching table can be used to match the data synchronization method corresponding to the data synchronization task based on real-time requirements. For example, full synchronization first clears the corresponding local data and then inserts all external data. Incremental synchronization, the external data source is updated daily, and the corresponding local data is not cleared. The external data source data is synchronized by updating if it exists and inserting if it doesn't. A hybrid full-incremental synchronization method does not clear local data, and the external data source data is synchronized by updating if it exists and inserting if it doesn't. Once the data synchronization method corresponding to the data synchronization task is determined, the data synchronization task is executed. Based on the network bandwidth configuration policy, the corresponding data synchronization method is used to efficiently and conveniently synchronize the data to be synchronized to the target database.

[0172] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0173] Corresponding to the external data synchronization method described above, some embodiments of this specification also provide an external data synchronization device, see reference. Figure 8 As shown, in some embodiments, the apparatus may include:

[0174] Receiver module 801 is used to receive data synchronization requests;

[0175] The determining module 802 is used to determine the data to be synchronized and the corresponding external data source according to the data synchronization request, so as to establish a data synchronization task based on the external data source and the data to be synchronized;

[0176] The classification module 803 is used to determine the real-time requirements of the data synchronization task based on preset business needs;

[0177] The first sorting module 804 is used to obtain the first weight of the external data source based on the real-time requirements of the data synchronization task.

[0178] The acquisition module 805 is used to acquire historical synchronization task execution information of the external data source within a preset time interval;

[0179] The second sorting module 806 is used to calculate the busyness of the external data source based on the historical synchronization task execution information, and obtain the second weight of the external data source.

[0180] The evaluation module 807 is used to determine the corresponding priority scoring method based on the busyness, and to calculate the priority score of the external data source based on the first weight and the second weight using the corresponding priority scoring method.

[0181] The matching module 808 is used to adjust the bandwidth of the external data source according to the priority score, so that the bandwidth of the external data source is positively correlated with the priority score, thereby determining the network bandwidth configuration strategy corresponding to the external data source.

[0182] The synchronization module 809 is used to match the data synchronization method corresponding to the data synchronization task based on the real-time requirements, and synchronize the data to be synchronized to the target database using the data synchronization method corresponding to the data synchronization task according to the network bandwidth configuration strategy.

[0183] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0184] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized and agreed upon by the user and fully authorized by all parties.

[0185] It should be noted that the computer program product described in this specification is a software product that mainly implements the methods described in this specification through a computer program.

[0186] Embodiments of this specification also provide a computer device. For example... Figure 9As shown, in some embodiments of this specification, the computer device 902 may include one or more processors 904, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each processing unit implementing one or more hardware threads. The computer device 902 may also include any memory 906 for storing information of any kind, such as code, settings, data, etc. In one specific embodiment, a computer program is stored on the memory 906 and can run on the processor 904. When the computer program is run by the processor 904, it can execute instructions described in any of the above embodiments. Without limitation, for example, the memory 906 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 902. In one case, when the processor 904 executes associated instructions stored in any memory or combination of memories, the computer device 902 can perform any operation of the associated instructions. The computer device 902 also includes one or more drive mechanisms 908 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0187] Computer device 902 may also include an input / output interface 910 (I / O) for receiving various inputs (via input device 912) and providing various outputs (via output device 914). A specific output mechanism may include a presentation device 916 and an associated graphical user interface 918 (GUI). In other embodiments, the input / output interface 910 (I / O), input device 912, and output device 914 may be omitted, and the device may function solely as a computer device within a network. Computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the components described above together.

[0188] Communication link 922 can be implemented in any way, such as via a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0189] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), computer-readable storage media, and computer program products according to some embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create a mechanism for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions may also be loaded onto a computer or other programmable data processor, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0193] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0194] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by computer equipment. As defined in this specification, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0195] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processors connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0197] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0198] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0199] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0200] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An external data synchronization method, characterized by, The method comprises: receiving a data synchronization request; determining to-be-synchronized data and a corresponding external data source according to the data synchronization request, to establish a data synchronization task based on the external data source and the to-be-synchronized data; determining a real-time requirement of the data synchronization task according to a preset business requirement; obtaining a first weight of the external data source based on the real-time requirement of the data synchronization task; obtaining historical synchronization task execution information of the external data source within a preset time interval; calculating a busy degree of the external data source according to the historical synchronization task execution information, and obtaining a second weight of the external data source according to the busy degree; determining a corresponding priority scoring mode according to the busy degree, and calculating a priority score of the external data source by using the corresponding priority scoring mode according to the first weight and the second weight; adjusting a bandwidth of the external data source according to the priority score, so that the bandwidth size of the external data source is positively correlated with the priority score, thereby determining a network bandwidth configuration strategy corresponding to the external data source; matching a data synchronization mode corresponding to the data synchronization task based on the real-time requirement, and synchronizing the to-be-synchronized data to a target database by using the data synchronization mode corresponding to the data synchronization task according to the network bandwidth configuration strategy.

2. The method of claim 1, wherein, Determining a real-time requirement of the data synchronization task according to a preset business requirement comprises: obtaining a set of external data sources with real-time requirements and a set of to-be-synchronized data with real-time requirements according to the preset business requirement; judging whether the external data source and the to-be-synchronized data corresponding to the data synchronization task have real-time requirements by using the set of external data sources and the set of to-be-synchronized data, to obtain a judgment result; dividing the data synchronization task according to the judgment result, and determining the real-time requirement of the data synchronization task according to the division result.

3. The method of claim 2, wherein, Dividing the data synchronization task according to the judgment result, and determining the real-time requirement of the data synchronization task according to the division result comprises: if the external data source and the to-be-synchronized data corresponding to the data synchronization task both have real-time requirements, dividing the data synchronization task into a first level; if the external data source and the to-be-synchronized data corresponding to the data synchronization task both do not have real-time requirements, dividing the data synchronization task into a second level; if the external data source corresponding to the data synchronization task has a real-time requirement, and the to-be-synchronized data corresponding to the data synchronization task does not have a real-time requirement, dividing the data synchronization task into a third level; if the external data source corresponding to the data synchronization task does not have a real-time requirement, and the to-be-synchronized data corresponding to the data synchronization task has a real-time requirement, dividing the data synchronization task into a fourth level; matching the real-time requirement of the data synchronization task from a preset real-time requirement matching table by using the level corresponding to the data synchronization task and the size of the to-be-synchronized data.

4. The method of claim 3, wherein, Obtaining a first weight of the external data source based on the real-time requirement of the data synchronization task comprises: counting all data synchronization tasks corresponding to each external data source; Determine the real-time requirement of each data synchronization task corresponding to each external data source, and the real-time requirement of each external data source itself; Linearly weight the real-time requirement of each data synchronization task corresponding to each external data source, and the real-time requirement of each external data source itself as a linear weighting factor, to obtain the first weight of the external data source.

5. The method of claim 1, wherein, The historical synchronization task execution information includes one or more of the synchronization task amount, the synchronization data amount, the update frequency, the update data amount, and the access frequency of the external data source.

6. The method of claim 5, wherein, According to the historical synchronization task execution information, calculate the busy degree of the external data source, and according to the busy degree, obtain the second weight of the external data source, including: Nonlinearly attenuate the historical synchronization task execution information by using a logarithmic function; Linearly weight the nonlinearly attenuated historical synchronization task execution information as a linear weighting factor to obtain the busy degree of the external data source; The busy degree of the external data source is taken as the second weight of the external data source.

7. The method of claim 1, wherein, According to the busy degree, determine the corresponding priority scoring method, and according to the first weight and the second weight, calculate the priority score of the external data source by using the corresponding priority scoring method, including: Determine whether the busy degree exceeds a preset threshold; If the busy degree does not exceed the preset threshold, take the second weight as a gain parameter, linearly add the first weight and the second weight to obtain an operation result; If the busy degree exceeds the preset threshold, take the second weight as an attenuation parameter, linearly subtract the first weight and the second weight to obtain an operation result; According to the business importance of the external data source, optimize the operation result to obtain the priority score of the external data source.

8. The method of claim 7, wherein, According to the business importance of the external data source, optimize the operation result to obtain the priority score of the external data source, further including: According to the business requirement and the first weight, determine the business importance of the external data source; Add the business importance to the operation result to obtain the priority score of the external data source.

9. The method of claim 1, wherein, According to the priority score, adjust the bandwidth of the external data source, so that the bandwidth size of the external data source is positively correlated with the priority score, thereby determining the network bandwidth configuration strategy corresponding to the external data source, including: Obtain the communication state information of the external data source; the communication state information includes one or more of bandwidth, time delay, jitter, packet loss rate, and storage utilization; According to the priority score, adjust the bandwidth of all external data sources, so that the bandwidth size of the external data source is positively correlated with the priority score, and optimize the specified type of data in the communication state information according to the adjusted bandwidth; Calculate the data transmission cost of each external data source by using the optimized communication state information, and add the data transmission cost of each external data source to obtain the total data transmission cost; The step of adjusting the bandwidth of all external data sources according to the priority score is repeated to obtain a plurality of total data transmission costs; The minimum total data transmission cost is selected from the plurality of total data transmission costs, and a network bandwidth configuration strategy is determined according to the bandwidth of the external data source corresponding to the minimum total data transmission cost.

10. The method of claim 9, wherein, The specified type of data in the communication state information is optimized according to the adjusted bandwidth, including: determining whether the adjusted bandwidth is less than a preset minimum bandwidth; if the adjusted bandwidth is not less than the preset minimum bandwidth, the specified type of data in the communication state information does not need to be adjusted; the specified type of data includes one or more of latency, jitter and packet loss rate; if the adjusted bandwidth is less than the preset minimum bandwidth, the value of the specified type of data is optimized using a preset optimization gradient.

11. The method of claim 1, wherein, The data synchronization mode includes incremental synchronization, full-amount synchronization, and full-amount-incremental mixed synchronization.

12. An external data synchronization apparatus, comprising: The device includes: a receiving module for receiving a data synchronization request; a determining module for determining the data to be synchronized and the corresponding external data source based on the data synchronization request, and establishing a data synchronization task based on the external data source and the data to be synchronized; a classification module for determining the real-time requirement of the data synchronization task according to the preset business requirement; a first sorting module for obtaining the first weight of the external data source based on the real-time requirement of the data synchronization task; an obtaining module for obtaining the historical synchronization task execution information of the external data source in a preset time interval; a second sorting module for calculating the busy degree of the external data source based on the historical synchronization task execution information, and obtaining the second weight of the external data source; an evaluation module for determining the corresponding priority scoring method according to the busy degree, and calculating the priority score of the external data source using the corresponding priority scoring method based on the first weight and the second weight; a matching module for adjusting the bandwidth of the external data source according to the priority score, so that the bandwidth of the external data source is positively correlated with the priority score, thereby determining the network bandwidth configuration strategy corresponding to the external data source; a synchronization module for matching the data synchronization mode corresponding to the data synchronization task based on the real-time requirement, and synchronizing the data to be synchronized to the target database using the data synchronization mode corresponding to the data synchronization task according to the network bandwidth configuration strategy.

13. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein, The computer program is run by the processor to execute the instructions of the method according to any one of claims 1-11.

14. A computer storage medium having stored thereon a computer program, characterized in that The computer program is run by the processor of the computer device to execute the instructions of the method according to any one of claims 1-11.

15. A computer program product, characterised in that, The computer program product includes a computer program, which is run by the processor to execute the instructions of the method according to any one of claims 1-11.

Images