Method, apparatus, device and computer readable medium for synchronizing data

Abstract

The application discloses a method, device and equipment for synchronizing data and a computer readable medium, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: detecting the network residual bandwidth between a database source server and a database target server; setting the initial number of data transmission channels based on a linear regression model and the network residual bandwidth, wherein the linear regression model is obtained by training historical network residual bandwidth and historical number of data transmission channels; locating the number interval of the data transmission channels according to the initial number, and taking the maximum value of the number interval as the synchronization number of the data transmission channels; starting the task of synchronizing data from the database source server to the database target server by using the synchronization number of data transmission channels, and synchronizing data based on the network residual bandwidth. The embodiment can accelerate the speed of data synchronization.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and computer-readable medium for synchronizing data. Background Technology

[0002] Big data platforms often need to access and push data between multiple data sources through offline pipelines.

[0003] In the process of developing this invention, the inventors discovered at least the following problems in the prior art: due to the large amount of data in some data tables, the extraction speed is slow, and even with sharding, repeated manual parameter adjustments are required for trial and error. Therefore, there is a technical problem of slow data synchronization speed. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, and computer-readable medium for synchronizing data, which can accelerate the speed of data synchronization.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for synchronizing data is provided, comprising:

[0006] Check the remaining network bandwidth between the database source server and the database target server;

[0007] Based on the linear regression model and the remaining network bandwidth, the initial number of data transmission channels is set. The linear regression model is trained using historical network remaining bandwidth and historical number of data transmission channels.

[0008] Based on the initial quantity, the quantity range of the data transmission channels is determined, and the maximum value of the quantity range is taken as the synchronization quantity of the data transmission channels;

[0009] Using the specified number of data transmission channels, the task of synchronizing data from the database source server to the database target server is initiated, based on the remaining network bandwidth.

[0010] The process of probing the remaining network bandwidth between the database source server and the database target server includes:

[0011] Use data statistics tools to investigate the remaining network bandwidth between the database source server and the database target server.

[0012] The method further includes:

[0013] A collection table is established in the collection library, which includes the remaining network bandwidth for each process and the number of data transmission channels for each process;

[0014] The remaining network bandwidth in each process is taken as the historical remaining network bandwidth, and the number of data transmission channels in each process is taken as the historical number. The linear regression model is trained using the historical remaining network bandwidth and the historical number of data transmission channels.

[0015] The linear regression model is published.

[0016] The data collection table also includes historical write speeds;

[0017] The process of training the linear regression model using the historical remaining bandwidth of the network and the historical number of data transmission channels includes:

[0018] The linear regression model is trained using the remaining bandwidth of the historical network, the historical number of data transmission channels, and the historical write speed.

[0019] The initial number of data transmission channels, based on a linear regression model and the remaining network bandwidth, includes:

[0020] Based on a linear regression model combined with the remaining network bandwidth and current write speed, the initial number of data transmission channels is set.

[0021] The linear regression model is trained using the historical remaining bandwidth of the network and the historical number of data transmission channels, including:

[0022] The linear regression model is trained using a machine learning framework with the historical remaining bandwidth of the network and the historical number of data transmission channels.

[0023] The step of locating the number range of the data transmission channels according to the initial number, and using the maximum value of the number range as the synchronization number of the data transmission channels, includes:

[0024] Based on the initial quantity, the quantity range of the data transmission channel is located by comparing the extreme values ​​of multiple quantity ranges;

[0025] The maximum value in the specified range is used as the synchronization quantity of the data transmission channel, and the maximum value is also used as the synchronization quantity of the data transmission channel for the restart task.

[0026] The step of using the maximum value of the specified quantity range as the synchronization quantity of the data transmission channel includes:

[0027] If the difference between the maximum value of the specified quantity range and the initial quantity is greater than a preset quantity difference threshold, then the maximum value of the adjacent low quantity range is taken as the synchronization quantity of the data transmission channel.

[0028] According to a second aspect of the present invention, an apparatus for synchronizing data is provided, comprising:

[0029] The probe module is used to probe the remaining network bandwidth between the database source server and the database target server;

[0030] The setting module is used to set the initial number of data transmission channels based on a linear regression model combined with the remaining network bandwidth. The linear regression model is trained using historical remaining network bandwidth and historical numbers of data transmission channels.

[0031] The quantity module is used to locate the quantity range of the data transmission channel according to the initial quantity, and to use the maximum value of the quantity range as the synchronization quantity of the data transmission channel;

[0032] The synchronization module is used to initiate the task of synchronizing data from the database source server to the database target server using the number of data transmission channels to achieve the specified synchronization, based on the remaining network bandwidth.

[0033] According to a third aspect of the present invention, an electronic device for synchronizing data is provided, comprising:

[0034] One or more processors;

[0035] Storage device for storing one or more programs.

[0036] When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0037] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0038] One embodiment of the above invention has the following advantages or beneficial effects: It investigates the remaining network bandwidth between the database source server and the database target server; based on a linear regression model combined with the network bandwidth, it sets an initial number of data transmission channels, wherein the linear regression model is trained using historical network bandwidth and the historical number of data transmission channels; according to the initial number, it locates a range of the number of data transmission channels, and uses the maximum value of the range as the synchronization number of the data transmission channels; using the synchronization number of data transmission channels, it initiates the task of synchronizing data from the database source server to the database target server to synchronize the data. Since the synchronization number is determined by the network bandwidth and matches the network bandwidth, fully utilizing the network bandwidth to synchronize data with the synchronization number of data transmission channels can accelerate the data synchronization speed.

[0039] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0040] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0041] Figure 1 This is a schematic diagram of the main flow of the method for synchronizing data according to an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of synchronization data according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the process of training a linear regression model according to an embodiment of the present invention;

[0044] Figure 4 This is a flowchart illustrating the process of determining the number of synchronization channels according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram illustrating the execution of synchronization data according to an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the main structure of a data synchronization device according to an embodiment of the present invention;

[0047] Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0048] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0049] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0050] Currently, when initiating pipeline tasks for data extraction, network bandwidth is sometimes not fully utilized due to insufficient control over the parallelism of network bandwidth. For example, when initiating data transmission tasks, a fixed number of data transmission channels are mainly used for scheduling.

[0051] However, because the available network bandwidth varies for each data source during data extraction, some tasks may not achieve optimal read / write speeds or may have excessively high read / write speeds, wasting system resources. This ultimately leads to longer data synchronization times, impacting user experience. Subsequently, debugging personnel need to adjust the number of data transmission channels. Furthermore, debugging personnel require in-depth knowledge of the data transmission channels to reconfigure them, then test and reuse them, resulting in significant waste of manpower and time.

[0052] In summary, there is a technical problem of slow data synchronization speed.

[0053] To address the technical problem of slow data synchronization speed, the following technical solutions from the embodiments of the present invention can be adopted.

[0054] See Figure 1 , Figure 1 This is a schematic diagram of the main flow of a data synchronization method according to an embodiment of the present invention. The number of data transmission channels to be synchronized is determined based on network bandwidth, and data is synchronized according to the aforementioned number of channels. Figure 1 As shown, the specific steps include:

[0055] S101. Investigate the remaining network bandwidth between the database source server and the database target server.

[0056] A database source server is the database that outputs data, and a database target server is the database that receives and stores data. Data output from the database source server is stored on the database target server via a data transmission channel.

[0057] See Figure 2 , Figure 2 This is a schematic diagram of synchronized data according to an embodiment of the present invention. Figure 2 The source server of the Chinese database synchronizes data to the target server through three data transmission channels. The number three of these data transmission channels is manually configured.

[0058] In this embodiment of the invention, the network bandwidth between multiple data sources varies greatly, and considering the complexity of manually setting up a number of data transmission channels, it cannot solve the technical problem of slow data synchronization.

[0059] To achieve fast data synchronization between the database source server and the database target server, it is necessary to know the remaining network bandwidth between the two servers. Remaining network bandwidth refers to the unused network bandwidth between the two servers. For example, if the network bandwidth between the source and target servers is 100 Mbps, the used bandwidth is 60 Mbps, and the unused bandwidth is 40 Mbps.

[0060] In one embodiment of the present invention, a data statistics tool can be used to investigate the remaining network bandwidth between the database source server and the database target server. As an example, a DataX engine is deployed exclusively on the database source server, dedicating system resources to it. The CommunicationTool object, a data statistics tool provided by the DataX framework, is used to investigate the remaining network bandwidth between the database source server and the database target server.

[0061] S102. Based on the linear regression model and the remaining network bandwidth, set the initial number of data transmission channels. The linear regression model is trained using the historical remaining network bandwidth and the historical number of data transmission channels.

[0062] In an embodiment of the present invention, a linear regression model is used to set the initial number of data transmission channels. It should be noted that the linear regression model is trained using historical network remaining bandwidth and the historical number of data transmission channels.

[0063] See Figure 3 , Figure 3 This is a schematic diagram of the process of training a linear regression model according to an embodiment of the present invention. Figure 3 The determination of the linear regression model is illustrated by example.

[0064] S301. Establish a collection table in the collection library. The collection table includes the remaining network bandwidth in each process and the number of data transmission channels in each process.

[0065] To train a linear regression model, a data collection table is created in the data collection library. As an example, to facilitate the analysis of the collected data, the data collection library includes the Hive library. Hive is a data warehouse analysis system built on Hadoop.

[0066] Create a collection table in the collection library. The collection table includes the remaining network bandwidth for each process and the number of data transmission channels for each process.

[0067] Specifically, a data collection and reporting service, `reportDataService`, can be developed. This service first stores the remaining network bandwidth and the number of data transmission channels for each process to a disk file on the server, and then uploads this data to the Hadoop Distributed File System (HDFS). As an example, to ensure timely storage of remaining network bandwidth and data transmission channel counts, the data can be stored to the server's disk file at preset intervals. For example, the preset interval is 5 seconds. The disk file includes a file named `data.txt`.

[0068] The remaining network bandwidth and the number of data transmission channels after uploading to HDFS are loaded into the Hive collection library using the Hive load command.

[0069] S302. Take the remaining network bandwidth in each process as the historical remaining network bandwidth, and the number of data transmission channels in each process as the historical quantity, and train a linear regression model using the historical remaining network bandwidth and the historical quantity of data transmission channels.

[0070] To improve the accuracy of the output data from the linear regression model, it is necessary to clean the sample data. This sample data includes the data collected from the data collection table, i.e., the data loaded into the Hive collection library.

[0071] Data cleaning includes filtering invalid and missing data. Invalid data includes data with incorrect formatting. Missing data includes data with missing data items.

[0072] The cleaned data can be divided into a training set and a validation set. As an example, 70% of the cleaned data can be used as the training set, and 30% as the validation set.

[0073] A data transmission channel typically has a maximum read / write speed of no more than 10MB / s, consuming approximately M*10MB of memory (M>1). Assuming sufficient memory, a maximum of 8 channels is generally sufficient for most uses. Therefore, fully utilizing the remaining network bandwidth is approximately N*10Mb (M>N≥1), exhibiting a linear relationship between the two. A linear regression model h is then set. θ (x).

[0074] h θ (x)=θ0+xθ1 Formula 1

[0075] Where x is the remaining network bandwidth, and θ0 and θ1 are the model coefficients.

[0076] In an embodiment of the present invention, a linear regression model is trained by minimizing a cost function. The cost function is shown in Formula 2.

[0077]

[0078] In an embodiment of the present invention, by minimizing the cost function of Formula 2, Formula 1 is trained using the historical remaining bandwidth of the network and the historical number of data transmission channels to obtain a linear regression model.

[0079] In one embodiment of the present invention, to improve the speed of training a linear regression model, a machine learning framework can be employed to train the linear regression model using historical network remaining bandwidth and the historical number of data transmission channels. As an example, the machine learning framework includes the Python framework sklearn.

[0080] In one embodiment of the present invention, to improve the accuracy of the linear regression model, historical write speed is added to the acquisition table. That is, the acquisition table also includes historical write speed. In other words, the acquisition table includes the remaining network bandwidth for each process, the number of data transmission channels for each process, and the historical write speed for each process.

[0081] During the training of the linear regression model, the historical remaining network bandwidth, the historical number of data transmission channels, and the historical write speed are used to train the linear regression model.

[0082] Considering that the linear regression model is trained using historical network remaining bandwidth, historical number of data transmission channels, and historical write speed, the initial number of data transmission channels can be set based on the linear regression model combined with the network remaining bandwidth and current write speed.

[0083] It should be noted that, because historical write speeds are taken into account during the training of the linear regression model, the data transmission channel settings for the linear regression model need to consider not only the remaining network bandwidth but also the current write speed, resulting in a more accurate data transmission channel setting.

[0084] S303, Publish the linear regression model.

[0085] Specifically, after multiple rounds of training until the cost function converges, a fitted model is obtained. Finally, the model is validated using a validation set. Multiple rounds of validation improve the model's accuracy. Finally, the Flask framework is used for HTTP service deployment. That is, the Flask framework is used to deploy the linear regression model. The Flask framework is developed in Python, the same language used by machine learning frameworks, which facilitates the deployment of linear regression models.

[0086] exist Figure 3 In one embodiment, a linear regression model is trained using data collected from a table.

[0087] S103. Based on the initial quantity, determine the quantity range of the data transmission channels, and use the maximum value of the quantity range as the synchronization quantity of the data transmission channels.

[0088] In this embodiment of the invention, after the initial number of data transmission channels is determined, the number of synchronization channels needs to be determined by locating the data interval.

[0089] The data range of a data transmission channel is defined by the number of data transmission channels. As an example, the first data range is 0 to 1; the second is 1 to 2; the third is 2 to 4; the fourth is 4 to 8; and the fifth is greater than 8. Setting data ranges helps control the number of synchronization channels. The data ranges of the data transmission channels can be preset based on the actual application scenario.

[0090] See Figure 4 , Figure 4 This is a flowchart illustrating the process of determining the number of synchronization channels for a data transmission channel according to an embodiment of the present invention. Specifically, it includes the following steps:

[0091] S401. Based on the initial quantity, locate the quantity range of the data transmission channel by comparing the extreme values ​​of multiple quantity ranges.

[0092] The quantity range is determined by the boundary values, i.e., the maximum and minimum values, within a range of values. Based on the initial quantity, the quantity range of the data transmission channel is located by comparing the extreme values ​​of multiple quantity ranges.

[0093] As an example, the first data range is 0 to 1; the second data range is 1 to 2; the third data range is 2 to 4; the fourth data range is 4 to 8; and the fifth data range is greater than 8. The initial quantity is 3.2. By comparing the extreme values ​​of these multiple quantity ranges, the third data range is located as the quantity range for the data transmission channel.

[0094] S402. The maximum value in the quantity range is used as the synchronization quantity of the data transmission channel, and the maximum value is also used as the synchronization quantity of the data transmission channel for the restart task.

[0095] Once the range of data transmission channels is determined, to ensure normal data transmission, the maximum value of the range is used as the synchronization value for the data transmission channels. In other words, data is transmitted using the maximum value of the range.

[0096] As an example, the initial quantity is 3.2, and the range for the number of data transmission channels is the third data range. The third data range is from 2 to 4. Therefore, 4 can be used as the synchronization quantity for the data transmission channels.

[0097] In one embodiment of the present invention, to accelerate the data synchronization speed between the database source server and the database target server, for the next data synchronization, the source server and the target server can directly use the number of data transmission channels to synchronize data, without having to determine the number of data transmission channels again based on the remaining network bandwidth.

[0098] In other words, the method of synchronizing data also includes: using a synchronized number of data transmission channels to restart the task of synchronizing data from the database source server to the database target server.

[0099] exist Figure 4 In this embodiment, the number of synchronization channels for data transmission is determined by a quantity range, and the quantity range is preset according to the application scenario.

[0100] In one embodiment of the present invention, the quantity range can also be adjusted based on the initial quantity and the maximum value of the quantity range. That is, if the difference between the maximum value of the quantity range and the initial quantity is greater than a preset quantity difference threshold, it indicates that the difference between the current quantity range and the initial quantity is too large, which will waste network bandwidth. In this case, the maximum value of the adjacent lower quantity range can be used as the synchronization quantity of the data transmission channel. The lower quantity range is the range where the maximum value is smaller than the minimum value in the quantity range.

[0101] As an example, the first data range is 0 to 1; the second data range is 1 to 2; the third data range is 2 to 4; the fourth data range is 4 to 8; and the fifth data range is greater than 8. The initial quantity is 3.2. By comparing the extreme values ​​of these multiple quantity ranges, the third data range is located as the quantity range for the data transmission channel. The preset quantity difference threshold is 0.5. Since the difference between the initial quantity of 3.2 and the third data range (0.5) is greater than the preset quantity difference threshold, the maximum value of 2 in the lower quantity range adjacent to the third data range (i.e., the second quantity range) is taken as the synchronization quantity of the data transmission channel.

[0102] The quantity range can be automatically adjusted using the above method without manual intervention.

[0103] S104. Using a synchronous number of data transmission channels, initiate the task of synchronizing data from the database source server to the database target server, based on the remaining network bandwidth to synchronize data.

[0104] Once the number of data transmission channels to be synchronized is determined, the task of synchronizing data from the database source server to the database target server is initiated using that number of channels. Data synchronization is performed according to the remaining network bandwidth.

[0105] In the above embodiment, the remaining network bandwidth between the database source server and the database target server is investigated; based on a linear regression model combined with the network bandwidth, an initial number of data transmission channels is set, whereby the linear regression model is trained using historical network bandwidth and the historical number of data transmission channels; according to the initial number, a range of the number of data transmission channels is determined, and the maximum value of the range is used as the synchronization number of the data transmission channels; using the synchronization number of data transmission channels, the task of synchronizing data from the database source server to the database target server is initiated to synchronize the data. Since the synchronization number is determined by the network bandwidth and matches the network bandwidth, fully utilizing the network bandwidth to synchronize data with the synchronization number of data transmission channels can accelerate the data synchronization speed.

[0106] See Figure 5 , Figure 5 This is a schematic diagram illustrating the execution of synchronized data according to an embodiment of the present invention. Figure 5 First, the remaining network bandwidth between the database source server and the database target server is explored. Based on the remaining network bandwidth, the number of data transmission channels to be synchronized is determined, and then the data is synchronized using the number of data transmission channels to be synchronized.

[0107] See Figure 6 , Figure 6 This is a schematic diagram of the main structure of a data synchronization device according to an embodiment of the present invention. The data synchronization device can implement a data synchronization method, such as... Figure 6 As shown, the device for synchronizing data specifically includes:

[0108] The exploration module 601 is used to explore the remaining network bandwidth between the database source server and the database target server.

[0109] Setting module 602 is used to set the initial number of data transmission channels based on a linear regression model combined with the remaining network bandwidth. The linear regression model is trained using historical remaining network bandwidth and historical numbers of data transmission channels.

[0110] The quantity module 603 is used to locate the quantity range of the data transmission channel according to the initial quantity, and to use the maximum value of the quantity range as the synchronization quantity of the data transmission channel;

[0111] Synchronization module 604 is used to initiate the task of synchronizing data from the database source server to the database target server using the number of data transmission channels to synchronize the data, based on the remaining network bandwidth.

[0112] In one embodiment of the present invention, the exploration module 601 is specifically used to use data statistics tools to explore the remaining network bandwidth between the database source server and the database target server.

[0113] In one embodiment of the present invention, the setting module 602 is further configured to establish a collection table in the collection library, the collection table including the remaining network bandwidth in each process and the number of data transmission channels in each process;

[0114] The remaining network bandwidth in each process is taken as the historical remaining network bandwidth, and the number of data transmission channels in each process is taken as the historical number. The linear regression model is trained using the historical remaining network bandwidth and the historical number of data transmission channels.

[0115] The linear regression model is published.

[0116] In one embodiment of the present invention, the acquisition table further includes historical write speed;

[0117] The setting module 602 is specifically used to train the linear regression model using the historical network remaining bandwidth, the historical number of data transmission channels, and the historical write speed.

[0118] Furthermore, based on a linear regression model combined with the remaining network bandwidth and the current write speed, the initial number of data transmission channels is set.

[0119] In one embodiment of the present invention, the setting module 602 is specifically used to train the linear regression model using a machine learning framework with the historical remaining bandwidth of the historical network and the historical number of data transmission channels.

[0120] In one embodiment of the present invention, the quantity module 603 is specifically used to locate the quantity range of the data transmission channel by comparing the extreme values ​​of multiple quantity ranges according to the initial quantity;

[0121] The maximum value in the specified range is used as the synchronization quantity of the data transmission channel, and the maximum value is also used as the synchronization quantity of the data transmission channel for the restart task.

[0122] In one embodiment of the present invention, the quantity module 603 is specifically used to take the maximum value of the adjacent low quantity interval as the synchronization quantity of the data transmission channel if the difference between the maximum value of the quantity interval and the initial quantity is greater than a preset quantity difference threshold.

[0123] Figure 7 An exemplary system architecture 700 is shown, in which a method or apparatus for synchronizing data can be applied according to embodiments of the present invention.

[0124] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0125] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0126] Terminal devices 701, 702, and 703 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0127] Server 705 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 701, 702, and 703 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal devices.

[0128] It should be noted that the data synchronization method provided in the embodiments of the present invention is generally executed by server 705, and correspondingly, the data synchronization device is generally set in server 705.

[0129] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0130] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0131] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0132] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0133] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0134] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0136] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor may be described as including a probing module, a setting module, a quantity module, and a synchronization module. The names of these modules do not necessarily limit the module itself; for example, a probing module may also be described as "used to probe the remaining network bandwidth between the database source server and the database target server."

[0137] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0138] Check the remaining network bandwidth between the database source server and the database target server;

[0139] Based on the linear regression model and the remaining network bandwidth, the initial number of data transmission channels is set. The linear regression model is trained using historical network remaining bandwidth and historical number of data transmission channels.

[0140] Based on the initial quantity, the quantity range of the data transmission channels is determined, and the maximum value of the quantity range is taken as the synchronization quantity of the data transmission channels;

[0141] Using the specified number of data transmission channels, the task of synchronizing data from the database source server to the database target server is initiated, based on the remaining network bandwidth.

[0142] According to the technical solution of this embodiment of the invention, the remaining network bandwidth between the database source server and the database target server is investigated; based on a linear regression model combined with the network bandwidth, an initial number of data transmission channels is set, wherein the linear regression model is trained using historical network bandwidth and the historical number of data transmission channels; according to the initial number, the range of the number of data transmission channels is determined, and the maximum value of the range is taken as the synchronization number of the data transmission channels; using the synchronization number of data transmission channels, the task of synchronizing data from the database source server to the database target server is initiated to synchronize the data. Since the synchronization number is determined by the network bandwidth and matches the network bandwidth, fully utilizing the network bandwidth to synchronize data with the synchronization number of data transmission channels can accelerate the data synchronization speed.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention. It should be noted that the acquisition, storage, and application of user personal information involved in the technical solutions of this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. A method of synchronizing data, characterized by, The method comprises: probing the network residual bandwidth between a database source server and a database target server; setting an initial number of data transmission channels based on a linear regression model in combination with the network residual bandwidth, the linear regression model being trained using historical network residual bandwidth and historical number of data transmission channels; locating a number interval of the data transmission channels by comparing extreme values of a plurality of number intervals according to the initial number; taking a maximum value of the number interval as a synchronization number of the data transmission channels, and taking the maximum value as a synchronization number of data transmission channels of a restart task; starting a task of synchronizing data of the database source server to the database target server using the synchronization number of data transmission channels, and synchronizing data based on the network residual bandwidth.

2. The method of synchronizing data of claim 1, wherein, The method further comprises: establishing a collection table in a collection library, the collection table comprising network residual bandwidth in each process and number of data transmission channels in each process; 3. The method of synchronizing data of claim 1, wherein, taking the network residual bandwidth in each process as historical network residual bandwidth, and taking the number of data transmission channels in each process as historical number, and training the linear regression model using the historical network residual bandwidth and historical number of data transmission channels; publishing the linear regression model. The collection table further comprises historical write speed. The training of the linear regression model using the historical network residual bandwidth and historical number of data transmission channels comprises:

4. The method of synchronizing data of claim 3, wherein, training the linear regression model using the historical network residual bandwidth, historical number of data transmission channels and historical write speed; The setting of the initial number of data transmission channels based on the linear regression model in combination with the network residual bandwidth comprises: setting the initial number of data transmission channels based on the linear regression model in combination with the network residual bandwidth and current write speed. The training of the linear regression model using the historical network residual bandwidth and historical number of data transmission channels comprises: training the linear regression model using the historical network residual bandwidth and historical number of data transmission channels by using a machine learning framework.

5. The method of synchronizing data of claim 3, wherein, The taking of the maximum value of the number interval as the synchronization number of the data transmission channels comprises: if a number difference between the maximum value of the number interval and the initial number is greater than a preset number difference threshold, taking a maximum value of a lower number interval adjacent to the number interval as the synchronization number of the data transmission channels.

6. The method of synchronizing data of claim 1, wherein, The method comprises: a probing module configured to probe the network residual bandwidth between a database source server and a database target server; 7. An apparatus for synchronizing data, the apparatus comprising: a setting module configured to set an initial number of data transmission channels based on a linear regression model in combination with the network residual bandwidth, the linear regression model being trained using historical network residual bandwidth and historical number of data transmission channels; ​ ​ The quantity module is configured to locate a quantity interval of the data transmission channel by comparing extreme values of a plurality of quantity intervals according to the initial quantity; take a maximum value of the quantity interval as a synchronization quantity of the data transmission channel, and take the maximum value as a synchronization quantity of a data transmission channel of a restart task; The synchronization module is configured to start a task of synchronizing data of the database source server to the database target server by using the data transmission channel with the synchronization quantity, and synchronize the data based on the network residual bandwidth.

8. An electronic device for synchronizing data, characterized by The system comprises: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1-6.

9. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method as claimed in any one of claims 1-6.

Images