Traffic control method, system, medium and program product for cross-network data synchronization

Abstract

The embodiment of the application provides a traffic control method, system, medium and program product for cross-network data synchronization, and relates to the technical field of traffic control. The method comprises the following steps: reading data by a collection unit and putting the data into a first cache queue; obtaining the data from the queue by a sending unit and sending the data to a second flow control server; in a flow limiting state, controlling the data sending rate of the sending unit based on a flow limiting signal; wherein the second flow control server receives the data by a receiving unit and puts the data into a second cache queue, so that a writing unit obtains the data from the second cache queue and writes the data into a destination database; and sending a flow limiting signal to the first flow control server according to the load rate of the second cache queue. The embodiment of the application sets flow control servers at the source end and the destination end respectively, and realizes data blocking control through the cache queue. The rate of sending data at the source end is controlled according to the load condition of the queue at the destination end, so that the cross-network synchronization efficiency of data can be improved while avoiding the overflow of the network middleware memory.

Description

Technical Field

[0001] The present application relates to the field of flow control technology, and in particular to a flow control method, system, medium and program product for cross-network data synchronization. Background Art

[0002] The secure data exchange system has cross-network data synchronization functions such as file synchronization and database synchronization. Cross-network data synchronization refers to the process of transmitting data from the source server to the destination server through the network gateway.

[0003] Data synchronization efficiency is not only affected by network bandwidth, but the bottleneck that often impacts synchronization performance lies in the business logic of data processing on both the source and destination ends. For example, during database synchronization, due to various uncertainties, the rate at which the source reads and sends data often mismatches the rate at which the destination receives and writes data. Consequently, maximizing synchronization performance between the source and destination ends is impossible. Furthermore, if the source's data read and send rate consistently exceeds the destination's data receive and write rate, data accumulation can lead to memory overflows in the network middleware.

[0004] In summary, there is an urgent need for a flow control solution that can avoid memory overflow in network middleware while improving the efficiency of cross-network data synchronization. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a flow control method, system, medium and program product for cross-network data synchronization, which can improve the efficiency of cross-network data synchronization while avoiding memory overflow of network middleware.

[0006] In a first aspect, an embodiment of the present application provides a flow control method for cross-network data synchronization, which is applied to a first flow control server, including:

[0007] Reading data from a source database through a collection unit and transmitting the read data to a first cache queue;

[0008] Acquire data from the first cache queue through a sending unit, and send the acquired data to a second flow control server through a network middleware;

[0009] In response to the current limiting signal sent by the second flow control server and when switching to the current limiting state, controlling the data sending rate of the sending unit based on the current limiting signal;

[0010] In which, the second flow control server includes a receiving unit and a writing unit, the receiving unit is used to receive the data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains the data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a flow limiting signal to the first flow control server when the load rate is not less than a preset load threshold.

[0011] In an embodiment of the present application, by setting up flow control servers at the source end and the destination end respectively, and setting up cache queues in the two flow control servers respectively to buffer and control the data transmission rate of each end, and controlling the rate at which the source end sends data according to the load of the destination end queue, it is possible to improve the efficiency of cross-network data synchronization while avoiding memory overflow of the network middleware.

[0012] In some possible embodiments, the current limiting signal includes the load ratio and the data writing rate of the writing unit, and the data sending rate of the sending unit is calculated according to the load ratio and the data writing rate.

[0013] In an embodiment of the present application, the rate at which the source end sends data is calculated and controlled based on the queue load rate of the destination end and the rate at which data is written, thereby further improving the efficiency of cross-network data synchronization.

[0014] In some possible embodiments, the current limiting signal includes the load rate and the data writing rate of the writing unit;

[0015] The step of controlling the data sending rate of the sending unit based on the current limiting signal in response to the current limiting signal sent by the second flow control server and switching to the current limiting state includes:

[0016] In response to the current limiting signal sent by the second flow control server and when switching to the current limiting state, calculating a semaphore placement rate based on the current limiting signal, and placing a semaphore into the semaphore container at the semaphore placement rate;

[0017] The data sending rate of the sending unit is controlled based on the semaphore in the semaphore container.

[0018] In an embodiment of the present application, the data sending rate of the sending unit is controlled by utilizing an independently running semaphore control thread, thereby further improving the accuracy and reliability of flow control for data synchronization across the network.

[0019] In some possible embodiments, the cross-network data synchronization flow control method further includes:

[0020] acquire a load rate of the first cache queue in real time, and control a data reading rate of the collection unit from the source-end database based on the load rate of the first cache queue.

[0021] In the embodiments of the present application, the data reading rate of the collection unit is controlled based on the load rate of the source-end cache queue, so that the reliability of flow control for data cross-network synchronization can be further improved.

[0022] In a second aspect, the embodiments of the present application provide a flow control method for cross-network data synchronization, applied to a second flow control server, comprising:

[0023] receiving data sent by a first flow control server through network middleware by a receiving unit, and transmitting the received data to a second cache queue;

[0024] acquiring data from the second cache queue by a writing unit and writing the data into a destination-end database;

[0025] in a case where the load rate of the second cache queue is not less than a preset load threshold, sending a flow limiting signal to the first flow control server, so that the first flow control server controls a data sending rate based on the flow limiting signal;

[0026] wherein the first flow control server comprises a collection unit and a sending unit, the collection unit is configured to read data from a source-end database and transmit the data to a first cache queue, so that the sending unit acquires data from the first cache queue and sends the data to the second flow control server through the network middleware.

[0027] In a third aspect, the embodiments of the present application provide a flow control system for cross-network data synchronization, comprising a first flow control server and a second flow control server, the first flow control server comprises a collection unit and a sending unit, and the second flow control server comprises a receiving unit and a writing unit;

[0028] the first flow control server is configured to read data from a source-end database by the collection unit, and transmit the read data to a first cache queue;

[0029] the first flow control server is further configured to acquire data from the first cache queue by the sending unit, and send the acquired data to the second flow control server through network middleware;

[0030] the first flow control server is further configured to, in a case where the second flow control server sends a flow limiting signal and switches to a flow limiting state, control a data sending rate of the sending unit based on the flow limiting signal;

[0031] The second flow control server is configured to receive, through the receiving unit, the data sent by the first flow control server through the network middleware, and transmit the received data to a second cache queue;

[0032] The second flow control server is further configured to obtain data from the second cache queue through the writing unit and write the data into the destination database;

[0033] The second flow control server is also used to send a flow limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data sending rate of the sending unit based on the flow limiting signal.

[0034] In a fourth aspect, an embodiment of the present application provides a flow control device for cross-network data synchronization, which is applied to a first flow control server, including:

[0035] A data acquisition module, configured to read data from a source database through an acquisition unit and transmit the read data to a first cache queue;

[0036] a data sending module, configured to obtain data from the first cache queue through a sending unit, and send the obtained data to a second flow control server through a network middleware;

[0037] a flow control module, configured to, in response to a flow limiting signal sent by the second flow control server and switching to a flow limiting state, control a data sending rate of the sending unit based on the flow limiting signal;

[0038] In which, the second flow control server includes a receiving unit and a writing unit, the receiving unit is used to receive the data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains the data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a flow limiting signal to the first flow control server when the load rate is not less than a preset load threshold.

[0039] In a fifth aspect, an embodiment of the present application provides a flow control device for cross-network data synchronization, which is applied to a second flow control server, including:

[0040] a data receiving module, configured to receive, through a receiving unit, data sent by the first flow control server through the network middleware, and transmit the received data to the second cache queue;

[0041] A data writing module, configured to obtain data from the second cache queue through a writing unit and write the data into a destination database;

[0042] a current limiting judgment module, configured to, when it is determined that the load rate of the second cache queue is not less than a preset load threshold, send a current limiting signal to the first flow control server, so that the first flow control server controls the data sending rate based on the current limiting signal;

[0043] The first flow control server includes a collection unit and a sending unit. The collection unit is used to read data from the source database and transmit it to the first cache queue, so that the sending unit obtains data from the first cache queue and sends it to the second flow control server through the network middleware.

[0044] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented.

[0045] In a seventh aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, it can implement the method described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A schematic diagram of the architecture of a traffic control system for cross-network data synchronization provided by an embodiment of the present application;

[0048] Figure 2 A flowchart of the sending unit provided in an embodiment of the present application;

[0049] Figure 3 A flowchart of the semaphore setting unit provided in an embodiment of the present application;

[0050] Figure 4 A flowchart of the receiving unit provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a flow control device for cross-network data synchronization provided by an embodiment of the present application;

[0052] Figure 6 A schematic structural diagram of another cross-network data synchronization flow control device provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0055] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0056] Existing flow control technologies often directly control the source's data read rate based on the target's data write rate. Therefore, during cross-network data synchronization, if the data write rate is unstable, controlling the data read rate at a lower rate will not maximize data synchronization performance. However, controlling the data read rate at a higher rate will cause data accumulation and memory overflow in the network middleware, as the source's data read rate will continuously exceed the target's data write rate.

[0057] In response to the problems existing in the above-mentioned prior art, the embodiment of the present application sets up a flow control server at the source end and the destination end respectively, and synchronizes the data writing rate of the destination end to the source end and controls the data reading rate of the source end, thereby effectively avoiding the problem of network middleware data accumulation due to the source end reading and sending data too quickly and the destination end receiving and writing data too slowly; at the same time, by setting up cache queues in the flow control servers at the source end and the destination end respectively, the writing and reading of queue data are controlled by the blocking effect of the cache queue, thereby avoiding the problem of low overall data synchronization efficiency due to the unstable data writing rate at the destination end.

[0058] like Figure 1 As shown, an embodiment of the present application provides a flow control system for cross-network data synchronization, including a first flow control server and a second flow control server, the first flow control server including a collection unit and a sending unit, and the second flow control server including a receiving unit and a writing unit;

[0059] The first flow control server is used to read data from the source database through the acquisition unit and transmit the read data to the first cache queue;

[0060] The first flow control server is further configured to obtain data from the first cache queue through the sending unit, and send the obtained data to the second flow control server through the network middleware;

[0061] The first flow control server is further configured to control the data sending rate of the sending unit based on the current limiting signal when responding to the current limiting signal sent by the second flow control server and switching to the current limiting state;

[0062] The second flow control server is used to receive the data sent by the first flow control server through the network middleware through the receiving unit, and transmit the received data to the second cache queue;

[0063] The second flow control server is further configured to obtain data from the second cache queue through the writing unit and write the data into the destination database;

[0064] The second flow control server is also used to send a flow limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data sending rate of the sending unit based on the flow limiting signal.

[0065] It is understandable that, assuming that the data of the source database needs to be synchronized to the destination database across the network, the flow control system for cross-network data synchronization provided by the embodiment of the present application can be used, such as Figure 1 As shown, the system collects the data that needs to be synchronized from the source database by the first flow control server, and then transmits the data to the second flow control server through the network middleware (which can be a security isolation device) (the network middleware mainly plays the role of network isolation and data ferrying), and then synchronizes the data to the destination database through the second flow control server.

[0066] Among them, the first flow control server collects the data that needs to be synchronized from the source database through its collection unit, and then sends the data to the network middleware through the sending unit. It should be noted that the collection unit and the sending unit exchange data through the first cache queue. The collection unit and the sending unit are two independent threads. During the data synchronization process, the collection unit collects the data that needs to be synchronized from the source database and places it in the first cache queue. The sending unit then takes the data out of the first cache queue and sends it to the network middleware. It can be understood that the first cache queue can achieve a blocking effect. When the first cache queue is empty, the sending unit will be blocked (no data can be taken out for data sending). When the first cache queue is full, the collection unit will be blocked (no data can be put into the first cache queue).

[0067] It should be noted that during the data synchronization process, when the first flow control server reads data at a faster rate and the cross-network transmission rate is slower in a short period of time, it will not immediately affect the data reading rate of the acquisition unit. There is a period of data buffering time, which can ensure that the system as a whole has faster and more stable data synchronization efficiency.

[0068] In some possible embodiments, the first flow control server may also be used to:

[0069] The load rate of the first cache queue is acquired in real time, and the rate at which the acquisition unit reads data from the source database is controlled based on the load rate of the first cache queue.

[0070] It should be noted that in addition to the aforementioned control strategy of controlling the acquisition unit solely through the blocking effect of the first cache queue (i.e., the acquisition unit is blocked when the first cache queue is full), the following control strategy can also be added: controlling the data acquisition rate of the acquisition unit based on the load rate of the first cache queue. Specifically, when the load rate of the first cache queue exceeds a certain threshold, the higher the load rate, the lower the data acquisition rate of the acquisition unit needs to be controlled. The load rate is calculated as follows: load rate = queue used capacity / queue total capacity.

[0071] After the first flow control server sends the data to be synchronized to the second flow control server via the network middleware, the second flow control server receives the data to be synchronized from the network middleware through the receiving unit, and then writes the data to the destination database through the writing unit to complete the cross-network synchronization of the data. It should be noted that the receiving unit and the writing unit interact with each other through the second cache queue. The receiving unit and the writing unit are two independent threads. During the data synchronization process, the receiving unit places the data into the second cache queue after receiving the data sent by the network middleware, and then the writing unit takes out the data from the second cache queue and writes it to the destination database. It can be understood that the second cache queue can achieve a blocking effect. When the second cache queue is empty, the writing unit will be blocked (no data can be taken out and written to the destination database). When the second cache queue is full, the receiving unit will be blocked (no data can be put into the second cache queue).

[0072] It should be noted that during the data synchronization process, the write rate of the write unit of the destination database may become low due to high CPU usage of the operating system or high disk IO load. At this time, the second cache queue will gradually increase and eventually cause the receiving unit to be blocked. In order to avoid data accumulation in the network middleware and cause memory overflow, it is necessary to limit the data flow of the sending unit of the first flow control server.

[0073] Specifically, if it is determined that the load rate of the second cache queue is not less than a preset load threshold, a current limiting signal can be sent to the first flow control server. Upon receiving the current limiting signal, the first flow control server will switch to a current limiting state and perform current limiting control on the data transmission rate of the sending unit according to the current limiting signal. For example, the data transmission rate of the sending unit can be controlled based on the load rate of the second cache queue. It is understandable that the higher the load rate of the second cache queue, the lower the data transmission rate of the sending unit needs to be controlled.

[0074] In some possible embodiments, the throttling signal comprises the load rate and the data write rate of the write unit, and the data sending rate of the sending unit is calculated according to the load rate and the data write rate.

[0075] Specifically, when the data sending rate of the sending unit is controlled according to the throttling signal, the target sending rate of the sending unit can be determined according to the load rate of the second cache queue and the data write rate of the write unit. It can be understood that the load rate of the second cache queue is negatively correlated with the target sending rate, and the data write rate of the write unit is positively correlated with the target sending rate. The specific calculation formula can be set according to requirements.

[0076] In some possible embodiments, the throttling signal comprises the load rate and the data write rate of the write unit.

[0077] The first flow control server is further configured to:

[0078] In the case of switching to the throttling state in response to the throttling signal sent by the second flow control server, the signal quantity putting rate is calculated based on the throttling signal, and the signal quantity in the signal quantity container is put into the signal quantity container according to the signal quantity putting rate.

[0079] The data sending rate of the sending unit is controlled based on the signal quantity in the signal quantity container.

[0080] It should be noted that the data sending rate control of the sending unit can be directly performed by the sending unit itself. Since the throttling signal acquisition, rate calculation and other processes are involved, in order to avoid occupying too many computing resources of the sending unit, the signal quantity control mode can be used to realize the flow rate control of the sending unit.

[0081] Specifically, a signal quantity setting unit comprising a signal quantity container can be configured in the first flow control server, and the signal quantity container is used to put signal quantity into it according to a specific rate based on the throttling signal. The sending unit can be configured to listen to the signal quantity in the signal quantity container, and when the first flow control server is in the throttling state, the sending unit needs to determine its data sending rate according to the listened signal quantity.

[0082] Therefore, by using an independently running signal quantity control thread (signal quantity setting unit) to control the data sending rate of the sending unit, the problem of low performance of the sending unit due to the need to consider rate calculation is avoided, thereby further improving the accuracy and reliability of the flow control of the cross-network data synchronization.

[0083] The modules of the flow control system of the cross-network data synchronization are described in detail as follows:

[0084] (1) The acquisition unit can be used as an independent thread. When a signal requiring data synchronization is detected or the data synchronization policy is enabled, the thread is started. Thereafter, the acquisition unit continuously reads data from the source database and places it into the first cache queue. When the first cache queue is full, further attempts to add data will be blocked, thereby controlling the rate at which the acquisition unit acquires data. The size of the first cache queue and the blocking time can be configured according to the actual business scenario. The thread is closed when data synchronization is completed or the data synchronization policy is disabled.

[0085] (2) The sending unit and the collection unit are independent of each other. They exchange data through the first cache queue. When the collection unit collects data, the thread corresponding to the sending unit is started. Figure 2 The sending unit adopts different data sending modes according to whether the first flow control server is in the flow limiting state:

[0086] If it is not in the current limiting state, the data is directly taken out from the first cache queue and sent at the normal sending rate; wherein, the current limiting state can be switched according to the current limiting signal sent by the second flow control server, for example, it switches to the current limiting state in response to the limit_flag=true signal; it switches to the non-current limiting state in response to the limit_flag=false signal.

[0087] If it is in a current limiting state, it is necessary to control the data sending rate according to the current limiting signal. For example, a signal is obtained from the semaphore container, and data is sent only when the signal is obtained. Therefore, the data sending rate of the sending unit can be controlled by the rate at which the semaphore is put into the semaphore container. At this time, since the data sending rate of the sending unit is limited, the data acquisition rate of the acquisition unit will temporarily put data into the first cache queue at a normal rate, so the first cache queue will soon be filled with data. At this time, the acquisition unit will be blocked when writing data to the first cache queue, thereby achieving delayed control of the data acquisition rate of the acquisition unit, avoiding the direct impact of the overall data transmission efficiency of the system due to short-term fluctuations in the destination data write rate. Only when the write rate is small and lasts for a long time will the data acquisition be current limited.

[0088] (3) The semaphore setting unit can use an independent thread. After monitoring the current limiting signal sent by the second flow control server, the thread starts running. The working process of the semaphore setting unit is as follows: Figure 3 As shown. A semaphore container is defined in the shared memory. Based on the load rate of the second cache queue contained in the current limiting signal and the write rate of the write unit, the rate of placing signals into the semaphore container can be calculated, and the signal is placed into the semaphore container based on the rate. For example, the rate of placing signals can be calculated based on the following formula:

[0089] sem_rate = (1 - load_factor) * (write_rate / block_size)

[0090] Wherein sem_rate is the rate of putting into signal, load_factor is the load rate of the second cache queue, write_rate is the data write rate of the write unit, and block_size is the total data capacity of the first cache queue.

[0091] (4) The receiving unit is configured to listen to and receive data sent by the sending unit through the network middleware, and put the data into the second cache queue. In addition, the determination and setting process of the flow control signal can be performed by the receiving unit. Specifically, the receiving unit can first determine the load rate of the second cache queue each time data is received. If the load rate is greater than a set threshold (for example, 0.8), it means that the rate at which the write unit takes data from the second cache queue and writes it to the destination database is slower, and the rate at which the receiving unit receives data and puts it into the second cache queue is faster. At this time, the first flow control server needs to be throttled. The current load rate of the second cache queue, the data write rate of the write unit, and the flow control flag are sent to the first flow control server to enable the first flow control server to implement data throttling and sending.

[0092] When it is determined that the load rate of the second cache queue decreases to below a preset threshold (which can be set to be lower than the threshold for starting throttling, for example, 0.5), the flag for canceling throttling is sent to the first flow control server to stop the first flow control server from throttling and sending data, and to restore normal sending. Exemplarily, the working process of the receiving unit can be as shown in Figure 4 .

[0093] (5) The write unit and the receiving unit can run independently in threads. The write unit and the receiving unit interact with each other through the second cache queue. The write unit is configured to take data from the second cache queue and write it to the destination database, until all data of the current synchronization task is written to the destination database, thereby completing cross-network data synchronization.

[0094] Exemplarily, the following specific examples are given to explain the flow control process of the cross-network data synchronization of the present application in detail:

[0095] 1. The user configures the file synchronization strategy in the first flow control server and the second flow control server, and enables the file synchronization strategy. At this time, the first flow control server acquisition unit starts to listen to the source database.

[0096] 2. The user adds a 10G file to the source database and wants to synchronize the file to the destination database.

[0097] 3. The acquisition unit monitors the 10G file, reads the file stream of the source database, reads the file block size block_size = 4096 bytes each time (the data reading rate is set according to the actual situation), and writes the 4096 bytes of data read each time into the first cache queue.

[0098] 4. Start the thread of the sending unit. The sending unit reads the first cache queue, establishes a connection with the second flow control server through the network middleware, and then takes out the data from the first cache queue. At this time, the flow limiting flag limit_rate = false, that is, the system is in a non-flow limiting state by default. Therefore, the taken-out data is directly sent to the second flow control server through the network middleware.

[0099] 5. After receiving the data, the receiving unit of the second flow control server writes the data into the second cache queue. At the beginning, the second cache queue is empty and can be written directly.

[0100] 6. The write unit of the second flow control server takes out data from the second cache queue and writes it to the destination database. When the write efficiency of the write unit of the destination database deteriorates due to high CPU usage of the operating system or high disk IO, the rate at which data is read from the second cache queue will be slow. As the receiving unit continuously writes data to the second cache queue, the load rate load_factor of the second cache queue (load_factor = queue used capacity / total capacity) will gradually increase from 0 to 0.8 (the set threshold). At this time, the flow limiting flag limit_flag = true, the data writing rate write_rate (write_rate = the amount of data read from the queue per second, for example, write_rate = 1MB / s), and the current load rate of the second cache queue (for example, load_factor = 0.8) need to be sent to the first flow control server to form a flow limiting signal.

[0101] 7. The semaphore setting unit of the first flow control server obtains the current limiting signal (current limiting flag limit_flag = true, write rate write_rate = 1MB / s, load_factor = 0.8). Based on this information, the rate of placing the semaphore can be calculated:

[0102] sem_rate = (1-load_factor) * (write_rate / block_size) = 51.2 signals / second, rounded down, that is, 51 signals are placed in the semaphore container per second.

[0103] 8. After the semaphore setting unit obtains limit_flag=true from the second flow control server, it synchronizes the value of limit_flag=true to the sending unit; when the sending unit determines that limit_rate=true, it must determine whether there is a signal in the semaphore container semaphore before taking out the data from the first cache queue and sending it. If there is no signal, it will block until there is a signal and then send data. Since the rate of placing signals in the semaphore container is 51 / s, the sending unit will only read the queue data 51 times per second at this time. Assuming that the data sending rate of the sending unit is 4096 bytes / time, data is sent to the second flow control server at a rate of 204KB / s.

[0104] 9. As the rate at which data is taken out of the first cache queue slows down, the collection unit of the first flow control server will gradually accumulate data until the queue is full. At this time, the rate at which the collection unit reads data can be gradually controlled until the sending is blocked (blocking occurs when the queue is full).

[0105] 10. When the write rate of the destination database (the rate at which the write unit takes out data from the second cache queue for writing) is restored, the load rate load_factor of the second cache queue in the second flow control server will gradually decrease. When the load_factor decreases to 0.5 and below 0.5 (the set threshold), it is necessary to stop the flow limiting. At this time, the flag limit_flag = false to cancel the flow limiting is sent to the first flow control server.

[0106] 11. When the first flow control server receives limit_flag=false, it switches to the non-current-limited state. After the sending unit determines that limit_flag=false, it no longer needs to determine whether the semaphore container has a signal, and returns to directly taking data from the first cache queue and sending it to the second flow control server.

[0107] 12. Repeat this process until the 10G file synchronization is completed.

[0108] The present application also provides a flow control method for cross-network data synchronization, which is applied to a first flow control server and may include the following steps:

[0109] S101: Read data from a source database through a collection unit, and transfer the read data to a first cache queue;

[0110] S102: Obtain data from the first cache queue through a sending unit, and send the obtained data to the second flow control server through the network middleware;

[0111] S103, in response to the flow control server sent throttling signal and switch to the case of throttling state, based on the throttling signal control data sending rate of sending unit;

[0112] The second flow control server comprises a receiving unit and a writing unit. The receiving unit is configured to receive the data sent by the sending unit and transmit the data to the second cache queue, so that the writing unit obtains the data from the second cache queue and writes the data into the destination database. The second flow control server is further configured to obtain the load rate of the second cache queue in real time, and send a throttling signal to the first flow control server when the load rate is not less than a preset load threshold.

[0113] It should be noted that the method provided in the embodiments of the present application is a one-sided writing method applied to the working process of the first flow control server. The specific steps of the method can be referred to the description in the foregoing description. To avoid repetition, the detailed description is appropriately omitted here.

[0114] The embodiments of the present application further provide a flow control method for cross-network data synchronization, which is applied to the second flow control server and can comprise the following steps:

[0115] S201, receiving the data sent by the first flow control server through the network middleware by the receiving unit, and transmitting the received data to the second cache queue;

[0116] S202, obtaining the data from the second cache queue by the writing unit and writing the data into the destination database;

[0117] S203, in the case where the load rate of the second cache queue is not less than a preset load threshold, sending a throttling signal to the first flow control server, so that the first flow control server controls the data sending rate based on the throttling signal;

[0118] The first flow control server comprises a collection unit and a sending unit. The collection unit is configured to read the data from the source database and transmit the data to the first cache queue, so that the sending unit obtains the data from the first cache queue and sends the data to the second flow control server through the network middleware.

[0119] It should be noted that the method provided in the embodiments of the present application is a one-sided writing method applied to the working process of the second flow control server. The specific steps of the method can be referred to the description in the foregoing description. To avoid repetition, the detailed description is appropriately omitted here.

[0120] Please refer to Figure 5 , Figure 5A block diagram of a flow control device for cross-network data synchronization provided by some embodiments of the present application is shown. It should be understood that the flow control device for cross-network data synchronization corresponds to the above-mentioned method embodiment and can perform each step involved in the above-mentioned method embodiment. The specific functions of the flow control device for cross-network data synchronization can be found in the description above. To avoid repetition, a detailed description is appropriately omitted here.

[0121] Figure 5 The cross-network data synchronization flow control device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the cross-network data synchronization flow control device, and the cross-network data synchronization flow control device is applied to the first flow control server, including:

[0122] The data collection module 510 is configured to read data from a source database through a collection unit and transmit the read data to a first cache queue;

[0123] The data sending module 520 is configured to obtain data from the first cache queue through a sending unit, and send the obtained data to the second flow control server through the network middleware;

[0124] a flow control module 530 for controlling the data transmission rate of the sending unit based on the current limiting signal when responding to the current limiting signal sent by the second flow control server and switching to the current limiting state;

[0125] Among them, the second flow control server includes a receiving unit and a writing unit. The receiving unit is used to receive the data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains the data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a flow limiting signal to the first flow control server when the load rate is not less than the preset load threshold.

[0126] Please refer to Figure 6 , Figure 6 A block diagram of a flow control device for cross-network data synchronization provided by some embodiments of the present application is shown. It should be understood that the flow control device for cross-network data synchronization corresponds to the above-mentioned method embodiment and can perform each step involved in the above-mentioned method embodiment. The specific functions of the flow control device for cross-network data synchronization can be found in the description above. To avoid repetition, a detailed description is appropriately omitted here.

[0127] Figure 6 The cross-network data synchronization flow control device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the cross-network data synchronization flow control device, and the cross-network data synchronization flow control device is applied to the second flow control server, including:

[0128] The data receiving module 610 is configured to receive data sent by the first flow control server through the network middleware through a receiving unit, and transmit the received data to the second cache queue;

[0129] The data writing module 620 is configured to obtain data from the second cache queue through a writing unit and write the data into the destination database;

[0130] The current limiting judgment module 630 is configured to send a current limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data transmission rate based on the current limiting signal;

[0131] Among them, the first flow control server includes a collection unit and a sending unit. The collection unit is used to read data from the source database and transmit it to the first cache queue, so that the sending unit obtains data from the first cache queue and sends it to the second flow control server through the network middleware.

[0132] It can be understood that the above-mentioned device embodiment corresponds to the method embodiment of the present invention. The cross-network data synchronization flow control device provided by the embodiment of the present invention can implement the cross-network data synchronization flow control method provided by any method embodiment of the present invention.

[0133] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0134] like Figure 7 As shown, some embodiments of the present application provide an electronic device 700, which includes: a memory 710, a processor 720, and a computer program stored in the memory 710 and executable on the processor 720, wherein the processor 720 reads the program from the memory 710 through the bus 730 and executes the program to implement any of the embodiments of the flow control method for cross-network data synchronization as described above.

[0135] Processor 720 can process digital signals and can include various computing architectures, such as a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.

[0136] The memory 710 can be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all functions of one or more modules described in the embodiments of this application. The processor 720 of the embodiment of the present disclosure can be used to execute the instructions in the memory 710 to implement the method shown above. The memory 710 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory known to those skilled in the art.

[0137] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the method embodiment is executed.

[0138] Some embodiments of the present application further provide a computer program product, which, when running on a computer, enables the computer to execute the method described in the method embodiment.

[0139] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a portion of code, and the module, program segment, or a portion of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0141] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0142] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0143] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0144] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0145] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A flow control method for cross-network data synchronization, characterized in that: Applied to the first flow control server, including: Reading data from a source database through a collection unit and transmitting the read data to a first cache queue; Acquire data from the first cache queue through a sending unit, and send the acquired data to a second flow control server through a network middleware; In response to the current limiting signal sent by the second flow control server and when switching to the current limiting state, controlling the data sending rate of the sending unit based on the current limiting signal; In which, the second flow control server includes a receiving unit and a writing unit, the receiving unit is used to receive the data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains the data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a flow limiting signal to the first flow control server when the load rate is not less than a preset load threshold.

2. The flow control method for cross-network data synchronization according to claim 1, characterized in that: The current limiting signal includes the load ratio and the data writing rate of the writing unit, and the data sending rate of the sending unit is calculated according to the load ratio and the data writing rate.

3. The flow control method for cross-network data synchronization according to claim 1, characterized in that: The current limiting signal includes the load rate and the data writing rate of the writing unit; The step of controlling the data sending rate of the sending unit based on the current limiting signal in response to the current limiting signal sent by the second flow control server and switching to the current limiting state includes: In response to the current limiting signal sent by the second flow control server and when switching to the current limiting state, calculating a semaphore placement rate based on the current limiting signal, and placing a semaphore into the semaphore container at the semaphore placement rate; The data sending rate of the sending unit is controlled based on the semaphore in the semaphore container.

4. The flow control method for cross-network data synchronization according to claim 1, characterized in that: Also includes: The load rate of the first cache queue is acquired in real time, and the rate at which the acquisition unit reads data from the source database is controlled based on the load rate of the first cache queue.

5. A flow control method for cross-network data synchronization, characterized in that: Applied to the second flow control server, including: receiving, by the receiving unit, data sent by the first flow control server through the network middleware, and transmitting the received data to the second cache queue; Obtaining data from the second cache queue and writing it into the destination database through the writing unit; When it is determined that the load rate of the second cache queue is not less than a preset load threshold, sending a current limiting signal to the first flow control server, so that the first flow control server controls the data sending rate based on the current limiting signal; The first flow control server includes a collection unit and a sending unit. The collection unit is used to read data from the source database and transmit it to the first cache queue, so that the sending unit obtains data from the first cache queue and sends it to the second flow control server through the network middleware.

6. The flow control method for cross-network data synchronization according to claim 5, characterized in that: The current limiting signal includes the load ratio and the data writing rate of the writing unit, and the data sending rate is calculated according to the load ratio and the data writing rate.

7. The flow control method for cross-network data synchronization according to claim 5, characterized in that: The current limiting signal includes the load rate and the data writing rate of the writing unit; The first flow control server controls the data sending rate based on the current limiting signal, including: The first flow control server calculates a semaphore placement rate based on the current limiting signal, and places a semaphore into a semaphore container according to the semaphore placement rate; The first flow control server controls the data sending rate of the sending unit based on the semaphore in the semaphore container.

8. A traffic control system for cross-network data synchronization, characterized in that: It includes a first flow control server and a second flow control server, wherein the first flow control server includes a collection unit and a sending unit, and the second flow control server includes a receiving unit and a writing unit; The first flow control server is used to read data from the source database through the acquisition unit and transmit the read data to the first cache queue; The first flow control server is further configured to obtain data from the first cache queue through the sending unit, and send the obtained data to the second flow control server through the network middleware; The first flow control server is further configured to, in response to the flow limiting signal sent by the second flow control server and switching to the flow limiting state, control the data sending rate of the sending unit based on the flow limiting signal; The second flow control server is configured to receive, through the receiving unit, the data sent by the first flow control server through the network middleware, and transmit the received data to a second cache queue; The second flow control server is further configured to obtain data from the second cache queue through the writing unit and write the data into the destination database; The second flow control server is also used to send a flow limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data sending rate of the sending unit based on the flow limiting signal.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the flow control method for cross-network data synchronization according to any one of claims 1 to 7 is executed.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the flow control method for cross-network data synchronization according to any one of claims 1 to 7 is implemented.

Images