Cross-network data synchronization flow control method and system, medium and program product

Abstract

The embodiment of the invention provides a cross-network data synchronization flow control method and system, a medium and a program product, and relates to the technical field of flow control. The method comprises the following steps: reading data through an acquisition unit and putting the data into a first cache queue; acquiring the data from the queue through a sending unit and sending the data to a second flow control server; in the current limiting state, the data sending rate of the sending unit is controlled based on the current limiting signal; wherein the second flow control server receives the data through the receiving unit and puts the data into a second cache queue, so that the writing unit obtains the data from the data and writes the data into a destination end database; and sending a flow limiting signal to the first flow control server according to the load rate of the second cache queue. According to the embodiment of the invention, the flow control servers are respectively arranged at the source end and the destination end, the data blocking control is realized through the cache queue, and the data sending rate of the source end is controlled according to the load condition of the destination end queue, so that the data cross-network synchronization efficiency can be improved while the memory overflow of the network middleware is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of flow control, and in particular to a flow control method, system, medium and program product for cross-network data synchronization. Background Art

[0002] The secure data exchange system has cross-network data synchronization functions such as file synchronization and database synchronization. Cross-network data synchronization refers to the process of transmitting data from the source server to the destination server through the network gate.

[0003] The efficiency of data synchronization is not only affected by the size of the network bandwidth, but the bottleneck that affects synchronization performance is often the business logic of data processing at the source and destination. For example, during database synchronization, due to various uncertain factors, the rate at which the source reads and sends data is usually inconsistent with the rate at which the destination receives and writes data, so the synchronization performance of the source and destination cannot be maximized. In addition, if the rate at which the source reads and sends data is continuously greater than the rate at which the destination receives and writes data, it will also cause problems such as memory overflow in the network middleware due to data accumulation.

[0004] In summary, there is an urgent need for a flow control solution that can improve the efficiency of cross-network data synchronization while avoiding memory overflow in the network middleware. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a flow control method, system, medium and program product for cross-network data synchronization, which can improve the efficiency of cross-network data synchronization while avoiding memory overflow of network middleware.

[0006] In a first aspect, an embodiment of the present application provides a flow control method for cross-network data synchronization, which is applied to a first flow control server, including:

[0007] Reading data from a source database through a collection unit, and transmitting the read data to a first cache queue;

[0008] Acquire data from the first cache queue through a sending unit, and send the acquired data to a second flow control server through a network middleware;

[0009] In response to the current limiting signal sent by the second flow control server and switching to the current limiting state, controlling the data sending rate of the sending unit based on the current limiting signal;

[0010] Among them, the second flow control server includes a receiving unit and a writing unit, the receiving unit is used to receive the data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a flow limiting signal to the first flow control server when the load rate is not less than a preset load threshold.

[0011] In an embodiment of the present application, flow control servers are respectively set at the source end and the destination end, and cache queues are respectively set in the two flow control servers to perform buffer control on the data transmission rate of each end, and the rate at which the source end sends data is controlled according to the load of the destination end queue, thereby improving the efficiency of cross-network data synchronization while avoiding memory overflow of the network middleware.

[0012] In some possible embodiments, the current limiting signal includes the load rate and the data writing rate of the writing unit, and the data sending rate of the sending unit is calculated according to the load rate and the data writing rate.

[0013] In an embodiment of the present application, the rate at which the source end sends data is jointly calculated and controlled based on the queue load rate of the destination end and the rate at which data is written, thereby further improving the efficiency of cross-network data synchronization.

[0014] In some possible embodiments, the current limiting signal includes the load rate and the data writing rate of the writing unit;

[0015] In response to the current limiting signal sent by the second flow control server and switching to the current limiting state, controlling the data sending rate of the sending unit based on the current limiting signal includes:

[0016] In response to the current limiting signal sent by the second flow control server and when switching to the current limiting state, calculating a semaphore placement rate based on the current limiting signal, and placing a semaphore into a semaphore container at the semaphore placement rate;

[0017] The data transmission rate of the transmission unit is controlled based on the semaphore in the semaphore container.

[0018] In an embodiment of the present application, the data sending rate of the sending unit is controlled by utilizing an independently running semaphore control thread, thereby further improving the accuracy and reliability of flow control for data cross-network synchronization.

[0019] In some possible embodiments, the flow control method for cross-network data synchronization further includes:

[0020] The load rate of the first cache queue is acquired in real time, and the rate at which the acquisition unit reads data from the source database is controlled based on the load rate of the first cache queue.

[0021] In an embodiment of the present application, the read data rate of the acquisition unit is controlled based on the load rate of the source cache queue, so as to further improve the reliability of flow control for data cross-network synchronization.

[0022] In a second aspect, an embodiment of the present application provides a flow control method for cross-network data synchronization, which is applied to a second flow control server, including:

[0023] Receiving, by a receiving unit, data sent by the first flow control server through the network middleware, and transmitting the received data to a second cache queue;

[0024] Obtaining data from the second cache queue through a writing unit and writing the data into a destination database;

[0025] When it is determined that the load rate of the second cache queue is not less than a preset load threshold, sending a current limiting signal to the first flow control server, so that the first flow control server controls the data sending rate based on the current limiting signal;

[0026] The first flow control server includes a collection unit and a sending unit, wherein the collection unit is used to read data from a source database and transmit the data to a first cache queue, so that the sending unit obtains data from the first cache queue and sends the data to the second flow control server through the network middleware.

[0027] In a third aspect, an embodiment of the present application provides a flow control system for cross-network data synchronization, including a first flow control server and a second flow control server, wherein the first flow control server includes a collection unit and a sending unit, and the second flow control server includes a receiving unit and a writing unit;

[0028] The first flow control server is used to read data from the source database through the acquisition unit, and transmit the read data to the first cache queue;

[0029] The first flow control server is further used for acquiring data from the first cache queue through the sending unit, and sending the acquired data to the second flow control server through the network middleware;

[0030] The first flow control server is further configured to control the data transmission rate of the transmission unit based on the current limiting signal in response to the current limiting signal sent by the second flow control server and switch to the current limiting state;

[0031] The second flow control server is used for receiving, through the receiving unit, the data sent by the first flow control server through the network middleware, and transmitting the received data to the second cache queue;

[0032] The second flow control server is further used to obtain data from the second cache queue through the writing unit and write the data into the destination database;

[0033] The second flow control server is also used to send a flow limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data sending rate of the sending unit based on the flow limiting signal.

[0034] In a fourth aspect, an embodiment of the present application provides a flow control device for cross-network data synchronization, which is applied to a first flow control server, including:

[0035] A data acquisition module, used for reading data from a source database through an acquisition unit, and transmitting the read data to a first cache queue;

[0036] A data sending module, configured to obtain data from the first cache queue through a sending unit, and send the obtained data to a second flow control server through a network middleware;

[0037] a flow control module, configured to control the data transmission rate of the transmission unit based on the current limiting signal in response to the current limiting signal sent by the second flow control server and switch to the current limiting state;

[0038] Among them, the second flow control server includes a receiving unit and a writing unit, the receiving unit is used to receive the data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a flow limiting signal to the first flow control server when the load rate is not less than a preset load threshold.

[0039] In a fifth aspect, an embodiment of the present application provides a flow control device for cross-network data synchronization, which is applied to a second flow control server, including:

[0040] A data receiving module, configured to receive data sent by the first flow control server through the network middleware through a receiving unit, and transmit the received data to a second cache queue;

[0041] A data writing module, used for acquiring data from the second cache queue through a writing unit and writing the data into a destination database;

[0042] a current limiting judgment module, configured to send a current limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data sending rate based on the current limiting signal;

[0043] The first flow control server includes a collection unit and a sending unit, wherein the collection unit is used to read data from a source database and transmit the data to a first cache queue, so that the sending unit obtains data from the first cache queue and sends the data to the second flow control server through the network middleware.

[0044] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented.

[0045] In a seventh aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of the architecture of a traffic control system for cross-network data synchronization provided by an embodiment of the present application;

[0048] Figure 2 A flowchart of the sending unit provided in the embodiment of the present application;

[0049] Figure 3 A flowchart of a semaphore setting unit provided in an embodiment of the present application;

[0050] Figure 4 A flowchart of a receiving unit provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a flow control device for cross-network data synchronization provided by an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of another flow control device for cross-network data synchronization provided by an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0055] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0056] In the existing flow control technology, the data reading rate of the source end is often directly controlled according to the data writing rate of the target end. Therefore, during the cross-network data synchronization process, when the data writing rate is unstable, if the data reading is controlled at a lower rate, the maximum data synchronization performance cannot be exerted. If the data reading is controlled at a higher rate, then the source end reads data at a rate that is continuously greater than the target end writes data, which will cause data accumulation and memory overflow in the network middleware.

[0057] In view of the problems existing in the above-mentioned prior art, the embodiments of the present application respectively set up a flow control server at the source end and the destination end, and synchronize the data writing rate of the destination end to the source end and control the data reading rate of the source end, thereby effectively avoiding the problem of data accumulation in the network middleware due to the source end reading and sending data too quickly and the destination end receiving and writing data too slowly; at the same time, by setting up cache queues in the flow control servers at the source end and the destination end respectively, the writing and reading of queue data are controlled by the blocking effect of the cache queue, thereby avoiding the problem of low overall data synchronization efficiency due to the unstable data writing rate at the destination end.

[0058] like Figure 1 As shown, an embodiment of the present application provides a flow control system for cross-network data synchronization, including a first flow control server and a second flow control server, the first flow control server includes a collection unit and a sending unit, and the second flow control server includes a receiving unit and a writing unit;

[0059] The first flow control server is used to read data from the source database through the acquisition unit, and transmit the read data to the first cache queue;

[0060] The first flow control server is also used to obtain data from the first cache queue through the sending unit, and send the obtained data to the second flow control server through the network middleware;

[0061] The first flow control server is further configured to control the data transmission rate of the transmission unit based on the current limiting signal when responding to the current limiting signal sent by the second flow control server and switching to the current limiting state;

[0062] The second flow control server is used for receiving data sent by the first flow control server through the network middleware through the receiving unit, and transmitting the received data to the second cache queue;

[0063] The second flow control server is further used to obtain data from the second cache queue through the writing unit and write the data into the destination database;

[0064] The second flow control server is also used to send a flow limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data sending rate of the sending unit based on the flow limiting signal.

[0065] It is understandable that, assuming that the data of the source database needs to be synchronized to the destination database across the network, the flow control system for cross-network data synchronization provided by the embodiment of the present application can be adopted, such as Figure 1 As shown, the system collects the data that needs to be synchronized from the source database by the first flow control server, and then transmits the data to the second flow control server through the network middleware (which can be a security isolation device) (the network middleware mainly plays the role of network isolation and data ferrying), and then synchronizes the data to the destination database through the second flow control server.

[0066] Among them, the first flow control server collects the data that needs to be synchronized from the source database through its collection unit, and then sends the data to the network middleware through the sending unit. It should be noted that the collection unit and the sending unit exchange data through the first cache queue. The collection unit and the sending unit are two independent threads. During the data synchronization process, the collection unit collects the data that needs to be synchronized from the source database and places it in the first cache queue, and then the sending unit takes out the data from the first cache queue and sends it to the network middleware. It can be understood that the first cache queue can achieve a blocking effect. When the first cache queue is empty, the sending unit will be blocked (no data can be taken out for data sending). When the first cache queue is full, the collection unit will be blocked (unable to put data into the first cache queue).

[0067] It should be noted that during the data synchronization process, when the first flow control server reads data at a faster rate and the cross-network transmission rate is slower in a short period of time, it will not immediately affect the data reading rate of the collection unit. There is a period of data buffering time, thereby ensuring that the system as a whole has faster and more stable data synchronization efficiency.

[0068] In some possible embodiments, the first flow control server may also be used for:

[0069] The load rate of the first cache queue is acquired in real time, and the rate at which the acquisition unit reads data from the source database is controlled based on the load rate of the first cache queue.

[0070] It should be noted that, in addition to the above-mentioned control strategy of controlling the acquisition unit only through the blocking effect of the first cache queue (i.e., the acquisition unit is blocked when the first cache queue is full), the following control strategy can also be added: the data acquisition rate of the acquisition unit is controlled according to the load rate of the first cache queue. Specifically, when the load rate of the first cache queue is higher than a certain threshold, the higher the load rate, the lower the data acquisition rate of the acquisition unit needs to be controlled. The load rate is calculated as follows: load rate = queue used capacity / queue total capacity.

[0071] After the first flow control server sends the data to be synchronized to the second flow control server via the network middleware, the second flow control server receives the data to be synchronized from the network middleware through the receiving unit, and then writes the data to the destination database through the writing unit to complete the cross-network synchronization of the data. It should be noted that the receiving unit and the writing unit interact with each other through the second cache queue. The receiving unit and the writing unit are two independent threads. During the data synchronization process, the receiving unit places the data in the second cache queue after receiving the data sent by the network middleware, and then the writing unit takes out the data from the second cache queue and writes it to the destination database. It can be understood that the second cache queue can achieve a blocking effect. When the second cache queue is empty, the writing unit will be blocked (no data can be taken out and written to the destination database). When the second cache queue is full, the receiving unit will be blocked (no data can be put into the second cache queue).

[0072] It should be noted that during the data synchronization process, the write rate of the write unit of the destination database may become low due to high CPU usage of the operating system or high disk IO load. At this time, the second cache queue will gradually increase and eventually cause the receiving unit to be blocked. In order to avoid data accumulation in the network middleware and cause memory overflow, it is necessary to limit the data flow of the sending unit of the first flow control server.

[0073] Specifically, when it is determined that the load rate of the second cache queue is not less than the preset load threshold, a current limiting signal can be sent to the first flow control server. When the first flow control server receives the current limiting signal, it will switch to the current limiting state and perform current limiting control on the data sending rate of the sending unit according to the current limiting signal. For example, the data sending rate of the sending unit can be controlled according to the load rate of the second cache queue. It is understandable that the higher the load rate of the second cache queue, the lower the data sending rate of the sending unit needs to be controlled.

[0074] In some possible embodiments, the current limiting signal includes a load rate and a data writing rate of the writing unit, and the data sending rate of the sending unit is calculated according to the load rate and the data writing rate.

[0075] Specifically, when the data transmission rate of the transmission unit is controlled according to the current limiting signal, the target transmission rate of the transmission unit can be determined according to the load rate of the second cache queue and the data writing rate of the writing unit. It can be understood that the load rate of the second cache queue is negatively correlated with the target transmission rate, and the data writing rate of the writing unit is positively correlated with the target transmission rate. The specific calculation formula can be set according to the requirements.

[0076] In some possible embodiments, the current limiting signal includes a load rate and a data writing rate of the writing unit;

[0077] The first flow control server is also used for:

[0078] In response to the current limiting signal sent by the second flow control server and when switching to the current limiting state, a semaphore placement rate is calculated based on the current limiting signal, and a semaphore is placed into the semaphore container according to the semaphore placement rate;

[0079] The data transmission rate of the transmission unit is controlled based on the semaphore in the semaphore container.

[0080] It should be noted that the data sending rate control of the sending unit can be directly executed by the sending unit itself. Since it involves processes such as current limiting signal acquisition and rate calculation, in order to avoid occupying too many computing resources of the sending unit, semaphore control can be used to achieve traffic rate control of the sending unit.

[0081] Specifically, a semaphore setting unit can be configured in the first flow control server, which includes a semaphore container for placing semaphores into it at a specific rate according to a current limiting signal. The sending unit can be configured to monitor the semaphore in the semaphore container. When the first flow control server is in a current limiting state, the sending unit needs to determine its own data sending rate based on the monitored semaphore.

[0082] Based on this, by utilizing an independently running semaphore control thread (semaphore setting unit) to control the data sending rate of the sending unit, the low performance problem of the sending unit due to the need to take into account the rate calculation can be avoided, thereby further improving the accuracy and reliability of data flow control for cross-network synchronization.

[0083] The following is a detailed description of each module of the traffic control system for cross-network data synchronization:

[0084] (1) The collection unit can be used as an independent thread. When a signal that requires data synchronization is detected or the data synchronization policy is enabled, the thread is started. After that, the collection unit continuously reads data from the source database and puts it into the first cache queue. When the first cache queue is filled, the data will be blocked when it is put in again, so as to control the rate at which the collection unit collects data. The size of the first cache queue and the blocking time can be configured according to the actual business scenario. The thread is closed when data synchronization is completed or the data synchronization policy is disabled.

[0085] (2) The sending unit and the collection unit are independent of each other. The two exchange data through the first cache queue. When the collection unit collects data, the thread corresponding to the sending unit is started. Figure 2 The sending unit adopts different data sending modes according to whether the first flow control server is in the flow limiting state:

[0086] If it is not in the current limiting state, data is directly taken out from the first cache queue and sent at the normal sending rate; wherein, the current limiting state can be switched according to the current limiting signal sent by the second flow control server, for example, switching to the current limiting state in response to the limit_flag=true signal; switching to the non-current limiting state in response to the limit_flag=false signal.

[0087] If it is in the current limiting state, it is necessary to control the data sending rate according to the current limiting signal. For example, a signal is obtained from the semaphore container, and data is sent only when the signal is obtained. Therefore, the data sending rate of the sending unit can be controlled by the rate at which the semaphore is put into the semaphore container. At this time, since the data sending rate of the sending unit is limited, the data acquisition rate of the acquisition unit will temporarily put data into the first cache queue at a normal rate, so the first cache queue will soon be filled with data. At this time, the acquisition unit will be blocked when writing data to the first cache queue, thereby achieving delayed control of the data acquisition rate of the acquisition unit, avoiding the direct impact of the overall data transmission efficiency of the system due to short-term fluctuations in the destination data write rate. Only when the write rate is small and lasts for a long time, will the data acquisition be limited.

[0088] (3) The semaphore setting unit may use an independent thread. After the current limiting signal sent by the second flow control server is monitored, the thread starts running. The working process of the semaphore setting unit is as follows: Figure 3 As shown. A semaphore container is defined in the shared memory. According to the load rate of the second cache queue contained in the current limiting signal and the write rate of the write unit, the rate of putting the signal into the semaphore container can be calculated, and the signal is put into the semaphore container based on the rate. Exemplarily, the rate of putting the signal can be calculated based on the following formula:

[0089] sem_rate=(1-load_factor)*(write_rate / block_size)

[0090] Wherein, sem_rate is the rate of putting in signals, load_factor is the load factor of the second cache queue, write_rate is the data writing rate of the writing unit, and block_size is the total data capacity of the first cache queue.

[0091] (4) The receiving unit is used to monitor and receive data sent by the sending unit through the network middleware, and put the data into the second cache queue. In addition, the determination and setting process of the current limiting signal can be performed by the receiving unit. Specifically, each time the receiving unit receives data, it can first determine the load rate of the second cache queue. If the load rate is greater than the set threshold (for example, 0.8), it means that the rate at which the writing unit takes out data from the second cache queue and writes it into the destination database is slow, and the rate at which the receiving unit receives data and puts it into the second cache queue is fast. At this time, the first flow control server needs to limit the flow, and the current load rate of the second cache queue, the data writing rate of the writing unit, and the current limiting flag are sent to the first flow control server, so that the first flow control server can implement data current limiting transmission.

[0092] When it is determined that the load rate of the second cache queue drops below a preset threshold (which can be set to be lower than the threshold for starting current limiting, such as 0.5), a flag for canceling current limiting is sent to the first flow control server, so that the first flow control server stops the current limiting transmission of data and resumes normal transmission. Figure 4 shown.

[0093] (5) The writing unit and the receiving unit can run in independent threads. The writing unit and the receiving unit exchange data through the second cache queue. The writing unit is used to take out data from the second cache queue and write it into the destination database until all data of this synchronization task is written to the destination database, thereby completing cross-network data synchronization.

[0094] For example, the following specific examples are listed to explain in detail the flow control process of cross-network data synchronization in this application:

[0095] 1. The user configures the file synchronization strategy on the first flow control server and the second flow control server, and enables the file synchronization strategy. At this time, the collection unit of the first flow control server starts to monitor the source database.

[0096] 2. The user adds a 10G file to the source database and wants to synchronize the file to the destination database.

[0097] 3. The acquisition unit monitors the 10G file, reads the file stream of the source database, reads the file block size block_size = 4096 bytes each time in a loop (the data reading rate is set according to actual conditions), and writes the 4096 bytes of data read each time into the first cache queue.

[0098] 4. Start the thread of the sending unit. The sending unit reads the first cache queue, establishes a connection with the second flow control server through the network middleware, and then takes out the data from the first cache queue. At this time, the flow limiting flag limit_rate = false, that is, the system is in a non-flow limiting state by default. Therefore, the taken-out data is directly sent to the second flow control server through the network middleware.

[0099] 5. After receiving the data, the receiving unit of the second flow control server writes the data into the second cache queue. At the beginning, the second cache queue is empty and can be written directly.

[0100] 6. The write unit of the second flow control server takes data out of the second cache queue and writes it into the destination database. When the write efficiency of the write unit of the destination database deteriorates due to high CPU usage of the operating system or high disk IO, the rate at which data is read from the second cache queue will be slow. As the receiving unit keeps writing data to the second cache queue, the load factor of the second cache queue (load factor = queue used capacity / total capacity) will gradually increase from 0 to 0.8 (the set threshold). At this time, the current limiting flag limit_flag = true, the data writing rate write_rate (write_rate = the amount of data read from the queue per second, for example, write_rate = 1MB / s), and the current load factor of the second cache queue (for example, load_factor = 0.8) need to be sent to the first flow control server to form a current limiting signal.

[0101] 7. The semaphore setting unit of the first flow control server obtains the current limiting signal (current limiting flag limit_flag = true, write rate write_rate = 1MB / s, load_factor = 0.8). Based on this information, the rate of placing the semaphore can be calculated:

[0102] sem_rate = (1-load_factor) * (write_rate / block_size) = 51.2 signals / second, rounded down, that is, 51 signals are put into the semaphore container per second.

[0103] 8. After the semaphore setting unit obtains limit_flag=true from the second flow control server, it synchronizes the value of limit_flag=true to the sending unit; when the sending unit determines that limit_rate=true, it also determines whether there is a signal in the semaphore container semaphore before taking out the data from the first cache queue and sending it. If there is no signal, it will block until there is a signal and then send data. Since the rate of putting signals into the semaphore container is 51 / s, the sending unit will only read the queue data 51 times per second at this time. Assuming that the data sending rate of the sending unit is 4096 bytes / time, data is sent to the second flow control server at a rate of 204KB / s.

[0104] 9. As the rate at which data is taken out of the first cache queue slows down, the collection unit of the first flow control server will gradually accumulate data until the queue is full. At this time, the rate at which the collection unit reads data can be gradually controlled until the sending is blocked (blocking occurs when the queue is full).

[0105] 10. After the write rate of the destination database (the rate at which the write unit takes out data from the second cache queue for writing) is restored, the load rate load_factor of the second cache queue in the second flow control server will gradually decrease. When the load_factor decreases to 0.5 and below 0.5 (the set threshold), the flow limiting needs to be stopped. At this time, the flag limit_flag = false to cancel the flow limiting is sent to the first flow control server.

[0106] 11. When the first flow control server receives limit_flag=false, it switches to the non-flow-limited state. After the sending unit determines that limit_flag=false, it no longer needs to determine whether the semaphore container has a signal, and it recovers to directly take data from the first cache queue and send it to the second flow control server.

[0107] 12. Repeat this process until the 10G file synchronization is completed.

[0108] The embodiment of the present application also provides a flow control method for cross-network data synchronization, which is applied to a first flow control server and may include the following steps:

[0109] S101, reading data from a source database through a collection unit, and transmitting the read data to a first cache queue;

[0110] S102, obtaining data from the first cache queue through a sending unit, and sending the obtained data to a second flow control server through a network middleware;

[0111] S103, in response to the current limiting signal sent by the second flow control server and switching to the current limiting state, controlling the data sending rate of the sending unit based on the current limiting signal;

[0112] Among them, the second flow control server includes a receiving unit and a writing unit. The receiving unit is used to receive data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a current limiting signal to the first flow control server when the load rate is not less than a preset load threshold.

[0113] It should be noted that the method provided in the embodiment of the present application is a one-sided writing method applied to the workflow of the first flow control server. The specific steps of the method can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0114] The embodiment of the present application also provides a flow control method for cross-network data synchronization, which is applied to a second flow control server and may include the following steps:

[0115] S201, receiving, by a receiving unit, data sent by a first flow control server through a network middleware, and transmitting the received data to a second cache queue;

[0116] S202, obtaining data from the second cache queue through a writing unit and writing the data into a destination database;

[0117] S203: When it is determined that the load rate of the second cache queue is not less than a preset load threshold, a current limiting signal is sent to the first flow control server, so that the first flow control server controls the data sending rate based on the current limiting signal;

[0118] Among them, the first flow control server includes a collection unit and a sending unit. The collection unit is used to read data from the source database and transmit it to the first cache queue, so that the sending unit obtains data from the first cache queue and sends it to the second flow control server through the network middleware.

[0119] It should be noted that the method provided in the embodiment of the present application is a one-sided writing method applied to the workflow of the second flow control server. The specific steps of the method can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0120] Please refer to Figure 5 , Figure 5The block diagram of the composition of the flow control device for cross-network data synchronization provided by some embodiments of the present application is shown. It should be understood that the flow control device for cross-network data synchronization corresponds to the above method embodiment and can execute each step involved in the above method embodiment. The specific functions of the flow control device for cross-network data synchronization can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0121] Figure 5 The flow control device for cross-network data synchronization includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the flow control device for cross-network data synchronization. The flow control device for cross-network data synchronization is applied to a first flow control server, including:

[0122] The data collection module 510 is used to read data from the source database through a collection unit and transmit the read data to the first cache queue;

[0123] A data sending module 520, configured to obtain data from the first cache queue through a sending unit, and send the obtained data to the second flow control server through a network middleware;

[0124] A flow control module 530, configured to control the data transmission rate of the transmission unit based on the current limiting signal in response to the current limiting signal sent by the second flow control server and switch to the current limiting state;

[0125] Among them, the second flow control server includes a receiving unit and a writing unit. The receiving unit is used to receive data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a current limiting signal to the first flow control server when the load rate is not less than a preset load threshold.

[0126] Please refer to Figure 6 , Figure 6 The block diagram of the composition of the flow control device for cross-network data synchronization provided by some embodiments of the present application is shown. It should be understood that the flow control device for cross-network data synchronization corresponds to the above method embodiment and can execute each step involved in the above method embodiment. The specific functions of the flow control device for cross-network data synchronization can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0127] Figure 6 The flow control device for cross-network data synchronization includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the flow control device for cross-network data synchronization. The flow control device for cross-network data synchronization is applied to the second flow control server, including:

[0128] The data receiving module 610 is used to receive data sent by the first flow control server through the network middleware through the receiving unit, and transmit the received data to the second cache queue;

[0129] A data writing module 620, configured to obtain data from the second cache queue through a writing unit and write the data into a destination database;

[0130] The current limiting judgment module 630 is used to send a current limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data transmission rate based on the current limiting signal;

[0131] Among them, the first flow control server includes a collection unit and a sending unit. The collection unit is used to read data from the source database and transmit it to the first cache queue, so that the sending unit obtains data from the first cache queue and sends it to the second flow control server through the network middleware.

[0132] It can be understood that the above-mentioned device item embodiments correspond to the method item embodiments of the present invention. A cross-network data synchronization flow control device provided by the embodiment of the present invention can implement the cross-network data synchronization flow control method provided by any method item embodiment of the present invention.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0134] like Figure 7 As shown, some embodiments of the present application provide an electronic device 700, which includes: a memory 710, a processor 720, and a computer program stored in the memory 710 and executable on the processor 720, wherein the processor 720 reads the program from the memory 710 through a bus 730 and executes the program to implement any of the methods of the flow control method for cross-network data synchronization as described above.

[0135] Processor 720 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.

[0136] The memory 710 may be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 720 of the disclosed embodiment may be used to execute instructions in the memory 710 to implement the method shown above. The memory 710 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0137] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon. The computer program is executed by a processor to execute the method described in the method embodiment.

[0138] Some embodiments of the present application further provide a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0139] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0142] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0143] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0145] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A flow control method for cross-network data synchronization, characterized in that: Applied to the first flow control server, including: Reading data from a source database through a collection unit, and transmitting the read data to a first cache queue; Acquire data from the first cache queue through a sending unit, and send the acquired data to a second flow control server through a network middleware; In response to the current limiting signal sent by the second flow control server and switching to the current limiting state, controlling the data sending rate of the sending unit based on the current limiting signal; Among them, the second flow control server includes a receiving unit and a writing unit, the receiving unit is used to receive the data sent by the sending unit and transmit it to the second cache queue, so that the writing unit obtains data from the second cache queue and writes it into the destination database; the second flow control server is also used to obtain the load rate of the second cache queue in real time, and send a flow limiting signal to the first flow control server when the load rate is not less than a preset load threshold.

2. The flow control method for cross-network data synchronization according to claim 1, characterized in that: The current limiting signal includes the load rate and the data writing rate of the writing unit, and the data sending rate of the sending unit is calculated according to the load rate and the data writing rate.

3. The flow control method for cross-network data synchronization according to claim 1, characterized in that: The current limiting signal includes the load rate and the data writing rate of the writing unit; In response to the current limiting signal sent by the second flow control server and switching to the current limiting state, controlling the data sending rate of the sending unit based on the current limiting signal includes: In response to the current limiting signal sent by the second flow control server and when switching to the current limiting state, calculating a semaphore placement rate based on the current limiting signal, and placing a semaphore into a semaphore container at the semaphore placement rate; The data transmission rate of the transmission unit is controlled based on the semaphore in the semaphore container.

4. The flow control method for cross-network data synchronization according to claim 1, characterized in that: Also includes: The load rate of the first cache queue is acquired in real time, and the rate at which the acquisition unit reads data from the source database is controlled based on the load rate of the first cache queue.

5. A flow control method for cross-network data synchronization, characterized in that: Applied to the second flow control server, including: Receiving, by a receiving unit, data sent by the first flow control server through the network middleware, and transmitting the received data to a second cache queue; Obtaining data from the second cache queue through a writing unit and writing the data into a destination database; When it is determined that the load rate of the second cache queue is not less than a preset load threshold, sending a current limiting signal to the first flow control server, so that the first flow control server controls the data sending rate based on the current limiting signal; The first flow control server includes a collection unit and a sending unit, wherein the collection unit is used to read data from a source database and transmit the data to a first cache queue, so that the sending unit obtains data from the first cache queue and sends the data to the second flow control server through the network middleware.

6. The flow control method for cross-network data synchronization according to claim 5, characterized in that: The current limiting signal includes the load rate and the data writing rate of the writing unit, and the data sending rate is calculated according to the load rate and the data writing rate.

7. The flow control method for cross-network data synchronization according to claim 5, characterized in that: The current limiting signal includes the load rate and the data writing rate of the writing unit; The first flow control server controls the data transmission rate based on the current limiting signal, including: The first flow control server calculates a semaphore placement rate based on the current limiting signal, and places the semaphore into the semaphore container according to the semaphore placement rate; The first flow control server controls the data sending rate of the sending unit based on the semaphore in the semaphore container.

8. A flow control system for cross-network data synchronization, characterized in that: It includes a first flow control server and a second flow control server, wherein the first flow control server includes a collection unit and a sending unit, and the second flow control server includes a receiving unit and a writing unit; The first flow control server is used to read data from the source database through the acquisition unit, and transmit the read data to the first cache queue; The first flow control server is further used for acquiring data from the first cache queue through the sending unit, and sending the acquired data to the second flow control server through the network middleware; The first flow control server is further configured to control the data transmission rate of the transmission unit based on the current limiting signal in response to the current limiting signal sent by the second flow control server and switch to the current limiting state; The second flow control server is used for receiving, through the receiving unit, the data sent by the first flow control server through the network middleware, and transmitting the received data to the second cache queue; The second flow control server is further used to obtain data from the second cache queue through the writing unit and write the data into the destination database; The second flow control server is also used to send a flow limiting signal to the first flow control server when it is determined that the load rate of the second cache queue is not less than a preset load threshold, so that the first flow control server controls the data sending rate of the sending unit based on the flow limiting signal.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the flow control method for cross-network data synchronization as described in any one of claims 1 to 7 is executed.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the flow control method for cross-network data synchronization according to any one of claims 1 to 7 is implemented.

Images