A multi-strategy breakpoint-resume transmission system based on data collection

Abstract

The present invention provides a multi-strategy breakpoint resume system based on data collection. By utilizing multiple feedbacks from a cache area, a pressure detection module, and a strategy switching module in the system, smooth switching among a real-time priority mode, a history priority mode, and a normal mode can be achieved during the breakpoint resume process, thereby calmly coping with abnormal conditions of server disconnection. In addition, in the process, feedback is used to fully avoid abnormal conditions such as exhaustion of real-time data and saturation of historical data in the system.

Description

Technical Field

[0001] The present invention relates to the field of databases, and more specifically, to a multi-strategy breakpoint resume transmission system based on data collection. Background Art

[0002] Breakpoint-resume transmission technology in data collection is a key technology that has emerged in the context of complex and changeable network transmission environments and frequent interruptions. This technology aims to solve the problem of transmission interruptions caused by factors such as network instability, server failures or user operations. By recording the data transmission progress and continuing the transmission from the interruption position after the interruption, it can effectively avoid transmission failures, save user time and traffic, and improve the efficiency and reliability of data transmission. This technology is widely used in scenarios such as file downloading, video playback, online updates, remote backup and industrial automation, ensuring the integrity and real-time nature of data during the collection and transmission process.

[0003] However, in reality, breakpoint resume technology is often rigid and can only be implemented under a single strategy, which cannot adapt to the current complex network situation. The limitation of a single function often leads to omissions or loss of data, which seriously affects the rights and interests of users. Summary of the invention

[0004] The present invention provides a multi-strategy breakpoint resume transmission system based on data collection, which effectively solves the above technical problems existing in the prior art.

[0005] Specifically, the present invention provides a multi-strategy breakpoint resume transmission system based on data acquisition, the system includes an acquisition thread, a joint cache area, a real-time data forwarding thread, a historical data forwarding thread, a pressure detection module, and a policy switching module, wherein the joint cache area includes a memory cache area and a file cache area. In normal mode, the acquisition thread acquires raw data from a data source and writes the raw data into the memory cache area. The policy switching module sends a real-time transmission instruction to the joint cache area to command the memory cache area to forward the raw data to the real-time data forwarding thread, and the real-time data forwarding thread forwards the raw data as real-time data to a database via the pressure detection module and the policy switching module. Once the system is disconnected from the database, the acquisition thread initiates intermittent automatic reconnection to the database at a specific time frequency. During the disconnection period between each two adjacent intermittent automatic reconnections, the acquisition thread still continuously acquires raw data from the data source, and these raw data are still continuously written into the memory cache area by the acquisition thread to form a real-time updated real-time data queue, and the raw data in the memory cache area is also continuously written into the file. The file cache area forms a real-time expanded historical data queue. Once the system resumes connection with the database, the memory cache area stops transmitting real-time data to the file cache area, and the historical data queue stops updating. The policy switching module sends a "real-time priority instruction" to the joint cache area, and the system enters the "real-time priority mode". The memory cache area forwards the real-time data in the real-time data queue to the database by the real-time data forwarding thread via the pressure detection module and then via the policy switching module. Once the pressure detection module detects that the data usage rate in the real-time data queue is less than the first specific ratio, the pressure detection module detects the data usage rate of the historical data queue in the file cache area again. Once the pressure detection module detects that the data usage rate in the historical data queue is greater than the second specific ratio, the policy switching module sends a "historical priority instruction" to the joint cache area, and the system switches from the "real-time priority mode" to the "historical priority mode". In the "historical priority mode", the file cache area forwards the historical data in the historical data queue to the database by the historical data forwarding thread via the pressure detection module and then via the policy switching module.

[0006] Preferably, once all historical data in the historical output queue have been transferred to the database in the "historical priority mode", the pressure detection module will detect that the historical data pressure in the file cache is 0, and the policy switching module will send a "real-time transmission instruction" to the joint storage area, and the system will immediately switch to normal mode.

[0007] Preferably, the acquisition thread automatically reconnects according to the following strategy: after the first reconnection fails, reconnect again after n seconds, and after the reconnection fails, the interval time is twice the previous time, until the interval time is k*n seconds, and each reconnection failure is reconnected again after k*n seconds, where n is an integer and n≥1, and k is an integer and k≥2.

[0008] Preferably, the real-time data queue is continuously converted to the historical data queue during the disconnection period.

[0009] Preferably, the memory cache receives data 1 at the first moment, then the current memory cache forms a real-time data queue consisting of data 1, and the historical data queue in the file cache is still empty. Subsequently, the memory cache receives data 2 at the second moment. At this time, data 1 has been written from the memory cache to the file cache and included in the historical data queue, and the real-time data queue in the memory cache is updated to the previous data 2.

[0010] Preferably, at the third moment, data 3 is received in the real-time data queue of the memory cache area again. At this time, data 2 has been input from the memory cache area to the file cache area. Therefore, the historical data queue in the file cache area will contain data 1 and data 2, and the real-time data queue will be updated to contain only data 3.

[0011] Preferably, once the pressure detection module detects that the data usage rate in the real-time data queue is always not less than the first specific ratio within the first specific time period, the pressure detection module directly sends a value increase instruction to the memory cache area to increase the value of the first specific ratio.

[0012] Preferably, before switching from the real-time priority mode to the historical priority mode, once the pressure detection module detects that the data usage rate in the historical data queue is always no greater than the second specific ratio within the second specific time period, the pressure detection module directly sends a value reduction instruction to the file cache area to lower the value of the second specific ratio.

[0013] In summary, the present invention provides a multi-strategy breakpoint resume system based on data collection. By utilizing multiple feedbacks from the cache area, pressure detection module, and strategy switching module in the system, smooth switching among the real-time priority mode, history priority mode, and normal mode can be achieved during the breakpoint resume process, thereby calmly coping with abnormal conditions such as server disconnection. In this process, feedback is used to fully avoid abnormal conditions such as exhaustion of real-time data and saturation of historical data in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be discussed below. Obviously, the technical solutions described in conjunction with the drawings are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments and their drawings can be obtained based on the embodiments shown in these drawings without paying creative work.

[0015] Figure 1A specific framework diagram of the multi-strategy breakpoint resume transmission system based on data collection according to the present invention is shown. DETAILED DESCRIPTION

[0016] The technical solutions of various embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments described in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] In summary, the present invention provides a multi-strategy breakpoint-resume system based on data collection, which has corresponding exception handling and error recovery strategies when facing network disconnection, abnormal situations and errors to ensure the continuity and integrity of data collection. Furthermore, compared with the breakpoint-resume technology currently available on the market, this patented technology can be flexibly deployed to achieve multi-function parallelism and flexible switching.

[0018] In the breakpoint resume technology of the present invention, by real-time monitoring of the connection status, in response to network disconnection or other abnormal connection situations, when the connection is disconnected, the reconnection mechanism is immediately triggered to ensure the continuity and reliability of data collection. At the same time, the automatic reconnection has certain strategies and controls. By setting the reconnection frequency, infinite reconnection operations are prevented, and frequent connection attempts are avoided. Too much burden on the network and server.

[0019] When the acquisition software detects that the network with the data target is disconnected, the breakpoint resume transmission system provided by the present invention can immediately adopt the disconnection cache strategy. This strategy temporarily saves the acquired data in a local cache file and waits for the network connection to be restored before sending the data to the target server.

[0020] In the breakpoint resume technology, it is extremely important to select the right strategy. For this reason, the strategy adopted by this patented technology is extremely flexible and can be compatible with a variety of complex emergencies. In the following, the multi-strategy breakpoint resume system based on data collection provided by the present invention will be specifically introduced, with particular emphasis on what is meant by "multi-strategy". It should be noted that the "multi-strategy" provided by the present invention is by no means the independent operation of multiple strategies, but rather the integration of multiple strategies into the same system to form a flexible operation.

[0021] Figure 1 A specific framework diagram of the multi-strategy breakpoint resume transmission system based on data collection according to the present invention is shown.

[0022] like Figure 1 As shown, in the multi-strategy breakpoint resume transmission system (hereinafter referred to as "this system") provided by the present invention, first, the collection thread continuously collects data from the data source, and the collected data is finally forwarded to Figure 1 In the database on the right, this constitutes a complete data transmission chain.

[0023] If everything is normal, then the data transmission chain can naturally operate normally and achieve stable data transmission. However, the present invention explores how to deal with abnormal data transmission.

[0024] In fact, many events may cause data transmission anomalies. For example, the database itself may fail or the network connection may be interrupted, which may lead to data transmission interruption.

[0025] To this end, the system first sets up a collection thread to directly collect raw data from the data source. Then, the collection thread writes the raw data into the memory buffer. Figure 1 The data source on the left is Figure 1 When the overall data transmission between the databases on the right side runs normally, the strategy switching module sends a real-time transmission instruction to the joint cache area, and the original data in the memory cache area in the joint cache area is forwarded to the real-time data forwarding thread, and the original data is sent from the memory cache area as real-time data in sequence to the real-time data forwarding thread. The real-time data forwarding thread smoothly forwards the real-time data to the database through the pressure detection module and the strategy switching module, thus forming a completely normal data transmission line from the data source to the database.

[0026] As mentioned above, in this completely normal data transmission line, the real-time data forwarding thread, pressure detection module, and strategy switching module do not play any role at all. They are just a station for data transmission. Figure 1 The file buffer and historical data forwarding threads shown in the figure are not mentioned at all. However, they will play a huge role in abnormal conditions. The following will explain in detail how the system containing these threads and modules will play an important role in abnormal connection conditions.

[0027] Once the system is disconnected from the database, the collection thread immediately starts the automatic reconnection mechanism. The collection thread initiates intermittent automatic reconnection to the database at a specific time frequency. For example, the collection thread automatically reconnects according to the following strategy: after the first reconnection fails, it will reconnect again after 3 seconds. After the reconnection fails, the interval time will be twice the previous time. After the interval time is 48 seconds, it will reconnect again after 48 seconds for each reconnection failure. The 3 seconds here is an example number. In fact, "3 seconds" can be replaced with n seconds in the above text. In other words, intermittent automatic reconnection can be started according to the disconnection period rules of 3 seconds, 6 seconds, 12 seconds, 24 seconds, 48 ​​seconds, 48 ​​seconds, 48 ​​seconds, ...

[0028] During the disconnection period between two adjacent intermittent automatic reconnections, the collection thread continues to collect raw data from the data source, and these raw data are still continuously written by the collection thread into the memory cache area to form a real-time data queue.

[0029] During the disconnection period, the original data in the memory cache is also continuously written into the file cache. In the file cache, the original data at this moment will become the historical data of the next moment. Therefore, a historical data queue will be formed in the file cache according to the historical evolution during the disconnection period.

[0030] It should be noted that the real-time data queue can be continuously converted to the historical data queue during the disconnection period. For example, the memory cache receives data 1, which occupies 1Mb of space, at the first moment (for example, the 10th millisecond), then the current memory cache forms a real-time data queue consisting of data 1, and the historical data queue in the file cache is still empty. Subsequently, at the second moment (for example, the 20th millisecond), data 2, which occupies 0.5Mb of space, is received. At this time, data 1 has been written from the memory cache to the file cache and included in the historical data queue, and the real-time data queue in the memory cache is updated to the previous data 2. Then, for example, at the third moment (for example, the 30th millisecond), the real-time data queue in the memory cache receives data 3, which occupies 1Mb of space, and the original data 2 has been input from the memory cache to the file cache. Therefore, the historical data queue in the file cache will contain the initial data 1 and the previous data 2, and the real-time data queue will be updated to contain only data 3, and so on.

[0031] In other words, during the disconnection period, the memory cache area keeps a real-time data queue that is updated at any time, and as the real-time data is continuously written to the file cache area, the file cache area keeps a continuously expanding historical data queue.

[0032] Once the system is reconnected to the database, the memory cache stops transmitting real-time data to the file cache. In other words, from the moment the connection is restored, the historical data queue stops updating, and the memory cache continues to receive real-time data collected by the collection thread from the data source.

[0033] At this time, the policy switching module sends a "historical priority instruction" or a "real-time priority instruction" to the joint cache area composed of the memory cache area and the file cache area (as to the basis on which the policy switching module sends which of the two types of instructions will be mentioned below).

[0034] Once the strategy switching module sends a "real-time priority instruction" to the joint cache area, the system enters the "real-time priority mode". In the "real-time priority mode", the memory cache area sends the real-time data in the real-time data queue to the pressure detection module by the real-time data forwarding thread, and then forwards it to the database through the pressure detection module and the strategy switching module, thereby completing the database's reception of real-time data.

[0035] Among them, once the pressure detection module receives the real-time data forwarded by the real-time data forwarding thread, the pressure detection module immediately starts the utilization rate detection of the real-time data queue in the memory cache area. Once the pressure detection module detects that the data utilization rate in the real-time data queue is less than a first specific proportion (for example, 40%), in other words, the real-time data queue itself has a real-time planned total capacity, and the actual space occupied by the real-time data queue after data transmission is less than the first specific proportion of the real-time planned total capacity, the pressure detection module will then turn to detect the utilization rate of the historical data queue in the file cache area.

[0036] Once the pressure detection module detects that the data usage rate in the historical data queue is greater than a second specific proportion (for example, 40%), in other words, the historical data queue itself has a historical planned total capacity, and the space actually occupied by the historical data queue after data transmission is greater than the second specific proportion of the historical planned total capacity, the policy switching module sends a "historical priority instruction" to the joint cache area, thereby completing the switching of the system from "real-time priority mode" to "historical priority mode".

[0037] In the "historical priority mode", the file cache sends the historical data in the historical data queue from the historical data forwarding thread to the pressure detection module, and then forwards it to the database through the pressure detection module and the policy switching module, thereby completing the database's reception of historical data.

[0038] Preferably, once all historical data in the historical output queue have been transmitted to the database in the "historical priority mode", the pressure detection module will detect that the historical data pressure in the file cache is 0, and the policy switching module will send a "real-time transmission instruction" to the joint storage area, and the system will transmit real-time data to the database normally.

[0039] It should be noted that the "real-time transmission instruction" and the "real-time priority instruction" are different. When sending the "real-time transmission instruction", the historical data in this system has been completely cleared through transmission, and the database connection is now completely unobstructed. Therefore, under the "real-time transmission instruction", this system will send real-time data to the database in the most normal way. However, the "real-time priority instruction" is different. When sending the "real-time priority instruction", since the historical data still exists in the file cache area, this system is still in an "abnormal" state. Under certain conditions (mentioned above, no longer repeated here), the "real-time priority instruction" will be converted into a "historical priority instruction".

[0040] It can be seen that in this system, the "historical priority mode" and "real-time priority mode" as well as the normal mode can be flexibly switched within the system according to specific conditions, thereby ensuring that the system can operate normally. In fact, when considering the "real-time priority mode", we strive to avoid the total capacity occupancy rate of real-time data being too low, because once the space occupied by real-time data is too low, it means that the real-time data is exhausted. At this time, there is no need to transmit real-time data, and instead switch to the "historical priority mode" to transmit historical data; conversely, when considering the "historical priority mode", we strive to avoid the total capacity occupancy rate of historical data being too high, in other words, avoid excessive accumulation of historical data.

[0041] As described above, it can be seen that the system has formed effective feedback among the pressure detection module, the strategy switching module and the joint cache area, thereby being able to flexibly switch between the three modes of "historical priority mode", "real-time priority mode" and normal mode, thereby ensuring that in the process of the system from "abnormal" to "normal", historical data and real-time data are transmitted at the appropriate time point without data exhaustion or data saturation during transmission.

[0042] Furthermore, the system can also provide further feedback. As mentioned above, to trigger the switch from "real-time priority mode" to "historical priority mode", two basic conditions need to be met, namely, first, the pressure detection module detects that the data usage rate in the real-time data queue is less than the first specific ratio, and secondly, the pressure detection module detects that the data usage rate in the historical data queue is greater than the second specific ratio. If these two basic conditions cannot be met for a long time, the switch from "real-time priority mode" to "historical priority mode" cannot be completed.

[0043] In view of this, preferably, once the pressure detection module detects that the data usage rate in the real-time data queue is always not less than (in other words, greater than or equal to) the first specific ratio within a first specific time period (for example, 30 minutes, or 2 hours, or 1 day, etc.), it indicates that the first specific ratio is set too low, and the pressure detection module directly sends a value increase instruction to the memory cache area to increase the value of the first specific ratio, for example, from 40% to 45%, or even to 50%.

[0044] Preferably, before the real-time priority mode is switched to the history priority mode, once the pressure detection module detects that the data usage rate in the historical data queue is always no greater than (in other words, less than or equal to) the second specific ratio within a second specific time period (for example, 30 minutes, or 2 hours, or 1 day, etc.), it indicates that the second specific ratio is set too high, and the pressure detection module directly sends a value reduction instruction to the file cache area to lower the value of the second specific ratio, for example, from 40% to 35%, or even to 30%.

[0045] At this point, the multi-strategy breakpoint resume system based on data collection provided by the present invention has been introduced. In summary, the present invention provides a multi-strategy breakpoint resume system based on data collection, which utilizes multiple feedbacks from the buffer area, pressure detection module, and strategy switching module in the system to smoothly switch between the real-time priority mode, the history priority mode, and the normal mode during the breakpoint resume process, thereby calmly coping with abnormal conditions of server disconnection, and in this process, fully avoiding abnormal conditions such as exhaustion of real-time data and saturation of historical data in the system through feedback.

[0046] The above description is only an exemplary embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-strategy breakpoint resume system based on data collection, characterized in that: The system includes a collection thread, a joint cache area, a real-time data forwarding thread, a historical data forwarding thread, a pressure detection module, and a strategy switching module. The joint cache area includes a memory cache area and a file cache area. In normal mode, the acquisition thread collects raw data from the data source and writes the raw data into the memory cache area. The strategy switching module sends a real-time transmission instruction to the joint cache area to command the memory cache area to forward the raw data to the real-time data forwarding thread, and the real-time data forwarding thread forwards the raw data as real-time data to the database via the pressure detection module and the strategy switching module. Once the system is disconnected from the database, the collection thread will initiate intermittent automatic reconnection to the database at a specific time frequency. During the disconnection period between each two adjacent intermittent automatic reconnections, the collection thread will continue to collect raw data from the data source. These raw data will still be written by the collection thread into the memory cache to form a real-time updated real-time data queue. The raw data in the memory cache will also be continuously written into the file cache to form a real-time expanded historical data queue. Once the system is reconnected to the database, the memory cache will stop transmitting real-time data to the file cache, and the historical data queue will stop updating. The strategy switching module sends a "real-time priority instruction" to the joint cache area, and the system enters the "real-time priority mode". The memory cache area forwards the real-time data in the real-time data queue to the database through the real-time data forwarding thread, the pressure detection module, and the strategy switching module. Once the pressure detection module detects that the data usage rate in the real-time data queue is less than the first specific ratio, the pressure detection module will detect the data usage rate of the historical data queue in the file cache area. Once the pressure detection module detects that the data usage rate in the historical data queue is greater than the second specific ratio, the strategy switching module sends a "historical priority instruction" to the joint cache area, and the system switches from "real-time priority mode" to "historical priority mode". In the "historical priority mode", the file cache forwards the historical data in the historical data queue to the database through the historical data forwarding thread, the pressure detection module, and then the strategy switching module.

2. The system according to claim 1, characterized in that Once all historical data in the historical output queue have been transferred to the database in the "historical priority mode", the pressure detection module will detect that the historical data pressure in the file cache area is 0, and the strategy switching module will send a "real-time transmission instruction" to the joint cache area, and the system will immediately switch to normal mode.

3. The system according to claim 1, characterized in that The collection thread automatically reconnects according to the following strategy: after the first reconnection fails, it will reconnect again after n seconds. After the reconnection fails, the interval between each reconnection is twice the previous one until the interval is k*n seconds. After each reconnection failure, it will reconnect again after k*n seconds, where n is an integer and n≥1, and k is an integer and k≥2.

4. The system according to claim 1, characterized in that The real-time data queue is continuously converted to the historical data queue during the disconnection period.

5. The system according to claim 4, characterized in that The memory cache receives data 1 at the first moment, then the current memory cache forms a real-time data queue consisting of data 1. At this time, the historical data queue in the file cache is still empty. Subsequently, the memory cache receives data 2 at the second moment. At this time, data 1 has been written from the memory cache to the file cache and included in the historical data queue, and the real-time data queue in the memory cache is updated to the data 2.

6. The system according to claim 5, characterized in that At the third moment, data 3 is received in the real-time data queue of the memory cache area. At this time, data 2 has been input from the memory cache area to the file cache area. Therefore, the historical data queue in the file cache area will contain data 1 and data 2, and the real-time data queue will be updated to contain only data 3.

7. The system according to claim 1, characterized in that Once the pressure detection module detects that the data usage rate in the real-time data queue is always not less than the first specific ratio within the first specific time period, the pressure detection module directly sends a value increase instruction to the memory cache area to increase the value of the first specific ratio.

8. The system according to claim 1, characterized in that Before switching from real-time priority mode to historical priority mode, once the pressure detection module detects that the data usage rate in the historical data queue is always no greater than the second specific ratio within the second specific time period, the pressure detection module directly sends a value reduction instruction to the file cache area to lower the value of the second specific ratio.

Images