A data recovery method and apparatus

Abstract

The application relates to the technical field of system management, and provides a data recovery method and device.The method comprises the following steps: when being online, generating target data according to historical service data received from an upstream component, storing the target data into a target cache, sequentially taking out the target data from the target cache, and delivering the target data to a downstream component for data synchronization of the downstream component; during the delivery process, generating incremental target data according to incremental service data received from the upstream component, storing the incremental target data into the target cache; after the delivery is completed, sequentially taking out the incremental target data from the target cache, and delivering the incremental target data to the downstream component for data updating of the downstream component.The application avoids the intervention of external storage, makes the data recovery process more stable, and avoids communication interruption in the data recovery process by means of lag processing of incremental service data delivered in the normal operation process of the system, so that the system as a whole still maintains response to the outside.

Description

Technical Field

[0001] This invention relates to the field of system management technology, and in particular to a data recovery method and apparatus. Background Technology

[0002] In systems with multiple components, it is often necessary to upgrade, restart, or take a component offline for maintenance without shutting down the system. During this process, the data in the component may be changed. When the component comes back online, in order to ensure that the component can work normally, it is necessary to ensure that the data in the component is consistent with the data when it was offline. If the component is connected to downstream components, since the data in the downstream components needs to be consistent with the data in the component, it is also necessary to synchronize the data of the downstream components. This process is collectively referred to as data recovery.

[0003] When a component has both upstream and downstream components, it is called a midstream component. When the midstream component performs data recovery, the methods commonly used in existing technologies are as follows: Figure 1 As shown, before a midstream component goes offline, its own midstream business data is completely copied to external storage. When the midstream component comes back online, it reads the midstream business data from external storage and verifies it with the business data received from the upstream component after the midstream component comes back online. Based on the verification result, it updates and restores its own midstream business data. After its own midstream business data is restored, it generates target data for downstream component data synchronization and sends the target data to the downstream component.

[0004] In existing technologies, the process of restoring data from external storage usually takes a long time, and the process of verifying and generating target data usually requires shutting down the data transmission of upstream components, that is, interrupting the communication between upstream and midstream components, in order to prevent the upstream components from sending business data that interferes with the normal operation of the system and causes data recovery errors. After the data is restored, the communication between upstream and midstream components is re-established, and only then can the business data that is in normal operation of the system be transmitted. Due to the interruption of communication, the system as a whole is unresponsive to external parties.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem this invention aims to solve is that during component data recovery, the entire system becomes unresponsive due to communication interruptions.

[0007] The present invention adopts the following technical solution:

[0008] A data recovery method includes: generating target data based on historical business data received from an upstream component upon going online, storing the target data in a target cache, retrieving the target data sequentially from the target cache, and sending the target data to a downstream component for data synchronization of the downstream component;

[0009] During the process of sending target data to downstream components, incremental target data is generated based on the incremental business data received from upstream components, and the incremental target data is stored in the target cache.

[0010] After the target data is sent to the downstream components, the incremental target data is retrieved from the target cache in sequence and sent to the downstream components for data updates.

[0011] Preferably, the step of generating target data based on historical business data received from upstream components and storing the target data in the target cache specifically includes:

[0012] Receive historical business data and generate one or more process data based on one or more business objects in the historical business data;

[0013] If it is found that the business object in the newly received historical business data is the same as the business object in the previously received historical business data, the process data of the business object will not be generated based on the newly received historical business data. Instead, the process data of the previously generated business object will be updated based on the newly received historical business data.

[0014] After receiving all historical business data, target data is generated based on the process data, and the target data is stored in the target cache.

[0015] Preferably, the step of generating target data based on process data and storing the target data in the target cache specifically includes:

[0016] Target data is generated according to the process data in the data format required by the downstream components, and the target data is stored in the target cache according to the receiving order required by the downstream components.

[0017] Preferably, storing the target data in the target cache according to the receiving order of the target data required by the downstream component specifically includes:

[0018] If the downstream component requires the target data to be received in the order of the target data types, then different linked lists are defined according to the different target data types, and target data of the same type are stored in the same linked list.

[0019] According to the receiving order of the target data type required by the downstream component, the nodes in each linked list are attached to the target cache in the required order.

[0020] Preferably, the step of generating incremental target data based on incremental service data received from upstream components and storing the incremental target data in the target cache specifically includes:

[0021] Based on the usage of the target cache, control the receiving rate of incremental service data, generate incremental target data based on the incremental service data received from the upstream component, and store the incremental target data in the target cache.

[0022] Preferably, controlling the receiving rate of incremental service data based on the usage of the target cache specifically includes:

[0023] If the amount of data in the target cache is greater than or equal to a preset first data amount, the receiving rate of incremental service data is reduced to the minimum receiving rate; if the amount of data in the target cache is less than or equal to a preset second data amount, the receiving rate of incremental service data is restored to the normal receiving rate; if the amount of data in the target cache is less than the preset first data amount but greater than the preset second data amount, the receiving rate of incremental service data is controlled according to the changing trend of the amount of data in the target cache.

[0024] Preferably, controlling the receiving rate of incremental service data based on the changing trend of the target cache data volume specifically includes:

[0025] If the data in the target cache continues to increase within a preset time period, and the rate of data increase always exceeds the preset maximum rate, then the receiving rate of incremental service data will be reduced to the minimum receiving rate; if the data in the target cache continues to increase, and the rate of data increase never exceeds the preset minimum rate, or the data in the target cache decreases or does not increase, then the receiving rate of incremental service data will be restored to the normal receiving rate.

[0026] Preferably, the step of sequentially retrieving target data from the target cache and sending the target data to downstream components specifically includes:

[0027] The target data in the target cache is retrieved sequentially and sent to the target pipeline. The downstream components read the target data from the target pipeline in real time.

[0028] If the number of target data in the target pipeline exceeds a preset number, the rate at which target data is retrieved from the target cache will be reduced to the minimum sending rate; if the number of target data in the target pipeline is less than the preset minimum number, the rate at which target data is retrieved from the target cache will be restored to the normal sending rate.

[0029] Preferably, the step of sequentially retrieving incremental target data from the target cache and sending the incremental target data to downstream components specifically includes:

[0030] Incremental target data is retrieved sequentially from the target cache and sent to downstream components. The receiving rate of incremental business data is controlled based on the usage of the target cache.

[0031] In a second aspect, the present invention also provides a data recovery apparatus for implementing the data recovery method described in the first aspect, the apparatus comprising:

[0032] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the data recovery method described in the first aspect.

[0033] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the data recovery method described in the first aspect.

[0034] This invention receives historical business data from upstream components and performs data recovery based on this data, avoiding the intervention of external storage and making the data recovery process more stable. Furthermore, by delaying the processing of incremental business data sent during normal system operation, it avoids communication interruptions during data recovery, ensuring the system remains responsive. A preferred implementation of this invention further controls the reception of incremental business data based on the usage of the target cache, effectively balancing the load on various components during data recovery. In the preferred method, the target cache is monitored in greater detail; based on the trend of data volume changes in the target cache, the possibility of system load imbalance is predicted and prevented in advance, further balancing the system load. Even after data recovery, the preferred method continues to monitor and control system load balance, ensuring that the load on various components remains balanced during normal system operation. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0036] Figure 1 This is a schematic diagram of a prior art data recovery method provided by the present invention;

[0037] Figure 2 This is a flowchart of a data recovery method provided in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of a data recovery method provided in an embodiment of the present invention;

[0039] Figure 4 This is a flowchart of a data recovery method provided in an embodiment of the present invention;

[0040] Figure 5 This is a flowchart of a data recovery method provided in an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram illustrating an application scenario of a data recovery method provided in an embodiment of the present invention;

[0042] Figure 7 This is a flowchart of a data recovery method provided in an embodiment of the present invention;

[0043] Figure 8 This is a schematic diagram illustrating an application scenario of a data recovery method provided in an embodiment of the present invention;

[0044] Figure 9 This is a schematic diagram of a data recovery method provided in an embodiment of the present invention;

[0045] Figure 10 This is a flowchart of a data recovery method provided in an embodiment of the present invention;

[0046] Figure 11 This is a schematic diagram of the architecture of a data recovery device provided in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0049] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0050] Example 1:

[0051] Embodiment 1 of the present invention provides a data recovery method, such as... Figure 2 As shown, the method specifically includes:

[0052] In step 201, when going online, target data is generated based on historical business data received from upstream components, the target data is stored in the target cache, the target data is retrieved from the target cache in sequence, and the target data is sent to downstream components for data synchronization of downstream components.

[0053] The historical business data refers to business data that has been previously issued and can generate target data. When going online, the target data is generated by re-issuing the historical business data, thereby achieving data synchronization of downstream components. The number of historical business data can be one or more. A piece of historical data can be issued as a complete data packet or split into multiple data packets. Depending on the application scenario of the present invention embodiment, it is possible to generate one or more target data based on one piece of historical business data, or to generate one target data based on multiple pieces of historical business data.

[0054] The components that go online are referred to as midstream components. After a midstream component goes online, the upstream component sends historical business data to it. The midstream component then generates its own midstream business data based on this historical data. This midstream business data is the data stored in the midstream component during normal business operation, restoring the midstream component's own data to be consistent with when it went offline. The target data is then generated based on this restored midstream business data. This process, through interaction with the upstream component, generates its own midstream business data and target data based on historical business data, eliminating the need for external storage during the data restoration process.

[0055] When a downstream component receives target data, it compares the target data with its own data. If the target data differs from its own data, it uses the target data to reconstruct its own data. The rate at which target data is retrieved sequentially from the target cache is determined by those skilled in the art based on performance analysis of downstream components processing target data.

[0056] In step 202, during the process of sending target data to downstream components, incremental target data is generated based on the incremental business data received from upstream components, and the incremental target data is stored in the target cache.

[0057] The incremental business data refers to the business data issued during the normal operation of the system. The step of generating incremental target data based on the incremental business data received from the upstream component specifically involves: updating the midstream business data of the midstream component based on the incremental business data, and generating incremental target data based on the incremental business data and the midstream business data.

[0058] Since the process of generating incremental target data also requires modifying midstream business data, if the generation of incremental target data is processed at the same time as the generation of target data for downstream data synchronization, the midstream business data may be modified by the incremental business data, resulting in incorrect generated target data and consequently data recovery errors. Therefore, the incremental business data is temporarily left unprocessed.

[0059] During the target data distribution process, the generation of the target data is already complete. However, the process of distributing the target data to downstream components takes a long time because the downstream components need to synchronize data based on the target data. During this process, the midstream components process incremental business data to generate incremental target data. In this way, on the one hand, the correctness of data recovery can be guaranteed without shutting down the data distribution of upstream components. On the other hand, by generating incremental target data and distributing target data simultaneously, the load of midstream components can be fully utilized, shortening the waiting time for incremental business data processing. This allows the system to enter normal operation as soon as possible after the data recovery is completed.

[0060] In step 203, after the target data is sent to the downstream components, incremental target data is retrieved from the target cache in sequence and sent to the downstream components for data updates.

[0061] The data update refers to updating component data according to the system's business needs during normal system operation. If incremental business data is processed and distributed during the distribution of target data, it may interfere with the data synchronization process of downstream components, leading to data synchronization errors. Alternatively, after the incremental business data is processed and distributed, data may be changed due to data synchronization, rendering the incremental business data ineffective. Therefore, incremental target data is distributed only after the target data distribution is completed.

[0062] A schematic diagram of a scenario according to an embodiment of the present invention is shown below. Figure 3 As shown. This embodiment of the invention generates midstream business data and target data by receiving historical business data from upstream components. On the one hand, it prevents the intervention of external storage during the data recovery process, making the data recovery process more stable. On the other hand, by delaying the processing of incremental business data sent during normal system operation until after the target data generation is completed, and delaying the sending of incremental target data until after the target data sending is completed, the data recovery process is not affected by the sending of incremental business data. This ensures that the data synchronization and data update processes of the components proceed in an orderly manner, guaranteeing normal data recovery. Furthermore, it ensures that communication between upstream and downstream components is uninterrupted, allowing the system to maintain responsiveness to the outside world during the data recovery process.

[0063] In practical use, the historical business data is typically all historical business data recorded when the service is taken offline. In conjunction with embodiments of the present invention, such as... Figure 4 As shown, the related steps in this embodiment will be further integrated to provide a relatively complete logical demonstration:

[0064] In step 200, the upstream component is notified when the system goes offline, and the upstream component records historical business data that can generate the target data at this time.

[0065] The historical business data refers to business data that has been issued by the upstream component and can generate the target data. Therefore, it can be recorded by the upstream component itself without the need for external storage. In some cases, the upstream component is a management component, that is, it has management functions and is used to store and manage the data structure that can generate data for itself, the midstream component, and the downstream component. In this case, the upstream component does not need to allocate an extra storage space for historical business data. Instead, it only needs to record the storage location of the data structure managed by the upstream component when the midstream component goes offline, keep the storage structure unchanged, and store the business data during the subsequent system operation in other locations. This is to record the historical business data that can generate the target data at this time, thereby saving cache space.

[0066] In some cases, multiple historical business data entries may target the same business object. That is, multiple historical business data entries may simply set different values ​​for the same piece of information about the same business object. Since downstream components only focus on the final value of the business object's information after processing all historical business data, and are not concerned with changes in these values ​​during the historical business data processing, there is no need to issue multiple target data entries causing frequent data updates in downstream components. Only target data with the final value needs to be issued. Based on the embodiments of this invention, the following preferred embodiments exist:

[0067] Receive historical business data and generate one or more process data based on one or more business objects in the historical business data.

[0068] If it is found that the business object in the newly received historical business data is the same as the business object in the previously received historical business data, then the process data of the business object will not be generated based on the newly received historical business data. Instead, the process data of the previously generated business object will be updated based on the newly received historical business data.

[0069] After receiving all historical business data, target data is generated based on the process data, and the target data is stored in the target cache.

[0070] A single historical business data entry may contain multiple business objects, or multiple historical business data entries may be used to describe different attribute values ​​of a single business object. That is, a single historical business data entry may generate multiple process data entries, or multiple historical business data entries may generate a single process data entry. Since historical business data may contain multiple business objects, some of these objects may be identical to those in previously received historical business data, while others may differ. In such cases, process data is updated only for the identical business objects; new process data still needs to be generated for the different business objects.

[0071] By updating the old process data, it is ensured that there is only one set of process data for each business object. Target data is not generated until all historical business data has been received; that is, data is not sent to downstream components to avoid sending values ​​that have not been fully updated. After receiving all historical business data, target data is generated based on the process data to ensure that the value of every piece of information for every business object received by downstream components is the latest value.

[0072] This preferred embodiment can avoid repeatedly sending target data to the same business object and reduce the data recovery time after component restart.

[0073] There are also cases where downstream components have strict requirements regarding the format and order of data transmission. In such cases, this preferred embodiment further includes:

[0074] Target data is generated according to the process data in the data format required by the downstream components, and the target data is stored in the target cache according to the receiving order required by the downstream components.

[0075] Since a process data is generated for a business object, but the data that the downstream component needs to receive may not be for the business object, the process data is processed to make the data format conform to the data format required by the downstream component. For example, if the data that the downstream component needs to receive is for the attributes under the business object, the process data is split into attribute values ​​as target data; if the downstream component needs data for the business object, the process data is used as target data.

[0076] The step of storing the target data in the target cache according to the receiving order of the target data required by the downstream components can be achieved by allocating a contiguous block of physical memory as the target cache and storing the target data contiguously according to a certain byte size, or by using a linear list as the target cache, or by generating a set of data in a specified order, such as storing the addresses of each target data in order.

[0077] If the downstream component requires the target data to be received in the order of its type, then different linked lists are defined according to the different types of target data, and target data of the same type are stored in the same linked list.

[0078] According to the receiving order of the target data type required by the downstream component, the nodes in each linked list are attached to the target cache in the required order.

[0079] Each node in the linked list stores a target data. The step of attaching the nodes of each linked list to the target cache in the required order is to traverse the target data in the linked list starting from the head of the linked list, attach the target data to the target cache, and attach all the target data in one linked list to the target cache before attaching the next linked list.

[0080] This preferred embodiment can adapt to systems where downstream components have strict requirements on the form and order of data delivery, enabling target data to be delivered in the order and form required by downstream components, thus meeting the system's data recovery needs.

[0081] In this embodiment, since the processes of sending target data and generating incremental target data occur simultaneously, and given that the load that the upstream component can withstand, i.e., the amount of data it can process, is fixed, the processing rates of these two processes will inevitably affect each other. That is, when the upstream component processes a large amount of incremental target data generation, the target data sending rate will inevitably slow down. Furthermore, since the normal operation of the system will be affected before the data recovery is completed, it is necessary to prioritize ensuring that the data recovery can be completed as quickly as possible, i.e., to prioritize the target data sending process. To address this issue, in conjunction with the embodiments of the present invention, the following preferred implementation methods exist:

[0082] Based on the usage of the target cache, control the receiving rate of incremental service data, generate incremental target data based on the incremental service data received from the upstream component, and store the incremental target data in the target cache.

[0083] The usage of the target cache includes: the cache size occupied by target data, the cache size occupied by incremental target data, and the total amount of data the target cache can hold. Since the target cache is typically located within the midstream component, its data capacity matches the load capacity (i.e., the amount of data it can process) of the midstream component. Therefore, the usage of the target cache can, to some extent, represent the load condition of the midstream component. By analyzing the target cache usage, the quantity and distribution rate of target data are determined. Based on these, the receiving rate of incremental business data is controlled, thereby controlling the generation rate of incremental target data. This achieves a dynamic balance between the generation and distribution of incremental target data, ensuring that data recovery can be completed as quickly as possible while controlling the rate of incremental target data generation based on the load of the midstream component.

[0084] One approach to controlling the receiving rate of incremental service data is to notify the upstream component, enabling it to control the rate at which incremental service data is sent. Another approach is to establish a pipeline for sending incremental service data from the upstream component to the midstream component. The midstream component controls the rate at which it reads incremental service data from the pipeline, while the upstream component controls its sending rate based on the amount of incremental service data in the pipeline. Both approaches allow for control of the incremental service data sending rate, enabling the upstream component to adjust the rate at which data is transferred from the upstream component to the midstream component based on the midstream component's load, thereby maintaining load balance between the upstream and midstream components.

[0085] The preferred implementation described above can dynamically adjust the receiving rate of incremental business data based on the usage of the target cache, thereby controlling the dynamic balance between the generation of incremental target data and the distribution of target data by the midstream component. At the same time, by controlling the distribution rate of incremental business data based on the load of the midstream component, the load balance between the upstream and midstream components can be maintained.

[0086] The above implementation controls the receiving rate of incremental service data by adjusting the cache size of the target data, the cache size of the incremental target data, and the total amount of data the target cache can hold. When the target cache is too small, the receiving rate of incremental service data is too fast, or the sending rate of target data is too slow, the amount of data in the target cache may suddenly increase, resulting in an excessive load on the midstream component. In this case, although controlling the incremental service receiving rate reduces the generation rate of incremental target data, it still requires a period of time for the target data to be sent back to normal before the amount of data in the target cache can return to a normal range, thus restoring a balance between the generation and sending of incremental target data by the midstream component. To address this issue, this embodiment addresses the problem as follows: Figure 5As shown, the following are preferred implementation methods:

[0087] In step 301, if the amount of data in the target cache is greater than or equal to a preset first data amount, the receiving rate of incremental service data is reduced to the minimum receiving rate; if the amount of data in the target cache is less than or equal to a preset second data amount, the receiving rate of incremental service data is restored to the normal receiving rate; if the amount of data in the target cache is less than the preset first data amount but greater than the preset second data amount, the receiving rate of incremental service data is controlled according to the changing trend of the amount of data in the target cache.

[0088] The preset first data volume, preset second data volume, minimum receiving rate, and normal receiving rate are derived by those skilled in the art based on experience and analysis of the amount of target data that the target buffer can allow. Among them, the minimum receiving rate is less than the normal receiving rate, the preset first data volume is greater than or equal to the preset second data volume, and the first data volume is less than or equal to the maximum data volume that the target buffer can allow.

[0089] In step 302, if the data in the target cache continues to increase within a preset time period, and the rate of data increase always exceeds the preset maximum rate, then the receiving rate of incremental service data is reduced to the minimum receiving rate; if the data in the target cache continues to increase, and the rate of data increase never exceeds the preset minimum rate, or the data in the target cache decreases or does not increase, then the receiving rate of incremental service data is restored to the normal receiving rate.

[0090] The preset time, preset maximum rate, and preset minimum rate are derived by those skilled in the art based on experience, wherein the preset maximum rate is greater than or equal to the preset minimum rate.

[0091] When the amount of data in the target cache increases, it indicates that the rate at which the target data is sent is less than the rate at which the incremental business data is generated. When the amount of data in the target cache decreases or does not increase, it indicates that the rate at which the target data is sent is greater than or equal to the rate at which the incremental business data is generated. If the rate at which the data increases is greater than or equal to the preset maximum rate for a period of time, it may cause the amount of data in the target cache to exceed the preset first data amount. Conversely, if the rate at which the data increases is always less than the preset minimum rate, or if the rate at which the data in the target cache decreases or does not increase for a period of time, the amount of data in the target cache can be kept at or below the preset first data amount. Therefore, by observing the changing trend of the data volume in the target cache, the data volume in the target cache can be predicted in the future. By controlling the receiving rate of incremental business data based on the changing trend of the data volume in the target cache, the generation rate of incremental target data can be controlled. To a certain extent, this prevents the data volume in the target cache from exceeding the preset first data volume, thus achieving the purpose of preventing the data volume in the target cache from exceeding the limit. That is, based on the changing trend of the data volume in the target cache, the relative rate of incremental target data generation and target data distribution in the midstream component can be determined. Based on this relative rate, possible load changes in the midstream component can be predicted. Thus, by controlling the generation rate of incremental target data, the dynamic balance between the generation and distribution of incremental target data in the midstream component can be prevented from being broken, so that the load of the midstream component can remain stable. At the same time, the load between the midstream component and the upstream component can also be kept stable and balanced.

[0092] This preferred implementation can maintain a stable balance between the generation of incremental target data and the distribution of target data in the midstream component, and also maintain a stable load balance between the midstream component and the upstream component.

[0093] When the load capacity of downstream components is limited, and the rate at which target data is sent is too fast, the downstream components may not be able to process it in time. To address this situation, the following implementation method is provided in accordance with the embodiments of the present invention:

[0094] The rate at which target data is sent is controlled based on the load of downstream components.

[0095] In other words, the midstream component balances the load distribution between the midstream and downstream components based on the load status of the downstream components. There are several ways to implement this, one of which is that the downstream component announces its own load status to the midstream component, and the midstream component controls the data delivery rate of the target data according to the load status.

[0096] Another preferred implementation method is:

[0097] The target data is retrieved sequentially from the target cache and sent to the target pipeline. Downstream components read the target data from the target pipeline in real time.

[0098] If the number of target data in the target pipeline exceeds a preset number, the rate at which target data is retrieved from the target cache will be reduced to the minimum sending rate; if the number of target data in the target pipeline is less than the preset minimum number, the rate at which target data is retrieved from the target cache will be restored to the normal sending rate.

[0099] The preset quantity, minimum sending rate, and normal sending rate are derived by those skilled in the art based on experience, wherein the minimum sending rate is less than the normal sending rate.

[0100] The process of sending the extracted target data to the target pipeline, and the downstream components reading the target data from the target pipeline in real time, specifically involves:

[0101] The target data is sequentially sent to one end of the target pipeline, and the downstream component reads the target data from the other end of the target pipeline to achieve the first-in-first-out effect of the target data. When the downstream component has requirements on the receiving order of the target data, the target data is sequentially sent to one end of the target pipeline according to the receiving order required by the downstream component.

[0102] The rate at which downstream components read target data from the other end of the target pipeline is determined by the downstream component's own data processing capabilities, that is, by the load that the downstream component can withstand.

[0103] This preferred embodiment controls the target data delivery rate based on the load of downstream components, thereby balancing the load between midstream and downstream components. Since the target data delivery process involves retrieving the target data from the target cache and sending it to the downstream components, the delivery rate directly affects the amount of target data in the target cache. This implementation, combined with the aforementioned method of controlling the incremental service data reception rate based on target cache usage, can balance the load distribution between midstream and downstream components based on the load of downstream components, and further balance the load distribution between upstream and midstream components. This achieves load balancing among upstream, midstream, and downstream components in the system, ensuring overall load balancing during data recovery.

[0104] In some cases, it is necessary not only to ensure overall load balancing during data recovery but also to maintain load balancing during normal system operation. For such situations, the following are preferred implementation methods:

[0105] Incremental target data is retrieved sequentially from the target cache and sent to downstream components. The receiving rate of incremental business data is controlled based on the usage of the target cache.

[0106] By controlling the rate at which incremental service data is received and the rate at which incremental target data is generated, the rate at which incremental target data is sent to downstream components can be controlled, thereby controlling the load balancing of upstream, midstream, and downstream components.

[0107] In this embodiment of the invention, the upstream, midstream, and downstream components do not refer to components at fixed positions in the system, but rather to components with a hierarchical relationship during data distribution. Specifically, during data distribution, the midstream component is subordinate to the upstream component and receives data distributed by the upstream component, while the downstream component is subordinate to the midstream component and receives data distributed by the midstream component. When other components below the downstream component receive data distributed by the downstream component, these components are not directly aware of the midstream component's restart process. If the midstream component needs to distribute data to these components during its restart process, the downstream component will handle the distribution, without distinguishing between incremental target data and target data, thus not affecting the midstream component's restart process. When other components are superior to the upstream component, since data is sent from superior to subordinate, the midstream component's restart process does not affect components above the upstream component. In other words, this embodiment of the invention applies to the data recovery process of any midstream component within this hierarchical relationship.

[0108] In the embodiments of the present invention, the limiting descriptions such as "first" and "second" do not refer to a specific order. They are merely used to separate the corresponding limited objects from the same category and to facilitate the description of two or more different objects in the same category. They should not be interpreted as having a further limiting meaning.

[0109] Example 2:

[0110] In practical applications, multiple downstream components may exist in the system. Based on the method described in Embodiment 1, this invention illustrates the implementation process of the invention in this specific scenario. Figure 6 As shown, the system contains a first component, a second component, a third component, and a fourth component. The second component is a subordinate of the first component, and both the third and fourth components are subordinates of the second component.

[0111] When the second component restarts and goes online, such as Figure 7 As shown, the following implementation methods exist:

[0112] In step 401, when the second component goes online, it generates target data based on the historical business data received from the first component and stores the target data in the target cache.

[0113] In step 402, target data is retrieved sequentially from the target cache and sent to the first target pipeline and the second target pipeline. The third component and the fourth component read the target data from the first target pipeline and the second target pipeline in real time, respectively.

[0114] In step 403, if the number of target data in the first target pipeline or the second target pipeline exceeds a preset number, the rate at which target data is retrieved from the target cache is reduced to the minimum sending rate; if the number of target data in both the first target pipeline and the second target pipeline is less than the preset minimum number, the rate at which target data is retrieved from the target cache is restored to the normal sending rate.

[0115] In step 404, during the process of sending the target data to the third and fourth components, incremental target data is generated based on the incremental service data received from the upstream component, and the incremental target data is stored in the target cache.

[0116] In step 405, after the target data is sent to the third and fourth components, incremental target data is retrieved from the target cache in sequence and sent to the third and fourth components for data updates.

[0117] In the above implementation, when there are two downstream components, the processing speed of incremental service data is controlled according to the load of the two downstream components. When either of the two downstream components is under heavy load, the receiving rate of incremental service data is slowed down, thereby ensuring load balance among all downstream components during data recovery. When there are more than two downstream components, the implementation process can be extended from the above implementation method, and will not be elaborated here.

[0118] In the embodiments of the present invention, the limiting descriptions such as "first" and "second" do not refer to a specific order. They are merely used to separate the corresponding limited objects from the same category and to facilitate the description of two or more different objects in the same category. They should not be interpreted as having a further limiting meaning.

[0119] Example 3:

[0120] This invention is based on the method described in Embodiment 1, and combines it with specific application scenarios, using technical descriptions in related scenarios to illustrate the implementation process of the characteristics of this invention in those scenarios. For example... Figure 8The diagram illustrates a scenario of local functional components under a router in an L3VPN network. The unicast routing management component 501 manages all routing information and is an upstream component of the service dependency management component 502. The driver adaptation component 503 drives the management interface and other hardware and is a downstream component of the service dependency management component 502. The service dependency management component 502 generates the data required by the driver adaptation component 503 based on the historical service data issued by the service dependency management component and issues it to the driver adaptation component 503.

[0121] When the business dependency management component 502 goes offline, it notifies the unicast routing management component 501 and the driver adaptation component 503 through a three-way handshake. The unicast routing management component 501 records the storage location of the routing information it manages at this moment and uses the routing information it manages at this moment as historical business data. When it receives a command to change the routing information later, it uses the command to change the routing information as incremental business data.

[0122] When the business management dependency component 502 goes online, it notifies the unicast routing management component 501 through a three-way handshake. The unicast routing management component 501 then sequentially sends out historical business data. Upon receiving the historical business data, the business dependency management component 502 generates process data in real time based on the business objects of the historical business data. For example, the business object for generating the first piece of process data is the prefix address of route A, and the business object for generating the second piece of process data is the next hop out interface of route B. When subsequent pieces of process data generated from historical business data are received with the business object of the next hop out interface of route B, the second piece of process data is synchronized with the currently received historical business data to ensure that there is only one piece of process data for the next hop out interface of route B, and that the value of this process data is always the latest value.

[0123] Since the data received by the driver adaptation component 503 is also categorized according to business objects, the process data generated by the business dependency management component 502 can be directly used as target data. That is, the first piece of process data is the first piece of target data, and the second piece of process data is the second piece of target data. The driver adaptation component 503 requires that target data be distributed according to the type of different business objects, i.e., the prefix address is distributed first, followed by the next outgoing interface. Therefore, the first piece of target data is attached to the first linked list used to store process data of the prefix address type, and the second piece of target data is attached to the second linked list used to store process data of the next outgoing interface type. If other types of process data exist, they are attached to the linked list storing the corresponding type of process data, such as... Figure 9 As shown.

[0124] After the historical business data is distributed, the unicast routing management component 501 will push a message indicating that the distribution is complete. At this time, as shown in Figure 8, the business dependency management component 502 uses the target linked list as the target cache, traverses the linked lists of all types of process data, and according to the receiving requirements of the driver adaptation component 503, attaches the target data in the first linked list to the target linked list, and then attaches the target data in the second linked list to the target linked list. When all the target data has been attached, an end marker is added to the end of the last target data's linked list.

[0125] After the unicast routing management component 501 completes the distribution of historical service data, it pushes a completion message to the service management dependency component 502, and then continues to distribute incremental service data. Before the generation of target data is complete, the service management dependency component 502 keeps the incremental service data in the data pipeline without processing it. After the generation of target data is complete, the incremental service data is retrieved from the data pipeline, and the incremental target data generated based on the incremental service data is sequentially attached after the end marker in the target linked list.

[0126] After generating the target data, the business management dependency component 502 sequentially retrieves the target data from the target linked list and sends it to the driver adaptation component 503. Once all the target data in the target linked list has been sent, i.e., when the tail pointer of the target linked list points to the end marker, the incremental target data in the target linked list is sequentially retrieved and sent to the driver adaptation component 503. During the sending of target data and incremental target data, the usage of the target linked list and the data change trends in the target linked list are monitored to control the receiving rate of incremental business data.

[0127] Assuming the target linked list can cache a maximum of 8000 target data items, the preset first data quantity is set to 7000, the preset second data quantity is set to 0, the preset time is 10 seconds, and the preset maximum and minimum rates are both set to add 80 target data items per sampling period. A timer with a time interval of t is set up, i.e., the sampling period is t. Whenever the timer period expires, the usage of the target linked list and the data change trend in the target linked list are monitored to control the receiving rate of incremental service data. The specific steps are as follows: Figure 10 As shown, it includes:

[0128] In step 601, the total number of target data and incremental target data in the current target linked list is m; the number of incremental target data i newly added to the target linked list and the number of target data or incremental target data j taken out from the target linked list within a sampling time period t are counted, and the rate of increase of incremental target data within the sampling time period t is calculated as (ij) / t; where, when (ij) / t>0, it means that the number of newly added incremental target data is greater than the number of target data or incremental target data taken out from the target linked list, and the data volume of the target linked list shows an increasing trend; when (ij) / t<0 or (ij) / t=0, it means that the number of newly added incremental target data is less than or equal to the number of target data or incremental target data taken out from the target linked list, and the data volume of the target linked list does not increase or shows a decreasing trend.

[0129] In step 602, it is determined whether m is greater than or equal to 7000, where 7000 is the preset first data volume. If m is greater than the preset first data volume, it is determined that the data volume in the target linked list is too large, and the process proceeds to step 609; otherwise, the process proceeds to step 603, where the receiving rate of incremental service data is controlled according to the changing trend of the data volume in the target linked list.

[0130] In step 603, it is determined whether (ij) / t is greater than or equal to 80. If (ij) / t≥80, where 80 is the preset maximum rate and preset minimum rate, it indicates that the data volume growth rate in the target linked list is too fast, and proceed to step 605; otherwise, it indicates that the data volume growth rate in the target linked list is normal, or the data volume does not increase or decreases, and proceed to step 604.

[0131] In step 604, t1 is cleared, and the moment when (ij) / t>≥80 is first not satisfied is recorded as t0; where t1 is the moment when (ij) / t≥80 is first satisfied. Since (ij) / t≥80 is broken, the timing when (ij) / t≥80 is satisfied is cleared. Clearing here is not zeroing, but setting it to invalid, so that the subsequent t2-t1 is always greater than 10; after t0 has been recorded, that is, when t0 is valid, when entering this step in the subsequent timing cycle, the moment t1 is not recorded again; proceed to step 606.

[0132] In step 605, t0 is cleared, and the moment when (ij) / t≥80 is first satisfied is recorded as t1; where t0 is the moment when (ij) / t≥80 is not satisfied for the first time. Since (ij) / t≥80 is satisfied, the timing when (ij) / t≥80 is not satisfied is cleared. Here, clearing is not zeroing, but setting it to invalid, so that the t2-t0 calculated in the subsequent time period is always greater than 10. After t1 has been recorded, that is, when t1 is valid, when entering this step in the subsequent time period, the time t1 is not recorded again; proceed to step 607.

[0133] In step 606, the current time is recorded as t2, and it is determined whether t2-t0 is greater than or equal to 10, that is, whether the duration of (ij) / t>80 has reached the preset time. If it is greater than or equal to 10, then proceed to step 608.

[0134] In step 607, the current time is recorded as t2, and it is determined whether t2-t1 is greater than or equal to 10, that is, whether the duration of (ij) / t>80 has reached the preset time. If it is greater than or equal to 10, then proceed to step 609.

[0135] In step 608, the receiving rate of incremental service data is restored to the normal receiving rate. Specifically, this includes: if the current receiving rate of incremental service data is the normal receiving rate, then the normal receiving rate is maintained; if the current receiving rate of incremental service data is the minimum receiving rate, then the normal receiving rate is restored. The change of the receiving rate of incremental service data is achieved by changing the time interval of the timer that retrieves incremental service data from the pipeline of incremental service data, and waiting to enter the next sampling timing period.

[0136] In step 609, the receiving rate of incremental service data is reduced to the minimum receiving rate. Specifically, this includes: if the current receiving rate of incremental service data is the minimum receiving rate, then the minimum receiving rate is maintained; if the current receiving rate of incremental service data is the normal receiving rate, then it is reduced to the minimum receiving rate. The change in the receiving rate of incremental service data is achieved by changing the time interval of the timer that retrieves incremental service data from the pipeline that sends incremental service data, waiting to enter the next sampling timing period.

[0137] Since both incremental business data and target data are stored in the target linked list, during the target data distribution process, the target linked list is updated with incremental target data and retrieved with target data. By monitoring the usage and data volume trends of the target linked list, we can determine the load on the business management dependency component 502 and whether the processes of generating and distributing incremental target data are balanced. This balances the load on the business management dependency component 502 and the unicast routing management component 501, and maintains the balance between the generation and distribution of incremental target data. After the target data distribution is completed, the target linked list is updated with and retrieved with incremental target data. By monitoring the usage and data volume trends of the target linked list, we can determine the load on the business management dependency component 502 and balance the load on the business management dependency component 502 and the unicast routing management component 501.

[0138] This embodiment of the invention also includes controlling the distribution process of target data to balance the load between the business management dependency component 502 and the driver adaptation component 503. Combined with the above implementation, this balances the system load, specifically including:

[0139] The business management dependency component 502 sequentially retrieves the target data from the target chain list and sends the retrieved target data to the target pipeline, while the driver adaptation component 503 reads the target data from the target pipeline in real time.

[0140] The business management dependency component 502 monitors the number of target data in the target pipeline. The preset number is set to 30, and the preset minimum number is 20. If the number of target data in the target pipeline exceeds 30, the rate at which target data is retrieved from the target list will be reduced to the minimum sending rate. If the number of target data in the target pipeline is less than 20, the rate at which target data is retrieved from the target list will be restored to the normal sending rate.

[0141] To more accurately balance the system load, the receiving rate of incremental service data is also controlled by introducing the amount of incremental service data during the target data delivery process. Specifically, this includes:

[0142] If, within a preset time, the number of incremental target data in the target linked list exceeds the preset maximum amount of incremental target data, the rate of receiving incremental service data will be reduced to the minimum receiving rate. If the number of incremental target data in the target linked list is less than the preset minimum amount of incremental target data, the rate of receiving incremental service data will be restored to the normal receiving rate.

[0143] The preset maximum and minimum incremental business data amounts are derived from experience analysis by those skilled in the art.

[0144] Before data recovery is complete, i.e., when the target linked list still contains an end marker, the main tasks of business management dependency component 502 and driver adaptation component 503 are to generate their own midstream business data and achieve data synchronization with upstream components, i.e., to process historical business data. After data recovery is complete, i.e., after the target data is distributed, and only incremental target data exists in the target linked list, the main tasks of business management dependency component 502 and driver adaptation component 503 are to maintain the normal business operation of the system, i.e., to process incremental business data. The speed at which business management dependency component 502 and driver adaptation component 503 process historical business data and incremental business data may be different, resulting in different loads on the components. Therefore, preset first data volume, preset second data volume, preset time, preset maximum rate, and preset minimum rate can be set for the data recovery stage and the normal business operation stage, respectively. In this way, the rate of receiving incremental business data can be controlled according to the usage of the target linked list at different stages, thereby achieving overall load balancing during data recovery and normal system operation.

[0145] In the embodiments of the present invention, the limiting descriptions such as "first" and "second" do not refer to a specific order. They are merely used to separate the corresponding limited objects from the same category and to facilitate the description of two or more different objects in the same category. They should not be interpreted as having a further limiting meaning.

[0146] Example 4:

[0147] This invention provides a data recovery device.

[0148] like Figure 11 The diagram shown is a schematic representation of the architecture of a data recovery device according to an embodiment of the present invention. The data recovery device of this embodiment includes one or more processors 21 and a memory 22. Figure 11 Take a processor 21 as an example.

[0149] Processor 21 and memory 22 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0150] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the data recovery method in Embodiment 1. The processor 21 executes the data recovery method by running the non-volatile software program and instructions stored in the memory 22.

[0151] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the data recovery methods described in Embodiments 1 and 2 above, for example, the methods described above. Figure 2 , Figure 4 , Figure 5 , Figure 7 and Figure 10 The steps shown.

[0153] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0154] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data recovery method, characterized in that, The method includes: When going live, target data is generated based on historical business data received from upstream components, the target data is stored in the target cache, the target data is retrieved from the target cache in sequence, and the target data is sent to downstream components for data synchronization of downstream components; During the process of sending target data to downstream components, incremental target data is generated based on the incremental business data received from upstream components, and the incremental target data is stored in the target cache. After the target data is sent to the downstream components, incremental target data is retrieved from the target cache and sent to the downstream components for data updates; the upstream component is a unicast routing management component. If the amount of data in the target cache is greater than or equal to a preset first data amount, the receiving rate of incremental service data is reduced to the minimum receiving rate; if the amount of data in the target cache is less than or equal to a preset second data amount, the receiving rate of incremental service data is restored to the normal receiving rate; if the amount of data in the target cache is less than the preset first data amount but greater than the preset second data amount, the receiving rate of incremental service data is controlled according to the changing trend of the amount of data in the target cache. The step of sequentially retrieving target data from the target cache and sending the target data to downstream components specifically includes: sequentially retrieving target data from the target cache and sending the retrieved target data to the target pipeline; the downstream components reading target data from the target pipeline in real time; if the number of target data in the target pipeline exceeds a preset number, the rate at which target data is retrieved from the target cache is reduced to the minimum sending rate; if the number of target data in the target pipeline is less than the preset minimum number, the rate at which target data is retrieved from the target cache is restored to the normal sending rate.

2. The data recovery method according to claim 1, characterized in that, The step of generating target data based on historical business data received from upstream components and storing the target data in the target cache specifically includes: Receive historical business data and generate one or more process data based on one or more business objects in the historical business data; If it is found that the business object in the newly received historical business data is the same as the business object in the previously received historical business data, the process data of the business object will not be generated based on the newly received historical business data. Instead, the process data of the previously generated business object will be updated based on the newly received historical business data. After receiving all historical business data, target data is generated based on the process data, and the target data is stored in the target cache.

3. The data recovery method according to claim 2, characterized in that, The step of generating target data based on process data and storing the target data in the target cache specifically includes: Target data is generated according to the process data in the data format required by the downstream components, and the target data is stored in the target cache according to the receiving order required by the downstream components.

4. The data recovery method according to claim 3, characterized in that, The step of storing the target data in the target cache according to the receiving order of the target data required by the downstream component specifically includes: If the downstream component requires the target data to be received in the order of the target data types, then different linked lists are defined according to the different target data types, and target data of the same type are stored in the same linked list. According to the receiving order of the target data type required by the downstream component, the nodes in each linked list are attached to the target cache in the required order.

5. The data recovery method according to claim 1, characterized in that, The step of controlling the receiving rate of incremental service data based on the changing trend of the target cache data volume specifically includes: If the data in the target cache continues to increase within a preset time period, and the rate of data increase always exceeds the preset maximum rate, then the receiving rate of incremental service data will be reduced to the minimum receiving rate; if the data in the target cache continues to increase, and the rate of data increase never exceeds the preset minimum rate, or the data in the target cache decreases or does not increase, then the receiving rate of incremental service data will be restored to the normal receiving rate.

6. The data recovery method according to any one of claims 1-5, characterized in that, The step of sequentially retrieving incremental target data from the target cache and sending the incremental target data to the downstream target cache component specifically includes: Incremental target data is retrieved sequentially from the target cache and sent to downstream components. The receiving rate of incremental business data is controlled based on the usage of the target cache.

7. A data recovery device, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the data recovery method according to any one of claims 1-6.

Images