A data recovery method, device, and system
By recording the correspondence between log data and status data in stream computing, the problem of high transmission and computational overhead during data recovery in stream computing is solved, and an efficient data recovery process is achieved.
Patent Information
- Application Number
- CN202510066448.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-15
- Publication Date
- 2026-07-21
AI Technical Summary
In stream computing, the data recovery process incurs excessive transmission and computation overhead due to snapshot operations, and the client needs to repeatedly send business data.
Record the client's business data write operations and stream computing log data, recover the stream computing results through the log data, and perform data recovery by combining persistent state data, avoiding re-performing stream computing and data transmission.
It reduces the transmission and computation overhead during the data recovery process, achieves an accurate and complete stream computing process, and eliminates the need for the client to repeatedly send business data.
Smart Images
Figure CN122431954A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of communication technology, and in particular to a data recovery method, apparatus and system. Background Technology
[0002] Stream computing, also known as streaming computation, is a continuous, low-latency, event-triggered computational job. Stream computing can be implemented using a node system, which may include multiple business nodes responsible for executing operators and a control node on the control side. Algorithms can be deployed and executed on multiple business nodes, with each business node responsible for executing a portion of the operators. After execution, each business node can pass the results to adjacent downstream nodes. Stream computing typically includes data input (data source), processing operators (one or more), and data output. The data source and data output can be one or multiple.
[0003] A checkpoint, also known as a snapshot, is a global snapshot operation triggered from the source node of a stream computation to the completion of a downstream node. When a downstream operator performs a checkpoint snapshot, it can persistently back up its state data. This state data consists of the computation results obtained by the node after executing the corresponding operation of the operator. These computation results are still needed in subsequent computations. Typically, the state data obtained by the node is stored in memory.
[0004] When a node performs a snapshot operation at a checkpoint, it can persist the state data. When data recovery is needed, it can read the persisted state data and then perform operator operations based on the data repeatedly sent by the upstream node (this data may be lost due to node restart or failure; for the source node, it is the business data repeatedly sent by the client) and the read persistent state data, without having to calculate the state data from scratch.
[0005] However, the problem caused by the above is that although the persistently stored state data can be read directly, when performing data recovery, it is necessary to re-acquire the client's business data and re-perform stream computing on the business data through the system (including computing and data transmission between nodes). Then, the data recovery is performed by combining the stream computing results and the state data. This will result in additional stream computing overhead and require the client to support the repeated sending of business data. Summary of the Invention
[0006] This application provides a data recovery method, apparatus, and system that can reduce the transmission and computational overhead during the data recovery process caused by snapshot operations.
[0007] In a first aspect, this application provides a data recovery method, the method comprising: acquiring first business data from a client; processing the first business data through stream computing to obtain first state data; persistently storing the first state data when taking a data snapshot; recording write operations on the first business data and log data of stream computing performed after obtaining the first state data; and performing data recovery based on the persistently stored first state data and the log data.
[0008] In existing technologies, snapshot operations persistently store state data. During data recovery, it's necessary to re-acquire the client's business data and re-perform stream computing (including computation and data transmission between nodes) on the business data. Then, data recovery is performed by combining the stream computing results and state data. This results in additional stream computing overhead and requires the client to support retransmission of business data. In this embodiment, the client's business data write operations and stream computing log data can be recorded. The stream computing results can then be recovered based on the log data, and data recovery is performed by combining the stream computing results and state data without requiring the system to re-perform stream computing (including computation and data transmission between nodes). This reduces the transmission and computation overhead during data recovery caused by snapshot operations and ensures an accurate and complete stream computing process even when the client does not support retransmission of business data.
[0009] In one possible implementation, the method further includes: recording the correspondence between the first state data and the log data; before performing data recovery, obtaining the first state data that is persistently stored, and obtaining the log data corresponding to the first state data according to the correspondence.
[0010] The mapping relationship can be achieved by adding indicator information to the log data to identify the corresponding status data, or by recording it in some implicit way.
[0011] In one possible implementation, the log data further includes information indicating that the data type is stream computing.
[0012] In one possible implementation, the persistent storage of the first state data includes: persistently storing the first state data located in a first memory space; the method further includes: obtaining second service data from the client; processing the second service data through stream computing to obtain second state data; and during the persistent storage of the first state data located in the first memory space, writing the second state data into a second memory space, wherein the second memory space is different from the first memory space.
[0013] When performing a snapshot operation, the state data in memory needs to be stored on the hard disk (or other non-volatile storage media). This memory will not be released (data cannot be written) until the persistent storage is completed. In this case, if only one block of memory is allocated for storing state data, and it is not released, if the subsequent state data has been calculated, there will be no memory space for storing the calculated state data. This will block the subsequent incoming data packets and operator calculations. In other words, the data snapshot operation will block the stream computing process.
[0014] In this embodiment, at least two memory spaces (e.g., the first memory space and the second memory space in this embodiment) can be allocated for storing state data. If one of the memory spaces is not released due to a snapshot operation, the new state data can be stored in the other memory space, thereby ensuring that the process of stream computing will not be blocked due to the data snapshot operation.
[0015] In one possible implementation, during the process of persistently storing the first state data located in the first memory space, the first memory space has no write permission, and the second memory space has write permission.
[0016] Secondly, this application provides a data recovery apparatus, the apparatus comprising:
[0017] The acquisition module is used to acquire the client's initial business data.
[0018] The stream computing module is used to process the first business data through stream computing to obtain the first state data;
[0019] The snapshot module is used to persistently store the first state data when taking a data snapshot; record the write operations of the first business data and the log data of the stream processing performed after obtaining the first state data;
[0020] The data recovery module is used to perform data recovery based on the first state data and the log data stored in persistent storage.
[0021] In one possible implementation, the snapshot module is further configured to:
[0022] Record the correspondence between the first state data and the log data;
[0023] The data recovery module is further configured to obtain the first state data that is persistently stored before performing data recovery, and to obtain the log data corresponding to the first state data according to the correspondence relationship.
[0024] In one possible implementation, the log data further includes information indicating that the data type is stream computing.
[0025] In one possible implementation, the snapshot module is specifically used to: persistently store the first state data located in the first memory space;
[0026] The acquisition module is also used for:
[0027] Obtain the client's second business data;
[0028] The stream computing module is also used to process the second business data through stream computing to obtain the second state data;
[0029] The snapshot module is further configured to write the second state data into a second memory space, which is different from the first memory space, during the process of persistently storing the first state data located in the first memory space.
[0030] In one possible implementation, during the process of persistently storing the first state data located in the first memory space, the first memory space has no write permission, and the second memory space has write permission.
[0031] Thirdly, this application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory to cause the computing device or the computing device cluster to perform a method as described in the first aspect or any implementation thereof.
[0032] Fourthly, this application provides a computer-readable storage medium storing instructions that instruct a computing device or a cluster of computing devices to execute the method of the database system described in the first aspect or any implementation thereof.
[0033] Fifthly, this application provides a computer program product containing instructions that, when run on a computing device or a cluster of computing devices, causes the computing device or cluster of computing devices to execute the database system execution method described in the first aspect or any implementation thereof.
[0034] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description
[0035] Figure 1 This application provides an architectural illustration;
[0036] Figure 2 A flowchart illustrating a data recovery method provided in this application;
[0037] Figure 3 and Figure 4 This application provides an operational flowchart.
[0038] Figures 5 to 8 A schematic diagram of the hardware architecture provided for this application. Detailed Implementation
[0039] First, some expressions that may appear in this application will be explained.
[0040] "First" and "second" are used to distinguish different objects or to differentiate different treatments of the same object, rather than to describe a specific order of objects.
[0041] "At least one" means one or more, while "more" means two or more.
[0042] "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone. A and B can be singular or plural.
[0043] The character " / " generally indicates that the objects before and after it are in an "or" relationship. For example, A / B can mean A or B.
[0044] Furthermore, the terms "comprising," "including," and "having" used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.
[0045] It should be noted that in this application, the terms "exemplary" or "for example" are used to indicate that something is being described or illustrated. Any implementation or design scheme described as "exemplary" or "for example" (such as the embodiments in this application) should not be construed as being more preferred or advantageous than other implementations or design schemes. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.
[0046] In the specification and drawings of this application, the terms "of", "relevant", and "corresponding" may sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, they have the same meaning.
[0047] Below is a brief description of some of the terms used in this application.
[0048] Terminals can include desktop, laptop, handheld, and vehicle-mounted user equipment (UE) devices, such as smartphones, cellular phones, desktop computers, tablets, smart TVs, smart TV boxes, ultra-mobile personal computers (UMPCs), laptops, personal digital assistants (PDAs), portable multimedia players (PMPs), dedicated media players, consumer communication devices, wearable devices (such as smartwatches), AR (augmented reality) / VR (virtual reality) devices, and other types of communication devices.
[0049] Identifier: Used to distinguish one thing from others of the same or different categories. Examples include node identifiers, network identifiers, and network interface card (NIC) identifiers. Identifiers can be names, numbers, or identifiers based on a distinctive characteristic, such as category identifiers. This application does not limit the implementation of various identifiers, as long as they can achieve differentiation.
[0050] Storage space, also called address space, refers to one or more segments of addresses that can be used by a device or instance. For example, the virtual address space of a device or instance is the segment or more virtual addresses that can be used by that device or instance. The virtual address space of a device or instance is allocated by the operating system running that device or instance. Similarly, the physical address space of a device or instance is the segment or more physical addresses allocated to that device or instance. When a device or instance uses this physical address space, other devices or instances cannot use addresses within that physical address space. An instance's physical address space is allocated by the operating system running that instance. This allocation may be dynamic; for example, as the instance runs, the physical address space it occupies may increase, but there is an upper limit. The size and range of a device's physical address space are usually fixed.
[0051] Stream computing, also known as streaming computation, is a continuous, low-latency, event-triggered computational job. Stream computing can be implemented using a node system, which may include multiple business nodes responsible for executing operators and a control node on the control side. Algorithms can be deployed and executed on multiple business nodes, with each business node responsible for executing a portion of the operators. After execution, each business node can pass the results to adjacent downstream nodes. Stream computing typically includes data input (data source), processing operators (one or more), and data output. The data source and data output can be one or multiple.
[0052] State data: Historical data obtained by nodes in stream computing when executing operations corresponding to operators. In some scenarios, state data is stored in pre-configured storage space and used when subsequent operators are executed.
[0053] Stateless computation: In stream computing, nodes only need to process the current input data to compute the correct result when executing operators.
[0054] Stateful computation: In contrast to stateless computation, in stream computation, nodes need to perform related operations with some historical state data when executing operators in order to calculate the correct result (such as accumulation).
[0055] A checkpoint, also known as a checkpoint, is a global operation triggered from the source node of stream computing to the completion of all downstream operators. Downstream operators periodically persist their state in real-time tasks. The checkpoint mechanism is based on the Chandy-Lamport algorithm. The Chandy-Lamport algorithm simplifies a distributed system into a directed graph consisting of a finite number of processes and channels between them, where nodes are processes and edges are channels. The global state of the distributed system is composed of information from these processes and channels, which is what the Chandy-Lamport algorithm needs to save.
[0056] A key characteristic of streaming processing is its continuous processing of data. Unless manually stopped by the user, it typically runs indefinitely. During operation, it may encounter machine failures, network problems, external storage issues, and more. For real-time tasks to run stably, they need automatic fault tolerance and recovery capabilities. Checkpointing is one such fault tolerance and recovery mechanism. It ensures that a real-time program can self-recover even when encountering sudden exceptions or machine problems. Checkpointing reduces fault tolerance and recovery time because it restores the state from the latest checkpoint, rather than from the program's initial state.
[0057] Barrier: Also known as a barrier or data fence (which can be understood as a boundary line for persistent storage of the state of a detection point), it is a special internal message generated by the stream computing framework. It flows as part of the data stream to downstream operators without affecting the normal data stream.
[0058] In distributed checkpointing, barriers are a crucial concept, often referred to as data barriers or data fences. A barrier is a special type of internal message that is periodically injected at the source of the data stream during checkpointing. These barriers flow as part of the data stream to downstream operators without affecting the normal data flow.
[0059] Stream computing, also known as streaming computation, is a continuous, low-latency, event-triggered computational job. Stream computing can be implemented using a node system, which may include multiple business nodes responsible for executing operators and a control node on the control side. Algorithms can be deployed and executed on multiple business nodes, with each business node responsible for executing a portion of the operators. After execution, each business node can pass the results to adjacent downstream nodes. Stream computing typically includes data input (data source), processing operators (one or more), and data output. The data source and data output can be one or multiple.
[0060] A checkpoint, also known as a snapshot, is a global snapshot operation triggered from the source node of a stream computation to the completion of a downstream node. When a downstream operator performs a checkpoint snapshot, it can persistently back up its state data. This state data consists of the computation results obtained by the node after executing the corresponding operation of the operator. These computation results are still needed in subsequent computations. Typically, the state data obtained by the node is stored in memory.
[0061] When a node performs a snapshot operation at a checkpoint, it can persist the state data. When data recovery is needed, it can read the persisted state data and then perform operator operations based on the data repeatedly sent by the upstream node (this data may be lost due to node restart or failure; for the source node, it is the business data repeatedly sent by the client) and the read persistent state data, without having to calculate the state data from scratch.
[0062] However, the aforementioned problem is that although persistently stored state data can be read directly during data recovery, the client's business data needs to be retrieved again and re-processed through the system (including computation and data transmission between nodes). Then, the data recovery is performed by combining the stream computation results and state data. This results in additional stream computation overhead and requires the client to support the repeated sending of business data.
[0063] To address the aforementioned issues, this application provides a data recovery method.
[0064] Reference Figure 1 , Figure 1 This is a schematic diagram of an application architecture for this application.
[0065] Figure 1 An example of system 100 in an embodiment of this application is shown. For example... Figure 1 As shown, system 100 may include at least one control node 110 and multiple service nodes 120 (in this embodiment, service nodes can be simply referred to as nodes). The control node 110 is communicatively connected to each service node 120. The control node 110 can be used to coordinate and manage the operation of each service node 120, such as allocating resources like operator nodes to tasks, managing checkpoints, and controlling the data transmission paths between tasks on each service node. When system 100 includes one control node 110, the control node 110 performs the above operations within system 110; when system 100 includes multiple control nodes 110, one of the multiple control nodes 110 serves as the master control node, used to perform the above operations within system 110, while the other control nodes 110 serve as backup control nodes. Service nodes 120 can be worker nodes used to execute tasks (e.g., operator execution tasks). Service nodes 120 can perform data transmission and execution flow computation tasks with each other.
[0066] Reference Figure 2 , Figure 2 A flowchart illustrating a data recovery method provided in this application embodiment includes:
[0067] 201. Obtain the first business data from the client.
[0068] 202. Process the first service data using stream computing to obtain the first state data;
[0069] The embodiments of this application can be applied to stream computing systems, such as database systems.
[0070] For example, a stream computing system may include multiple nodes, such as a first node and a second node. The first node and the second node are communicatively connected; the first node executes a first operator in the algorithm, and the second node executes a second operator in the algorithm.
[0071] For example, the operator can be an accumulation.
[0072] The second node can obtain the first data by processing the data through the second operator. When the second node is the source node, the data processed by the second operator can be the business data sent by the client. When the second node is not the source node, the data processed by the second operator can be the data sent by the upstream node of the second node.
[0073] The first data packet may include first data and first indication information. The first indication information is used to instruct a snapshot operation to be performed. For example, the first indication information may be a barrier, which can be called a data fence or a data barrier. A barrier is a special internal message that is periodically injected at the source of the data flow (source node) during checkpointing. These barriers flow as part of the data flow to downstream operators without affecting the normal data flow. For downstream nodes, if the received data packet contains a barrier, a snapshot operation needs to be triggered.
[0074] In this embodiment of the application, if the first node receives a data packet (i.e., the first data packet) sent by the second node that carries first indication information (e.g., a barrier), the first node may trigger a snapshot operation.
[0075] 203. When taking a data snapshot, the first state data is persistently stored;
[0076] 204. Record the write operation of the first business data and the log data of the stream computation performed after obtaining the first status data.
[0077] The log data can include write operations to the first business data, and the first business data can be recovered based on this log data.
[0078] The log data can include the stream computation performed after the first state data. Based on this log data, the computation process of the stream computation can be reproduced, and thus the computation result of the stream computation can be reproduced.
[0079] In one possible implementation, the method further includes: recording the correspondence between the first state data and the log data; before performing data recovery, obtaining the first state data that is persistently stored, and obtaining the log data corresponding to the first state data according to the correspondence.
[0080] The mapping relationship can be achieved by adding indicator information to the log data to identify the corresponding status data, or by recording it in some implicit way.
[0081] In one possible implementation, the log data further includes information indicating that the data type is stream computing.
[0082] In existing technologies, snapshot operations persistently store state data. During data recovery, it's necessary to re-acquire the client's business data and re-perform stream computing (including computation and data transmission between nodes) on the business data. Then, data recovery is performed by combining the stream computing results and state data. This results in additional stream computing overhead and requires the client to support retransmission of business data. In this embodiment, the client's business data write operations and stream computing log data can be recorded. The stream computing results can then be recovered based on the log data, and data recovery is performed by combining the stream computing results and state data without requiring the system to re-perform stream computing (including computation and data transmission between nodes). This reduces the transmission and computation overhead during data recovery caused by snapshot operations and ensures an accurate and complete stream computing process even when the client does not support retransmission of business data.
[0083] In one possible implementation, the log data can be write-ahead logging (WAL).
[0084] Reference Figure 3 For example, you can refer to Figure 3 When performing persistent storage of the initial data, the database's own periodic flush mechanism can be utilized to periodically issue barrier events. The write module will simultaneously receive these barrier events. Upon receiving a barrier event, the write module will execute the mem table disk partitioning. After completion, the WAL file can be moved to the snapshots space. Upon receiving a barrier event, the stream module can execute the state disk partitioning. After completion, the WAL file in the snapshots space will be cleaned up.
[0085] When performing a snapshot operation, the state data in memory needs to be stored on the hard disk (or other non-volatile storage media). This memory will not be released (data cannot be written) until the persistent storage is completed. In this case, if only one block of memory is allocated for storing state data, and it is not released, if the subsequent state data has been calculated, there will be no memory space for storing the calculated state data. This will block the subsequent incoming data packets and operator calculations. In other words, the data snapshot operation will block the stream computing process.
[0086] In this embodiment, at least two memory spaces (e.g., the first memory space and the second memory space in this embodiment) can be allocated for storing state data. If one of the memory spaces is not released due to a snapshot operation, the new state data can be stored in the other memory space, thereby ensuring that the process of stream computing will not be blocked due to the data snapshot operation.
[0087] In one possible implementation, the first state data located in the first memory space can be persistently stored; the method further includes: obtaining second service data from the client; processing the second service data through stream computing to obtain second state data; during the process of persistently storing the first state data located in the first memory space, the second state data is written to the second memory space, which is different from the first memory space.
[0088] Specifically, during the process of persistently storing the first state data located in the first memory space, the write permission of the first memory space is non-writable, while the write permission of the second memory space is writable.
[0089] Before persistently storing the first state data located in the first memory space, the write permission of the first memory space can be changed from writable to non-writable, or from non-writable to writable, according to the first indication information.
[0090] For example, you can refer to Figure 4 The cache is divided into two parts: state and cur_cache. When a checkpoint is required, cur_cache switches to state, and the previously used state space switches to cur_cache space. The state recovery logic can be as follows:
[0091] old st+stream wal+common wal=st2, st1+common wal=st2.
[0092] It should be understood that "common wal" here refers to the wal file that has not been moved to the snapshots space, while "stream wal" refers to the wal file that has been moved to the snapshots space.
[0093] 205. Perform data recovery based on the persistently stored first state data and the log data;
[0094] The log data can include write operations to the first business data, and the first business data can be recovered based on this log data.
[0095] The log data can include the stream computation performed after the first state data. Based on this log data, the computation process of the stream computation can be reproduced, and thus the computation result of the stream computation can be reproduced.
[0096] The implementation of the embodiments of this application will be described next from the perspective of the device, referring to... Figure 5 This application provides a data recovery device 500, which includes:
[0097] Module 501 is used to acquire the first business data from the client;
[0098] The stream computing module 502 is used to process the first service data through stream computing to obtain the first state data;
[0099] The snapshot module 503 is used to persistently store the first state data when taking a data snapshot; record the write operations of the first business data and the log data of the stream computation performed after obtaining the first state data;
[0100] The data recovery module 504 is used to perform data recovery based on the first state data and the log data stored in persistent storage.
[0101] In one possible implementation, the snapshot module 503 is further configured to:
[0102] Record the correspondence between the first state data and the log data;
[0103] The data recovery module 504 is further configured to obtain the first state data that is persistently stored before performing data recovery, and to obtain the log data corresponding to the first state data according to the correspondence relationship.
[0104] In one possible implementation, the log data further includes information indicating that the data type is stream computing.
[0105] In one possible implementation, the snapshot module 503 is specifically used to: persistently store the first state data located in the first memory space;
[0106] The acquisition module 501 is further configured to:
[0107] Obtain the client's second business data;
[0108] The stream computing module 502 is further configured to process the second business data through stream computing to obtain the second state data;
[0109] The snapshot module 503 is further configured to write the second state data into a second memory space, which is different from the first memory space, during the process of persistently storing the first state data located in the first memory space.
[0110] In one possible implementation, during the process of persistently storing the first state data located in the first memory space, the first memory space has no write permission, and the second memory space has write permission.
[0111] This application also provides a computing device 1000. For example... Figure 6 As shown, the computing device 1000 includes a bus 1003, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1003. The computing device 1000 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.
[0112] Bus 1003 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 1003 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1003 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).
[0113] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0114] The memory 1006 may include volatile memory, such as random access memory (RAM). The memory 1006 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). The memory 1006 stores executable program code, which the processor 1004 executes to implement the methods performed by the aforementioned database system (e.g., including the acquisition module 501, stream computing module 502, snapshot module 503, and data recovery module 504). Specifically, the memory 1006 stores instructions for the methods performed by the database system (e.g., including the acquisition module 501, stream computing module 502, snapshot module 503, and data recovery module 504).
[0115] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.
[0116] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0117] like Figure 7 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store instructions for executing methods of a database system (e.g., including an acquisition module 501, a stream computing module 502, a snapshot module 503, and a data recovery module 504).
[0118] In some possible implementations, one or more computing devices 1000 in the computing device cluster can also be used to execute partial instructions of methods executed by the database system (e.g., including the acquisition module 501, stream computing module 502, snapshot module 503, and data recovery module 504). In other words, a combination of one or more computing devices 1000 can jointly execute instructions of methods executed by the database system (e.g., including the acquisition module 501, stream computing module 502, snapshot module 503, and data recovery module 504).
[0119] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions for executing some functions of the heterogeneous database system 100.
[0120] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 8 One possible implementation is shown. For example... Figure 8 As shown, the two computing devices 120A and 120B are connected via a network. Specifically, they are connected to the network through the communication interface in each computing device. In this possible implementation, the memory 116 in computing device 120A may store instructions for executing the data recovery method described in this embodiment. Simultaneously, the memory 116 in computing device 120B may store instructions for implementing the data recovery method. Alternatively, the memory 116 in computing device 120A may store instructions for implementing part of the data recovery method's functions. Simultaneously, the memory 116 in computing device 120B may store instructions for implementing another part of the data recovery method's functions.
[0121] It should be understood that Figure 8 The functions of computing device 120A shown can also be performed by multiple computing devices. Similarly, the functions of computing device 120B can also be performed by multiple computing devices.
[0122] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 7 and Figure 8 The connection method of the computing device cluster. The difference is that the memory 116 of one or more computing devices in the computing device cluster can store the same instructions for executing the template generation method.
[0123] In some possible implementations, the memory 116 of one or more computing devices in the computing device cluster may also store partial instructions for executing the template generation method. In other words, a combination of one or more computing devices can jointly execute the instructions for executing the template generation method.
[0124] It should be noted that the memory 116 in different computing devices within the computing device cluster can store different instructions for executing parts of the template generation method. That is, the instructions stored in the memory 116 of different computing devices can implement the functions of one or more modules in the second node and the first node.
[0125] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the methods described above for database system execution, training sample generation, and model training methods applied to the optimization problem solving system 100.
[0126] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute the methods of the database system described above, the training sample generation method, and the model training method.
[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data recovery method, characterized in that, The method includes: Obtain the client's first business data; The first business data is processed by stream computing to obtain the first state data; When taking a data snapshot, the first state data is persistently stored; Record the write operations of the first business data and the log data of the stream computation performed after obtaining the first status data; Data recovery is performed based on the first state data stored in persistent storage and the log data.
2. The method according to claim 1, characterized in that, The method further includes: Record the correspondence between the first state data and the log data; Before performing data recovery, the first state data in persistent storage is obtained, and the log data corresponding to the first state data is obtained according to the correspondence.
3. The method according to claim 1 or 2, characterized in that, The log data also includes information indicating that the data type is stream computing.
4. The method according to any one of claims 1 to 3, characterized in that, The step of persistently storing the first state data includes: persistently storing the first state data located in the first memory space; The method further includes: Obtain the client's second business data; The second business data is processed by stream computing to obtain the second state data; During the process of persistently storing the first state data located in the first memory space, the second state data is written to the second memory space, which is different from the first memory space.
5. The method according to claim 4, characterized in that, During the process of persistently storing the first state data located in the first memory space, the write permission of the first memory space is non-writable, and the write permission of the second memory space is writable.
6. A data recovery device, characterized in that, The device includes: The acquisition module is used to acquire the client's initial business data. The stream computing module is used to process the first business data through stream computing to obtain the first state data; The snapshot module is used to persistently store the first state data when taking a data snapshot; record the write operations of the first business data and the log data of the stream processing performed after obtaining the first state data; The data recovery module is used to perform data recovery based on the first state data and the log data stored in persistent storage.
7. The apparatus according to claim 6, characterized in that, The snapshot module is also used for: Record the correspondence between the first state data and the log data; The data recovery module is further configured to obtain the first state data that is persistently stored before performing data recovery, and to obtain the log data corresponding to the first state data according to the correspondence relationship.
8. The apparatus according to claim 6 or 7, characterized in that, The log data also includes information indicating that the data type is stream computing.
9. The apparatus according to any one of claims 6 to 8, characterized in that, The snapshot module is specifically used to: persistently store the first state data located in the first memory space; The acquisition module is also used for: Obtain the client's second business data; The stream computing module is also used to process the second business data through stream computing to obtain the second state data; The snapshot module is further configured to write the second state data into a second memory space, which is different from the first memory space, during the process of persistently storing the first state data located in the first memory space.
10. The apparatus according to claim 9, characterized in that, During the process of persistently storing the first state data located in the first memory space, the write permission of the first memory space is non-writable, and the write permission of the second memory space is writable.
11. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 5.
12. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 5.
13. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 5.