Data backup and recovery method, device, computer equipment and storage medium

Abstract

The present application belongs to the field of artificial intelligence and financial technology, and relates to a data backup and recovery method, including monitoring through the monitoring mechanism of a key-value database, encapsulating the backup request as a backup request event and writing the backup key-value pair into the key-value database; upon monitoring the backup request event, calling the data write instruction to write the data file to a preset storage location, while pausing the message consumption of the message middleware; after the data file is written, constructing an index file based on the metadata of the data file, and when the index file is constructed, executing a snapshot backup operation for backup; after the backup is completed, resuming normal data processing. The present application also provides a data backup and recovery device, a computer device and a storage medium. In addition, the present application also relates to blockchain technology, and data files can be stored in the blockchain. The present application can greatly shorten the time for data backup and recovery and improve the timeliness of data backup and recovery.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a data backup and recovery method, apparatus, computer equipment, and storage medium. Background Art

[0002] In the financial sector, data integrity, security, and recoverability are crucial. With the rapid development of financial services, data volumes are exploding, and traditional relational databases are struggling to handle large-scale, high-dimensional vector data. Milvus, a vector database designed specifically for vector data storage and retrieval, has been widely adopted in the financial sector. It efficiently stores and retrieves vector data such as financial transaction records and user behavior characteristics, providing strong support for applications such as financial risk control and robo-advisory.

[0003] To meet the high data backup and recovery requirements of the financial sector, the Milvus database supports backup and recovery capabilities. Through its client tool, milvus-backup, users can easily execute the create command to back up data and the restore command to restore data. More importantly, Milvus also supports backup and recovery at the library and collection level, providing financial users with more flexible data management options.

[0004] However, while Milvus's backup and recovery feature excels at backing up and restoring data files, it suffers from a significant flaw: it can only back up data files, not index files. This means that when restoring data, the new collection must be re-indexed based on the backed-up index metadata. The index node only begins the index building process after receiving the command to create the index. While this design provides a certain degree of flexibility—for example, the prefix and suffix of the collection can be specified during recovery—implementing index backup and recovery is relatively complex. In particular, after creating a new collection and writing data in batches based on the files, the old index files often cannot be directly matched with the new collection due to changes in the data organization structure.

[0005] In the financial sector, this index rebuilding process is not only time-consuming but also uncertain in terms of timeliness. During the operation of financial services, if urgent cluster switches or rapid recovery are required, existing backup and recovery solutions often fail to meet customer timeliness requirements. This timeliness issue is particularly prominent when processing large-scale data sets. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to propose a data backup and recovery method, device, computer equipment and storage medium to solve the technical problem that the existing data backup and recovery method cannot back up index files and needs to recreate the index, resulting in a long backup and recovery time and uncertain timeliness.

[0007] In order to solve the above technical problems, the present invention provides a data backup and recovery method, which adopts the following technical solutions:

[0008] Monitoring the change events of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database;

[0009] receiving a backup request, encapsulating the backup request into a backup request event, and writing the backup request event into the backup key-value pair, thereby triggering the change event;

[0010] Obtain backup data from the message middleware through the data node to form a data file, and call the data write instruction to write the data file to the preset storage location, while pausing the consumption of messages from the message middleware;

[0011] After the data file is written, the metadata of the data file is reported through the data node, and the metadata is written into a specified set of the key-value database;

[0012] Generate an index request according to the specified set, and construct an index file for the specified set based on the index request through an index node;

[0013] Monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built;

[0014] Backing up the data file, the index file, and the key-value pairs of the specified set in sequence based on the snapshot backup operation to obtain a backup file;

[0015] After the snapshot backup is completed, a backup completion event is written to the key-value database, and the data node is notified to resume consuming messages from the message middleware.

[0016] In order to solve the above technical problems, the present application also provides a data backup and recovery device, which adopts the following technical solution:

[0017] An event monitoring module, configured to monitor, through a monitoring mechanism of the key-value database, change events of backup key-value pairs in the key-value database;

[0018] An event writing module, configured to receive a backup request, encapsulate the backup request into a backup request event, and write the backup request event into the backup key-value pair to trigger the change event;

[0019] The data writing module is used to obtain backup data from the message middleware through the data node to form a data file, and call the data writing instruction to write the data file to a preset storage location, while suspending the consumption of messages from the message middleware;

[0020] A reporting module, configured to report metadata of the data file through the data node after the data file is written, and write the metadata into a specified set of the key-value database;

[0021] An index building module, configured to generate an index request according to the specified set, and build an index file for the specified set based on the index request through an index node;

[0022] A status monitoring module is used to monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built;

[0023] A backup module, configured to sequentially back up the data file, the index file, and the key-value pairs of the specified set based on the snapshot backup operation to obtain a backup file;

[0024] The recovery module is used to write a backup completion event to the key-value database after the snapshot backup is completed, and to notify the data node to resume consuming the messages of the message middleware.

[0025] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0026] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data backup and recovery method described above when executing the computer-readable instructions.

[0027] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0028] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data backup and recovery method described above.

[0029] Compared with the prior art, this application has the following beneficial effects:

[0030] The present application provides a data backup and recovery method, which encapsulates a backup request as a backup request event and writes it into a backup key-value pair of a key-value database, and monitors it through the monitoring mechanism of the key-value database, that is, the backup request is encapsulated and monitored by the key-value database, so as to realize real-time status management of the backup process; by calling a data write instruction to write the data file to a preset storage location, and at the same time suspending the message consumption of the message middleware, it is possible to avoid data changes during the backup and ensure data consistency; after the data file is written, an index file is constructed according to the metadata of the data file, and when the index file is constructed, a snapshot backup operation is executed for backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity. When encountering emergency production problems and performing cluster switching and rapid recovery, it can meet customer timeliness requirements, and effectively solve the consistency, integrity, availability and timeliness problems of data backup and recovery in a distributed environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0033] Figure 2 is a flow chart of an embodiment of a data backup and recovery method according to the present application;

[0034] Figure 3 This is a structural diagram of an embodiment of a data backup and recovery device according to the present application;

[0035] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0037] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0039] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0040] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0041] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0042] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0043] It should be noted that the data backup and recovery method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data backup and recovery device is generally set in the server / terminal device.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0045] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data backup and recovery method according to the present application, comprising the following steps:

[0046] Step S201: monitoring a modification event of a backup key-value pair in the key-value database through a monitoring mechanism of the key-value database.

[0047] The data backup and recovery method of the present application is implemented based on a vector database architecture. The vector database is used to process large-scale vector data, where vector data is usually generated by deep neural networks and other machine learning models. It can efficiently store, index and manage these vectors and support fast similarity search.

[0048] The vector database architecture uses a distributed key-value database to store core metadata, stored as key-value pairs. The vector database architecture includes components such as the access portal, root coordinator, data nodes, data coordinator, index coordinator, and index nodes. The components related to data files are the data nodes and data coordinator. The data nodes are responsible for listening to message queue logs and writing generated files to object storage. The data coordinator manages the metadata of data files written to object storage and triggers some background data operations. The components related to index files are the index nodes and index coordinator. The index nodes are responsible for building indexes, while the index coordinator is responsible for managing index nodes and tasks and maintaining index metadata. The access portal is responsible for receiving client requests, and the root coordinator is used to execute DCL (Data Control Language) and DDL (Data Definition Language) requests.

[0049] In this embodiment, the data nodes, data coordinator, and root coordinator monitor the backup key-value pairs in real time through the monitoring mechanism of the key-value database, and obtain the change information of the backup status in the key-value database to trigger the change event.

[0050] In some optional implementations of this embodiment, the step of monitoring the change event of the backup key-value pair in the key-value database through the monitoring mechanism of the key-value database includes:

[0051] A monitoring mechanism is used to monitor the key-value database in real time and capture the update operation of the backup status field of the backup key-value pair in the key-value database;

[0052] When an update operation is detected on the backup status field, a change event is triggered and the changed backup status information is recorded.

[0053] Specifically, data nodes, data coordinators, and root coordinators monitor the backup status of the key-value database directly through the database's monitoring mechanisms, such as the watch mechanism, for real-time monitoring. This is achieved by setting a key for the backup status field in the key-value database, such as " / backup / status," and writing the backup status to the value in JSON format. Simultaneously, the key is monitored using the watch mechanism, with the monitoring parameter set to " / backup / status," capturing changes to the value in real time. Once the backup status changes, the key-value database triggers a watch event, known as a change event, which returns the latest backup status as an event.

[0054] In some optional implementations, after obtaining the changed backup status information, the changed backup status information is matched and compared with the preset backup status change rules to determine whether the current backup status change meets the predetermined conditions; based on the matching results of the backup status change rules, a decision tree algorithm is used to classify and process the real-time changed backup status information to obtain a backup status category; for different backup status categories, corresponding information processing strategies are used, such as alarm notifications, log records, etc., to achieve real-time monitoring and management of the backup status. For example, the backup status change rule sets that when the backup status fails for three consecutive times, an abnormality alarm is triggered. Using the decision tree algorithm, the real-time backup status is classified by constructing a mapping relationship between status features and processing decisions; if the backup status is normal, a log is recorded; if an abnormality is detected, an alarm, retry, and other processing strategies are adopted according to the judgment results of the decision tree.

[0055] By monitoring the change events of the key-value database, data changes can be obtained in real time, ensuring that the obtained change information is reliable, and the changes can be notified to the listener in real time to ensure the efficiency of monitoring.

[0056] Step S202: Receive a backup request, encapsulate the backup request into a backup request event, write the backup request event into a backup key-value pair, and trigger a change event.

[0057] The backup request initiated by the client is received through the access portal, and after encapsulating the backup request into a backup request event, it is written into the backup key-value pair of the key-value database so that other components can monitor the backup request event. Because the data node, data coordinator, and root coordinator monitor the backup key-value pair in real time through the monitoring mechanism, when the backup request event is written to the backup key-value pair, the backup key-value pair is changed, triggering a change event, so that the data node, data coordinator, and root coordinator can all monitor the backup request event.

[0058] In this embodiment, the data backup and recovery method is executed on the electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the backup request via a wired connection or a wireless connection. It should be noted that the wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0059] In some optional implementations, the step of encapsulating the backup request into a backup request event and writing the backup request event into a backup key-value pair includes:

[0060] Extract key information from the backup request and construct a backup request object based on the key information;

[0061] Convert backup request objects into standardized event objects based on predefined event types and attribute templates;

[0062] Read the predefined key-value database connection information from the preset configuration file, and establish a connection between the event object and the key-value database according to the connection information;

[0063] Generate a unique key for storing the event according to the attributes of the event object, and serialize the event object into a preset data format to obtain a serialized event object as the value of the key-value pair;

[0064] Call the write interface of the key-value database based on the connection, and write the unique key and serialized event object into the backup key-value pair of the key-value database.

[0065] Specifically, extract key information from the request, such as the backup data type, backup time, etc., build a backup request object, and convert the backup request object into a standardized event object according to the event type and attribute template. The event object contains attributes such as event type, timestamp, and request parameters. The connection information includes the database address, port, access credentials, etc., and a connection is established between the event object and the database based on the connection information. Based on the attributes of the event object, a unique key for storing events is generated, such as using a combination of event type and timestamp as a unique key. The preset data format is a byte array or a JSON string. After the connection is established, call the write interface of the key-value database, pass in the generated key and serialized event object, and write them into the backup key of the key-value database in the form of key-value pairs to achieve persistent storage of event data in the key-value database.

[0066] In this embodiment, the results of event storage, such as success or failure, are recorded. Error information is recorded in case of failure, and retries are performed according to the configured retry strategy to ensure reliable storage of event data. For example, the configured retry strategy can use an exponential backoff algorithm, with the first retry waiting for 1 second, the second waiting for 2 seconds, the third waiting for 4 seconds, and so on, for a maximum of 3 retries.

[0067] By storing backup request events in a key-value database, the stored event data can be made highly reliable. When the stored data changes, corresponding events can be triggered, improving the response speed and responding to data changes instantly.

[0068] Step S203: Obtain backup data from the message middleware through the data node to form a data file, call a data write instruction to write the data file to a preset storage location, and pause consuming messages from the message middleware.

[0069] After the data node monitors the backup request event, it calls the data write instruction through the switch control. The data write instruction can ensure that all data in the message middleware is written to the persistent storage.

[0070] Specifically, it is determined through the data node whether there is currently any data operation in progress; when there is no data operation in progress, all data in the message middleware is obtained as backup data according to the preset data persistence strategy, and a data file is formed.

[0071] Data operations include data writing or data querying. For example, a read-write lock mechanism is used to determine whether a data writing or data query operation is currently in progress. If so, the persistence operation is performed after the data operation is completed. If not, the persistence operation is performed immediately. A persistence operation is an operation that writes a data file to a preset storage location, where the preset storage location is a persistent storage medium such as a disk.

[0072] The preset data persistence strategy includes the time point and frequency at which the data persistence operation needs to be performed, for example, performing the data persistence operation once every hour.

[0073] In this embodiment, while executing the data persistence operation, a notification to suspend consumption is sent to the message middleware. After receiving the notification, the message middleware stops pushing new messages to the data node to avoid data changes during the persistence process and ensure data consistency and stability.

[0074] By obtaining backup data from the message middleware through data nodes to form data files, and persisting the data files, while pausing the consumption of messages from the message middleware during persistence, it can prevent data loss, ensure data consistency, and thus improve data reliability; it can support fault recovery, improve fault tolerance, ensure business continuity, and meet business needs.

[0075] It should be emphasized that in order to further ensure the privacy and security of data files, the above data files can also be stored in a blockchain node.

[0076] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0077] In some optional implementations, the root coordinator monitors a backup request event and suspends processing of data definition language and data control language requests to ensure data structure stability. Specifically, upon monitoring a backup request event, the root coordinator stops receiving and distributing new DDL and DCL requests and waits for all previously distributed requests to be processed. After the root coordinator determines that all previously distributed DDL and DCL requests have been processed, it locks the metadata, prohibiting modifications to the data structure definition.

[0078] Step S204: After the data file is written, the metadata of the data file is reported through the data node, and the metadata is written into a specified set of the key-value database.

[0079] In this embodiment, after a data file is written, the data node reports the metadata of the data file to the data coordinator. After receiving the metadata, the data coordinator uses a distributed task scheduling system to send an index construction request to the index coordinator. This request contains index information for the data set to be indexed, including the starting primary key, ending primary key, index fields, etc. The metadata includes the file name, file type, file size, file creation time, last modification time, file storage location, and checksum.

[0080] In this embodiment, the metadata is written into a preset specified set in the key-value database, so as to construct an index according to the specified set.

[0081] Step S205 : Generate an index request according to the specified set, and construct an index file for the specified set based on the index request through the index node.

[0082] In this embodiment, the data coordinator receives metadata and obtains corresponding index information based on the metadata. The index information includes a starting primary key, an ending primary key, and index fields, where the index fields are keywords for the data file. Based on the index information, an index build request is sent to the index coordinator. The index coordinator analyzes the index information in the index request, determines the target index node based on the starting primary key and the ending primary key, and determines the index content based on the index fields. The target index node and index content are sent to the corresponding index node component. The index node component invokes an index build instruction, builds an index based on the index content, and writes the built index to a pre-created index file. After the index build is complete, a complete index file is obtained.

[0083] In this embodiment, an inverted index algorithm or a B+ tree algorithm may be used to construct an index for a specified collection.

[0084] In some optional implementations, after the index file is written, the index metadata is reported to the index coordinator, and the index metadata is written into the key-value database.

[0085] By building indexes based on metadata through index nodes, the efficiency of index building can be improved, which can improve the efficiency and accuracy of data retrieval.

[0086] Step S206: Monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and when the index file is built, call the backup thread to perform the snapshot backup operation.

[0087] During the indexing process, the data coordinator cyclically calls the application programming interface (API) to monitor the index node's index building status. Based on the index building status, it obtains the index building progress reported by the index node. Based on the obtained index building progress, it determines whether the index file has been built. When the index building progress reaches 100%, the index file construction is complete. When the index file is built, the backup thread is notified to perform a snapshot backup operation, performing a full backup of the data file, index file, and key-value pairs in the specified collection.

[0088] Step S207: backing up the data file, the index file, and the key-value pairs of the specified set in sequence based on the snapshot backup operation to obtain a backup file.

[0089] After responding to the call of the snapshot backup operation, the backup thread obtains a list of key-value pairs for the specified set and determines the data files and index files to be backed up based on the list of key-value pairs. The backup thread uses a multi-threaded parallel processing method to back up each data file and index file to be backed up by file copying, thereby obtaining the backed-up data files and index files. Simultaneously with the data files and index files, the backup thread obtains backup key-value pairs for the specified set from the key-value database and backs them up by key-value pair copying, thereby obtaining the backed-up backup key-value pairs. The backup thread compresses the backed-up data files, index files, and backup key-value pairs using a data compression algorithm to obtain a compressed backup file. The data compression algorithm may use the LZ4 compression algorithm.

[0090] By backing up data files, index files, and key-value pairs of specified collections in sequence based on snapshot backup operations, a global snapshot of the collection data is achieved, thereby improving backup efficiency and minimizing the impact of backup operations on the business.

[0091] In some optional implementations, after the step of sequentially backing up the data file, the index file, and the key-value pairs of the specified set based on the snapshot backup operation to obtain the backup file, the step further includes:

[0092] Use the verification algorithm to verify the backup file and obtain the verification result;

[0093] When the verification result is that the verification is passed, the backup thread is called to record the metadata information of the backup file, and the metadata information of the backup file is written into a preset backup metadata management library.

[0094] The backup thread stores the compressed backup file in a pre-set backup storage location, verifies the backup file using a verification algorithm, obtains a verification result, and determines the integrity and accuracy of the backup file based on the verification result. If the backup file passes the verification, the backup thread records the backup completion timestamp and the backup file's metadata, completing the snapshot backup operation and waiting for the next index replenishment completion notification. For example, the backup file integrity is verified using CRC32.

[0095] By verifying the backup files, the integrity and accuracy of the data backup can be guaranteed, thereby improving the service stability of the vector database product and providing reliability assurance for downstream applications.

[0096] Step S208: After the snapshot backup is completed, a backup completion event is written to the key-value database, and the data node is notified to resume consuming messages from the message middleware.

[0097] After the snapshot backup is complete, an event-driven mechanism creates a backup completion record in the key-value database. This event triggers a process that automatically notifies all relevant data nodes. Upon receiving the backup completion notification, the data nodes resume consuming messages from the message middleware using a pre-defined protocol, ensuring that unfinished data updates continue from the point where they stopped.

[0098] The root coordinator listens to the backup completion event and resumes processing data definition language DDL requests and data control language DCL requests; by parsing the DDL requests and DCL requests, it obtains the operation type and related parameter information in the request; based on the operation type and parameter information, it calls the corresponding processing module to process the request; when processing the DDL request, it updates the table structure metadata in the key-value database based on the table structure definition information in the request; when processing the DCL request, it updates the user permission metadata in the key-value database based on the user permission information in the request; and returns the processing results of the DDL request and DCL request to the request initiator to complete the recovery processing of the request.

[0099] In this embodiment, after the backup is complete, data processing is resumed. The recovery phase only involves uploading or copying the backup file, eliminating the need to rely on index nodes to build indexes. This reduces the cluster load pressure imposed by existing technologies during the index building phase, shortening wait times while ensuring timely recovery. Whether performing backup and restore within the same cluster or migrating across clusters, collection data and indexes can be quickly prepared, improving the service stability of the vector database product and ensuring reliability for downstream applications.

[0100] This application encapsulates the backup request as a backup request event and writes it into the backup key-value pair of the key-value database, and monitors it through the monitoring mechanism of the key-value database, that is, encapsulating and monitoring the backup request through the key-value database to achieve real-time status management of the backup process; by calling the data write instruction to write the data file to the preset storage location, and at the same time pausing the message consumption of the message middleware, it can avoid data changes during the backup and ensure data consistency; after the data file is written, an index file is constructed according to the metadata of the data file, and when the index file is constructed, a snapshot backup operation is executed for backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity.

[0101] In some optional implementations, after the step of notifying the data node to resume consuming messages of the message middleware, the method further includes:

[0102] Compare the timestamp in the key-value database and the timestamp in the message middleware to determine the start point of consumption;

[0103] Process data from the start of consumption through the data node and update the processed data into the key-value database;

[0104] During the data update process, all key operations are recorded through the log system.

[0105] Specifically, after notifying the data node to resume consuming messages from the message middleware, the timestamp of the last message stored in the message middleware is compared with the timestamp of the key-value database to ensure that consumption resumes from the first message after the breakpoint. For example, if the timestamp in the message middleware is 2023-03-01 12:00:00, and the timestamp in the key-value database is 2023-03-01 11:59:59, processing will begin with the message at 12:00:00. When processing new data, data nodes can use batch processing and synchronously update the key-value database. During the data update process, the system will monitor the status of data processing in real time and record all key operations through the log system, including data writing and error handling. Collecting log data can be used for real-time monitoring and analysis to facilitate problem tracking and system optimization.

[0106] By comparing the timestamp to determine the start point of consumption, the consistency and integrity of the data can be ensured, and the processed data can be synchronously updated to the key-value database to ensure the stability of the system.

[0107] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0108] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0109] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a data backup and recovery device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0110] like Figure 3 As shown, the data backup and recovery device 300 of this embodiment includes: an event monitoring module 301, an event writing module 302, a data writing module 303, a reporting module 304, an index building module 305, a status monitoring module 306, a backup module 307 and a recovery module 308. Among them:

[0111] The event monitoring module 301 is used to monitor the modification events of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database;

[0112] The event writing module 302 is configured to receive a backup request, encapsulate the backup request into a backup request event, and write the backup request event into the backup key-value pair to trigger the change event;

[0113] The data writing module 303 is used to obtain backup data from the message middleware through the data node to form a data file, and call the data writing instruction to write the data file to a preset storage location, while suspending the consumption of messages from the message middleware;

[0114] The reporting module 304 is configured to report the metadata of the data file through the data node after the data file is written, and write the metadata into a specified set of the key-value database;

[0115] The index building module 305 is configured to generate an index request according to the specified set, and build an index file for the specified set based on the index request through an index node;

[0116] The status monitoring module 306 is used to monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built;

[0117] The backup module 307 is used to sequentially back up the data file, the index file, and the key-value pairs of the specified set based on the snapshot backup operation to obtain a backup file;

[0118] The recovery module 308 is used to write a backup completion event to the key-value database after the snapshot backup is completed, and notify the data node to resume consuming messages from the message middleware.

[0119] It should be emphasized that in order to further ensure the privacy and security of data files, the above data files can also be stored in a blockchain node.

[0120] Based on the above-mentioned data backup and recovery device 300, by encapsulating the backup request as a backup request event and writing it into the backup key-value pair of the key-value database, and monitoring it through the monitoring mechanism of the key-value database, that is, by encapsulating and monitoring the backup request through the key-value database, real-time status management of the backup process is realized; by calling the data write instruction to write the data file to the preset storage location, and at the same time pausing the message consumption of the message middleware, data changes during the backup can be avoided and data consistency can be ensured; after the data file is written, an index file is constructed according to the metadata of the data file, and when the index file is constructed, a snapshot backup operation is executed for backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity.

[0121] In some optional implementations of this embodiment, the event monitoring module 301 is further configured to:

[0122] Using a monitoring mechanism to monitor the key-value database in real time, capturing the modification operation of the backup status field of the backup key-value pair in the key-value database;

[0123] When a change operation is detected in the backup status field, a change event is triggered and the changed backup status information is recorded.

[0124] By monitoring the change events of the key-value database, data changes can be obtained in real time, ensuring that the obtained change information is reliable, and the changes can be notified to the listener in real time to ensure the efficiency of monitoring.

[0125] In some optional implementations, the event writing module 302 includes:

[0126] A construction submodule, configured to extract key information from the backup request and construct a backup request object based on the key information;

[0127] a conversion submodule, configured to convert the backup request object into a standardized event object according to a predefined event type and attribute template;

[0128] A connection submodule, configured to read predefined connection information of the key-value database from a preset configuration file, and establish a connection between the event object and the key-value database according to the connection information;

[0129] A generating submodule, configured to generate a unique key for storing an event according to the attributes of the event object, and serialize the event object into a preset data format to obtain a serialized event object as a value of a key-value pair;

[0130] A writing submodule is used to call the writing interface of the key-value database based on the connection, and write the unique key and the serialized event object into the backup key-value pair.

[0131] By storing backup request events in a key-value database, the stored event data can be made highly reliable. When the stored data changes, corresponding events can be triggered, improving the response speed and responding to data changes instantly.

[0132] In this embodiment, the data writing module 303 is further configured to:

[0133] Determine whether there is currently an ongoing data operation through the data node;

[0134] When there is no ongoing data operation, all data in the message middleware is obtained as backup data according to a preset data persistence strategy, and a data file is formed.

[0135] By obtaining backup data from the message middleware through data nodes to form data files, and persisting the data files, while pausing the consumption of messages from the message middleware during persistence, it can prevent data loss, ensure data consistency, and thus improve data reliability; it can support fault recovery, improve fault tolerance, ensure business continuity, and meet business needs.

[0136] In some optional implementations, the index building module 305 is further configured to:

[0137] Analyze the index information of the index request to obtain the target index node and index content to be indexed;

[0138] An index building instruction is called through the target index node, an index is built according to the index content, and an index file is obtained.

[0139] By building indexes based on metadata through index nodes, the efficiency of index building can be improved, which can improve the efficiency and accuracy of data retrieval.

[0140] In some optional implementations, the data backup and recovery apparatus 300 further includes a verification module configured to:

[0141] Verifying the backup file using a verification algorithm to obtain a verification result;

[0142] When the verification result is that the verification is passed, the backup thread is called to record the metadata information of the backup file, and the metadata information of the backup file is written into a preset backup metadata management library.

[0143] By verifying the backup files, the integrity and accuracy of the data backup can be guaranteed, thereby improving the service stability of the vector database product and providing reliability assurance for downstream applications.

[0144] In some optional implementations, the data backup and recovery device 300 further includes a data update module configured to:

[0145] Compare the timestamp in the key-value database with the timestamp in the message middleware to determine the start point of consumption;

[0146] Processing data from the start consumption position through the data node, and updating the processed data into a key-value database;

[0147] During the data update process, all key operations are recorded through the log system.

[0148] By comparing the timestamp to determine the start point of consumption, the consistency and integrity of the data can be ensured, and the processed data can be synchronously updated to the key-value database to ensure the stability of the system.

[0149] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0150] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0151] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0152] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for data backup and recovery methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0153] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or process data, such as executing computer-readable instructions for the data backup and recovery method.

[0154] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0155] By encapsulating the backup request as a backup request event and writing it into the backup key-value pair of the key-value database, and monitoring it through the monitoring mechanism of the key-value database, that is, encapsulating and monitoring the backup request through the key-value database, real-time status management of the backup process is achieved; by calling the data write instruction to write the data file to the preset storage location, and pausing the message consumption of the message middleware, data changes during the backup can be avoided and data consistency can be ensured; after the data file is written, an index file is constructed according to the metadata of the data file. When the index file is constructed, a snapshot backup operation is performed for backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity.

[0156] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data backup and recovery method as described above.

[0157] By encapsulating the backup request as a backup request event and writing it into the backup key-value pair of the key-value database, and monitoring it through the monitoring mechanism of the key-value database, that is, encapsulating and monitoring the backup request through the key-value database, real-time status management of the backup process is achieved; by calling the data write instruction to write the data file to the preset storage location, and pausing the message consumption of the message middleware, data changes during the backup can be avoided and data consistency can be ensured; after the data file is written, an index file is constructed according to the metadata of the data file. When the index file is constructed, a snapshot backup operation is performed for backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0159] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A data backup and recovery method, characterized in that: The steps include: Monitoring the change events of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database; receiving a backup request, encapsulating the backup request into a backup request event, and writing the backup request event into the backup key-value pair, thereby triggering the change event; Obtain backup data from the message middleware through the data node to form a data file, and call the data write instruction to write the data file to the preset storage location, while pausing the consumption of messages from the message middleware; After the data file is written, the metadata of the data file is reported through the data node, and the metadata is written into a specified set of the key-value database; Generate an index request according to the specified set, and construct an index file for the specified set based on the index request through an index node; Monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built; Backing up the data file, the index file, and the key-value pairs of the specified set in sequence based on the snapshot backup operation to obtain a backup file; After the snapshot backup is completed, a backup completion event is written to the key-value database, and the data node is notified to resume consuming messages from the message middleware; The step of encapsulating the backup request into a backup request event and writing the backup request event into the backup key-value pair includes: Extracting key information from the backup request and constructing a backup request object based on the key information; converting the backup request object into a standardized event object based on a predefined event type and attribute template; reading predefined connection information for the key-value database from a preset configuration file, and establishing a connection between the event object and the key-value database based on the connection information; generating a unique key for storing events based on the attributes of the event object, and serializing the event object into a preset data format to obtain a serialized event object as the value of a key-value pair; calling a write interface of the key-value database based on the connection, and writing the unique key and the serialized event object into the backup key-value pair; The steps of constructing the index file include: Based on the index information, an index request for building an index is sent to the index coordinator. The index coordinator analyzes the index information in the index request, determines the target index node based on the starting primary key and the ending primary key, and determines the index content based on the index field; sends the target index node and index content to the corresponding index node component, calls the index building instruction through the index node component, builds the index according to the index content, and writes the built index into the pre-created index file. After the index building is completed, a complete index file is obtained.

2. The data backup and recovery method according to claim 1, wherein: The step of monitoring the modification event of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database includes: Using a monitoring mechanism to monitor the key-value database in real time, capturing the modification operation of the backup status field of the backup key-value pair in the key-value database; When a change operation is detected in the backup status field, a change event is triggered and the changed backup status information is recorded.

3. The data backup and recovery method according to claim 1, wherein: The step of obtaining backup data from the message middleware through the data node to form a data file includes: Determine whether there is currently an ongoing data operation through the data node; When there is no ongoing data operation, all data in the message middleware is obtained as backup data according to a preset data persistence strategy, and a data file is formed.

4. The data backup and recovery method according to claim 1, wherein: After the step of sequentially backing up the data file, the index file, and the key-value pairs of the specified set based on the snapshot backup operation to obtain a backup file, the step further includes: Verifying the backup file using a verification algorithm to obtain a verification result; When the verification result is that the verification is passed, the backup thread is called to record the metadata information of the backup file, and the metadata information of the backup file is written into a preset backup metadata management library.

5. The data backup and recovery method according to claim 1, wherein: After the step of notifying the data node to resume consuming messages of the message middleware, the method further includes: Compare the timestamp in the key-value database with the timestamp in the message middleware to determine the start point of consumption; Processing data from the start consumption position through the data node, and updating the processed data into a key-value database; During the data update process, all key operations are recorded through the log system.

6. A data backup and recovery device, characterized in that: include: An event monitoring module, configured to monitor, through a monitoring mechanism of the key-value database, change events of backup key-value pairs in the key-value database; An event writing module, configured to receive a backup request, encapsulate the backup request into a backup request event, and write the backup request event into the backup key-value pair to trigger the change event; The data writing module is used to obtain backup data from the message middleware through the data node to form a data file, and call the data writing instruction to write the data file to a preset storage location, while suspending the consumption of messages from the message middleware; A reporting module, configured to report metadata of the data file through the data node after the data file is written, and write the metadata into a specified set of the key-value database; An index building module, configured to generate an index request according to the specified set, and build an index file for the specified set based on the index request through an index node; A status monitoring module is used to monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built; A backup module, configured to sequentially back up the data file, the index file, and the key-value pairs of the specified set based on the snapshot backup operation to obtain a backup file; A recovery module is used to write a backup completion event to the key-value database after the snapshot backup is completed, and to notify the data node to resume consuming messages from the message middleware; The event writing module includes: A construction submodule, configured to extract key information from the backup request and construct a backup request object based on the key information; a conversion submodule, configured to convert the backup request object into a standardized event object according to a predefined event type and attribute template; A connection submodule, configured to read predefined connection information of the key-value database from a preset configuration file, and establish a connection between the event object and the key-value database according to the connection information; A generating submodule, configured to generate a unique key for storing an event according to the attributes of the event object, and serialize the event object into a preset data format to obtain a serialized event object as a value of a key-value pair; A writing submodule, configured to call a writing interface of the key-value database based on the connection, and write the unique key and the serialized event object into the backup key-value pair; The steps of constructing the index file include: Based on the index information, an index request for building an index is sent to the index coordinator. The index coordinator analyzes the index information in the index request, determines the target index node based on the starting primary key and the ending primary key, and determines the index content based on the index field; sends the target index node and index content to the corresponding index node component, calls the index building instruction through the index node component, builds the index according to the index content, and writes the built index into the pre-created index file. After the index building is completed, a complete index file is obtained.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data backup and recovery method according to any one of claims 1 to 5 when executing the computer-readable instructions.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data backup and recovery method according to any one of claims 1 to 5.

Images