Data backup recovery method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence and financial science and technology, and relates to a data backup recovery method which comprises the following steps: monitoring through a monitoring mechanism of a key value database, packaging a backup request into a backup request event, and writing the backup request event into a backup key value pair of the key value database; when a backup request event is monitored, calling a data writing instruction to write a data file into a preset storage position, and meanwhile, pausing a message of the consumption message middleware; after writing of the data file is completed, an index file is constructed according to metadata of the data file, and when construction of the index file is completed, snapshot backup operation is executed for backup; and after backup is completed, normal data processing is recovered. The invention further provides a data backup recovery device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the data file can be stored in a block chain. According to the method, the data backup and recovery time can be greatly shortened, and the data backup and recovery timeliness is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a data backup and recovery method, device, computer equipment and storage medium. Background Art

[0002] In the financial field, data integrity, security, and recoverability are crucial. With the rapid development of financial services, the amount of data has exploded, and traditional relational databases are unable to cope with large-scale, high-dimensional vector data. Vector database Milvus, as a storage and retrieval system designed specifically for vector data, has been widely used in the financial field. It can efficiently store and retrieve vector data such as financial transaction records and user behavior characteristics, providing strong support for applications such as financial risk control and smart investment advisors.

[0003] In order to meet the high requirements of the financial field for data backup and recovery, Milvus database supports backup and recovery functions. Through its client tool milvus-backup, users can easily execute the create command to back up data and the restore command to restore data. What is more worth mentioning is that Milvus also supports backup and recovery based on library and collection granularity, providing financial users with more flexible data management methods.

[0004] However, although Milvus' backup and recovery function performs well in backing up and restoring data files, it has a significant flaw: it can only back up data files, but cannot directly back up index files. This means that when restoring data, it is necessary to re-create the index for the new collection based on the backed-up index metadata. The index node starts the index building process only after receiving the command to create the index. Although this design provides flexibility to a certain extent, for example, the prefix and suffix of the collection can be specified during recovery, the implementation of index backup and recovery is relatively complicated. Especially after creating a new collection and writing data in batches according to the file, due to changes in the data organization structure, the old index files often cannot be directly matched with the new collection.

[0005] In the financial sector, this index reconstruction process is not only time-consuming, but also has the problem of uncertain timeliness. In the operation of financial business, if there is an urgent cluster switch or rapid recovery requirement, the existing backup and recovery solutions often cannot meet the customer's timeliness requirements. Especially when dealing with large-scale data sets, the timeliness problem of index reconstruction is particularly prominent. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose a data backup and recovery method, device, computer equipment and storage medium to solve the technical problem that the existing data backup and recovery method cannot back up the index file and needs to recreate the index, resulting in a long backup and recovery time and uncertain timeliness.

[0007] In order to solve the above technical problems, the present application provides a data backup and recovery method, which adopts the following technical solution:

[0008] Monitoring the change events of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database;

[0009] receiving a backup request, encapsulating the backup request into a backup request event, and writing the backup request event into the backup key-value pair, triggering the change event;

[0010] Obtain backup data from the message middleware through the data node to form a data file, and call the data write instruction to write the data file to a preset storage location, while pausing the consumption of messages from the message middleware;

[0011] After the data file is written, the metadata of the data file is reported through the data node, and the metadata is written into a specified set of the key-value database;

[0012] Generate an index request according to the specified set, and construct an index file for the specified set based on the index request through an index node;

[0013] Monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built;

[0014] Based on the snapshot backup operation, the data file, the index file and the key-value pairs of the specified set are backed up in sequence to obtain a backup file;

[0015] After the snapshot backup is completed, a backup completion event is written to the key-value database, and the data node is notified to resume consuming messages from the message middleware.

[0016] In order to solve the above technical problems, the embodiment of the present application further provides a data backup and recovery device, which adopts the following technical solution:

[0017] An event monitoring module, used for monitoring the change events of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database;

[0018] An event writing module, used for receiving a backup request, encapsulating the backup request into a backup request event, and writing the backup request event into the backup key-value pair to trigger the change event;

[0019] A data writing module is used to obtain backup data from the message middleware through a data node to form a data file, and to call a data writing instruction to write the data file to a preset storage location, while suspending the consumption of messages from the message middleware;

[0020] A reporting module, configured to report metadata of the data file through the data node after the data file is written, and write the metadata into a specified set of the key-value database;

[0021] An index building module, used to generate an index request according to the specified set, and build an index file for the specified set based on the index request through an index node;

[0022] A status monitoring module is used to monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built;

[0023] A backup module, used to sequentially back up the data file, the index file and the key-value pairs of the specified set based on the snapshot backup operation to obtain a backup file;

[0024] The recovery module is used to write a backup completion event to the key-value database after the snapshot backup is completed, and to notify the data node to resume consuming the messages of the message middleware.

[0025] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0026] The computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data backup and recovery method as described above when executing the computer-readable instructions.

[0027] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0028] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the data backup and recovery method described above are implemented.

[0029] Compared with the prior art, this application has the following beneficial effects:

[0030] The present application provides a data backup and recovery method, which encapsulates a backup request as a backup request event and writes it into a backup key-value pair of a key-value database, and monitors it through the monitoring mechanism of the key-value database, that is, encapsulates and monitors the backup request through the key-value database to achieve real-time status management of the backup process; writes the data file to a preset storage location by calling a data write instruction, and pauses the message consumption of the message middleware, which can avoid data changes during the backup and ensure data consistency; after the data file is written, an index file is constructed according to the metadata of the data file, and when the index file is constructed, a snapshot backup operation is performed to perform the backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity, and meeting customer timeliness requirements when cluster switching and rapid recovery are performed in the event of emergency production problems, effectively solving the consistency, integrity, availability and timeliness issues of data backup and recovery in a distributed environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0033] Figure 2 is a flow chart of an embodiment of a data backup and recovery method according to the present application;

[0034] Figure 3 is a structural schematic diagram of an embodiment of a data backup and recovery device according to the present application;

[0035] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0039] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0040] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0041] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0042] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0043] It should be noted that the data backup and recovery method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data backup and recovery device is generally set in the server / terminal device.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0045] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data backup and recovery method according to the present application, comprising the following steps:

[0046] Step S201: monitor the modification event of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database.

[0047] The data backup and recovery method of the present application is implemented based on a vector database architecture. The vector database is used to process large-scale vector data, wherein the vector data is usually generated by deep neural networks and other machine learning models. It can efficiently store, index and manage these vectors and support fast similarity search.

[0048] Among them, the vector database architecture uses a distributed key-value database to store core metadata, and the metadata is stored in the form of key-value pairs. The vector database architecture includes components such as access portal, root coordinator, data node, data coordinator, index coordinator and index node. The components related to data files are data nodes and data coordinators. Data nodes are responsible for listening to message queue logs and writing files to object storage after generating them. Data coordinators manage metadata written to object storage data files and trigger some background data operations; components related to index files are index nodes and index coordinators. Index nodes are responsible for building indexes, and index coordinators are responsible for managing index nodes and tasks and maintaining index metadata. The access portal is responsible for receiving client requests, and the root coordinator is used to execute DCL (Data Control Language) and DDL (Data Definition Language) requests.

[0049] In this embodiment, the data node, the data coordinator, and the root coordinator monitor the backup key-value pairs in real time through the monitoring mechanism of the key-value database, and obtain the change information of the backup status in the key-value database to trigger the change event.

[0050] In some optional implementations of this embodiment, the step of monitoring the change event of the backup key-value pair in the key-value database through the monitoring mechanism of the key-value database includes:

[0051] A monitoring mechanism is used to monitor the key-value database in real time, and the update operation of the backup status field of the backup key-value pair in the key-value database is captured;

[0052] When an update operation is detected on the backup status field, a change event is triggered and the changed backup status information is recorded.

[0053] Specifically, the data node, data coordinator, and root coordinator monitor the backup status of the key-value database, and monitor in real time directly through the monitoring mechanism of the key-value database, such as the watch mechanism. By setting the key of the backup status field in the key-value database, such as " / backup / status", and writing the backup status in the value in JSON format, the status storage is realized. At the same time, the key is monitored based on the watch mechanism, and the monitoring parameter is set to " / backup / status" to capture the changes of the value in real time. Once the backup status changes, the key-value database will trigger a watch event, that is, a change event, and return the latest backup status in the form of an event.

[0054] In some optional implementations, after obtaining the changed backup status information, the changed backup status information is matched and compared with the preset backup status change rules to determine whether the current backup status change meets the predetermined conditions; based on the matching results of the backup status change rules, a decision tree algorithm is used to classify and process the real-time changed backup status information to obtain the backup status category; for different backup status categories, corresponding information processing strategies are used, such as alarm notifications, log records, etc., to achieve real-time monitoring and management of the backup status. For example, the backup status change rule sets that when the backup status fails for three consecutive times, an abnormal alarm is triggered. Using the decision tree algorithm, the real-time backup status is classified by constructing a mapping relationship between status features and processing decisions; if the backup status is normal, a log is recorded; if an abnormality is detected, an alarm, retry, and other processing strategies are adopted according to the judgment results of the decision tree.

[0055] By monitoring the change events of the key-value database, data changes can be obtained in real time, ensuring that the obtained change information is reliable, and the changes can be notified to the listener in real time to ensure the efficiency of monitoring.

[0056] Step S202: receiving a backup request, encapsulating the backup request into a backup request event, and writing the backup request event into a backup key-value pair to trigger a change event.

[0057] The backup request initiated by the client is received through the access portal, and after the backup request is encapsulated into a backup request event, it is written into the backup key-value pair of the key-value database so that other components can monitor the backup request event. Since the data node, data coordinator, and root coordinator monitor the backup key-value pair in real time through the monitoring mechanism, when the backup request event is written into the backup key-value pair, the backup key-value pair is changed, triggering a change event, so that the data node, data coordinator, and root coordinator can monitor the backup request event.

[0058] In this embodiment, the data backup and recovery method is executed on the electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the backup request through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0059] In some optional implementations, the step of encapsulating the backup request into a backup request event and writing the backup request event into a backup key-value pair includes:

[0060] Extract key information from the backup request and construct a backup request object based on the key information;

[0061] Convert backup request objects into standardized event objects according to predefined event types and attribute templates;

[0062] Read the connection information of the predefined key-value database from the preset configuration file, and establish a connection between the event object and the key-value database according to the connection information;

[0063] Generate a unique key for storing the event according to the attributes of the event object, and serialize the event object into a preset data format to obtain a serialized event object as a value of the key-value pair;

[0064] Based on the connection, the write interface of the key-value database is called to write the unique key and serialized event object into the backup key-value pair of the key-value database.

[0065] Specifically, extract key information in the request, such as the backup data type, backup time, etc., build a backup request object, and convert the backup request object into a standardized event object according to the event type and attribute template. The event object contains attributes such as event type, timestamp, and request parameters. The connection information includes the database address, port, access credentials, etc., and the connection between the event object and the database is established based on the connection information. According to the attributes of the event object, a unique key for storing events is generated, for example, a combination of event type and timestamp is used as the unique key. The preset data format is a byte array or a JSON string. After the connection is established, call the write interface of the key-value database, pass in the generated key and serialized event object, and write it into the backup key value of the key-value database in the form of a key-value pair to achieve persistent storage of event data in the key-value database.

[0066] In this embodiment, the result of event storage is recorded, such as success or failure, error information is recorded when failure occurs, and retries are performed according to the configured retry strategy to ensure reliable storage of event data. For example, the configured retry strategy can use an exponential backoff algorithm, with the first retry waiting for 1 second, the second waiting for 2 seconds, the third waiting for 4 seconds, and so on, with a maximum of 3 retries.

[0067] By storing backup request events in a key-value database, the stored event data can be made highly reliable. When the stored data changes, corresponding events can be triggered, improving the response speed and responding to data changes instantly.

[0068] Step S203, obtaining backup data from the message middleware through the data node to form a data file, and calling a data write instruction to write the data file to a preset storage location, while suspending the consumption of messages from the message middleware.

[0069] After the data node monitors the backup request event, it calls the data write instruction through the switch control. The data write instruction can ensure that all data in the message middleware is written to the persistent storage.

[0070] Specifically, it is determined through the data node whether there is any ongoing data operation; when there is no ongoing data operation, all data in the message middleware is obtained as backup data according to the preset data persistence strategy, and a data file is formed.

[0071] Among them, data operations include data writing or data query, etc. For example, the read-write lock mechanism is used to determine whether there is a data writing or data query operation in progress. If so, wait until the data operation is completed before executing the persistence operation. If not, execute the persistence operation immediately. The persistence operation refers to the operation of writing a data file to a preset storage location, wherein the preset storage location is a persistent storage medium, such as a disk.

[0072] The preset data persistence strategy includes the time point and frequency at which the data persistence operation needs to be performed, for example, performing a data persistence operation once every hour.

[0073] In this embodiment, while executing the data persistence operation, a notification to suspend consumption is sent to the message middleware. After receiving the notification, the message middleware stops pushing new messages to the data node to avoid data changes during the persistence process and ensure data consistency and stability.

[0074] By obtaining backup data from the message middleware through data nodes to form data files, and persisting the data files, while pausing the consumption of messages from the message middleware, data loss can be prevented, data consistency can be ensured, and data reliability can be improved; it can support fault recovery, improve fault tolerance, ensure business continuity, and meet business needs.

[0075] It should be emphasized that in order to further ensure the privacy and security of data files, the above data files can also be stored in a blockchain node.

[0076] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0077] In some optional implementations, the root coordinator monitors the backup request event and suspends processing of data definition language requests and data control language requests to ensure the stability of the data structure. Specifically, the root coordinator monitors the backup request event, stops receiving and distributing new DDL requests and DCL requests, and waits for the distributed requests to be processed; after the root coordinator determines that all distributed DDL requests and DCL requests have been processed, it locks the metadata and prohibits the modification of the data structure definition.

[0078] Step S204: After the data file is written, the metadata of the data file is reported through the data node, and the metadata is written into a specified set of the key-value database.

[0079] In this embodiment, after the data file is written, the metadata of the data file is reported to the data coordinator through the data node. After receiving the metadata, the data coordinator uses a distributed task scheduling system to send a request to build an index to the index coordinator. This request contains the index information of the data set to be indexed, and the index information includes the start primary key, the end primary key, the index field, etc. The metadata includes the file name, file type, file size, file creation time, last modification time, file storage location, and checksum, etc.

[0080] In this embodiment, the metadata is written into a preset specified set in the key-value database to construct an index according to the specified set.

[0081] Step S205: Generate an index request according to the specified set, and construct an index file for the specified set based on the index request through the index node.

[0082] In this embodiment, the data coordinator receives metadata and obtains corresponding index information according to the metadata, the index information includes the starting primary key, the ending primary key, the index field, etc., and the index field is the keyword of the data file. Based on the index information, an index request for building an index is sent to the index coordinator, and the index coordinator analyzes the index information in the index request, determines the target index node according to the starting primary key and the ending primary key, and determines the index content according to the index field; sends the target index node and the index content to the corresponding index node component, calls the index building instruction through the index node component, builds the index according to the index content, and writes the built index into the pre-created index file. After the index building is completed, a complete index file is obtained.

[0083] In this embodiment, an inverted index algorithm or a B+ tree algorithm may be used to construct an index for a specified collection.

[0084] In some optional implementations, after the index file is written, the index metadata is reported to the index coordinator, and the index metadata is written into the key-value database.

[0085] By building indexes based on metadata through index nodes, the efficiency of index building can be improved, which can improve the efficiency and accuracy of data retrieval.

[0086] Step S206, monitoring the index building status of the index node by cyclically calling the application program interface, judging whether the index file is built according to the index building status, and calling the backup thread to perform the snapshot backup operation when the index file is built.

[0087] During the index building process, the data coordinator cyclically calls the application program interface to monitor the index building status of the index node, obtains the index building progress reported by the index node according to the index building status, and determines whether the index file has been built based on the obtained index building progress. When the index building progress reaches 100%, it means that the index file is built. When the index file is built, the backup thread is notified to perform a snapshot backup operation to fully back up the data file, index file, and key-value pairs of the specified collection.

[0088] Step S207: back up the data file, the index file and the key-value pairs of the specified set in sequence based on the snapshot backup operation to obtain a backup file.

[0089] After the backup thread responds to the call of the snapshot backup operation, it obtains the key-value pair list of the specified collection, and determines the data files and index files that need to be backed up according to the key-value pair list; the backup thread adopts a multi-threaded parallel processing method, and backs up each data file and index file that needs to be backed up by file copying to obtain the backed-up data files and index files; at the same time as the data files and index files, the backup thread obtains the backup key-value pairs of the specified collection in the key-value database, and backs them up by key-value pair copying to obtain the backed-up backup key-value pairs; the backup thread uses a data compression algorithm to compress the backed-up data files, index files and backup key-value pairs to obtain the compressed backup files. Among them, the data compression algorithm can adopt the LZ4 compression algorithm.

[0090] By backing up data files, index files, and key-value pairs of specified collections in sequence based on snapshot backup operations, a global snapshot of collection data is implemented, thereby improving backup efficiency and minimizing the impact of backup operations on business.

[0091] In some optional implementations, after the step of sequentially backing up the data file, the index file, and the key-value pairs of the specified set based on the snapshot backup operation to obtain the backup file, the step further includes:

[0092] Use the verification algorithm to verify the backup file and obtain the verification result;

[0093] When the verification result is that the verification is passed, the backup thread is called to record the metadata information of the backup file, and the metadata information of the backup file is written into a preset backup metadata management library.

[0094] The backup thread stores the compressed backup file to the preset backup storage location, verifies the backup file through the verification algorithm, obtains the verification result, and judges the integrity and accuracy of the backup file according to the verification result; if the backup file passes the verification, the backup thread records the backup completion timestamp and the metadata information of the backup file, completes the snapshot backup operation, and waits for the next index supplement completion notification. For example, the integrity of the backup file is verified by CRC32.

[0095] By verifying the backup files, the integrity and accuracy of the data backup can be guaranteed, thereby improving the service stability of the vector database product and providing reliability assurance for downstream applications.

[0096] Step S208, after the snapshot backup is completed, a backup completion event is written to the key-value database, and the data node is notified to resume consuming messages from the message middleware.

[0097] After the snapshot backup is completed, a backup completion event record is created in the key-value database through the event-driven mechanism to record the backup completion event. The backup completion event triggers a process to automatically notify all related data nodes. After the data node receives the backup completion notification, it resumes consuming messages from the message middleware through the preset protocol to ensure that the unfinished data updates are processed from the stop point.

[0098] The root coordinator listens to the backup completion event, and resumes processing the data definition language DDL request and the data control language DCL request; obtains the operation type and related parameter information in the request by parsing the DDL request and the DCL request; calls the corresponding processing module to process the request according to the operation type and parameter information; when processing the DDL request, updates the table structure metadata in the key-value database according to the table structure definition information in the request; when processing the DCL request, updates the user permission metadata in the key-value database according to the user permission information in the request; returns the processing results of the DDL request and the DCL request to the request initiator, and completes the recovery processing of the request.

[0099] In this embodiment, after the backup is completed, the data processing is restored. In the recovery phase, only the upload or copy of the backup file is involved, and the index is no longer dependent on the index node to build the index, which releases the cluster load pressure brought by the existing technology in the index building phase. While shortening the waiting time, it also ensures the recovery time. Whether in the same cluster backup and recovery or cross-cluster migration scenarios, the collection data and indexes can be quickly prepared, thereby improving the service stability of the vector database product and providing reliability guarantee for downstream applications.

[0100] The present application encapsulates the backup request as a backup request event and writes it into the backup key-value pair of the key-value database, and monitors it through the monitoring mechanism of the key-value database, that is, encapsulates and monitors the backup request through the key-value database to achieve real-time status management of the backup process; writes the data file to a preset storage location by calling the data write instruction, and pauses the message consumption of the message middleware, which can avoid data changes during the backup and ensure data consistency; after the data file is written, an index file is constructed according to the metadata of the data file, and when the index file is constructed, a snapshot backup operation is performed to ensure the consistency and integrity of the backup data, thereby improving the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity.

[0101] In some optional implementations, after the step of notifying the data node to resume consuming the message of the message middleware, the method further includes:

[0102] Compare the timestamp in the key-value database and the timestamp in the message middleware to determine the start point of consumption;

[0103] Process data from the start of consumption through the data node and update the processed data into the key-value database;

[0104] During the data update process, all key operations are recorded through the log system.

[0105] Specifically, after notifying the data node to resume consuming messages from the message middleware, the timestamp of the last message stored in the message middleware is compared with the timestamp of the key-value database to ensure that consumption starts again from the first message after the breakpoint. For example, if the timestamp in the message middleware is 2023-03-01 12:00:00, and the timestamp in the key-value database is 2023-03-01 11:59:59, then processing starts from the message at 12:00:00. When the data node processes new data, it can use batch processing and update it synchronously to the key-value database. During the data update process, the system will monitor the status of data processing in real time and record all key operations through the log system, including data writing, error handling, etc. Collecting log data can be used for real-time monitoring and analysis to facilitate problem tracking and system optimization.

[0106] By comparing the timestamp to determine the start of consumption, the consistency and integrity of the data can be ensured, and the processed data can be synchronously updated to the key-value database to ensure the stability of the system.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0108] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0109] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a data backup and recovery device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0110] like Figure 3 As shown, the data backup and recovery device 300 described in this embodiment includes: an event monitoring module 301, an event writing module 302, a data writing module 303, a reporting module 304, an index building module 305, a status monitoring module 306, a backup module 307 and a recovery module 308. Among them:

[0111] The event monitoring module 301 is used to monitor the change event of the backup key-value pair in the key-value database through the monitoring mechanism of the key-value database;

[0112] The event writing module 302 is used to receive a backup request, encapsulate the backup request into a backup request event, and write the backup request event into the backup key-value pair to trigger the change event;

[0113] The data writing module 303 is used to obtain backup data from the message middleware through the data node to form a data file, and call the data writing instruction to write the data file to a preset storage location, while suspending the consumption of messages from the message middleware;

[0114] The reporting module 304 is used to report the metadata of the data file through the data node after the data file is written, and write the metadata into a specified set of the key-value database;

[0115] The index building module 305 is used to generate an index request according to the specified set, and build an index file for the specified set based on the index request through an index node;

[0116] The status monitoring module 306 is used to monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform the snapshot backup operation when the index file is built;

[0117] The backup module 307 is used to sequentially back up the data file, the index file and the key-value pair of the specified set based on the snapshot backup operation to obtain a backup file;

[0118] The recovery module 308 is used to write a backup completion event to the key-value database after the snapshot backup is completed, and to notify the data node to resume consuming the messages of the message middleware.

[0119] It should be emphasized that in order to further ensure the privacy and security of data files, the above data files can also be stored in a blockchain node.

[0120] Based on the above-mentioned data backup and recovery device 300, by encapsulating the backup request as a backup request event and writing it into the backup key-value pair of the key-value database, and listening through the listening mechanism of the key-value database, that is, encapsulating and listening to the backup request through the key-value database, real-time status management of the backup process is realized; by calling the data write instruction to write the data file to the preset storage location, and pausing the message consumption of the message middleware at the same time, data changes during the backup can be avoided and data consistency can be ensured; after the data file is written, an index file is constructed according to the metadata of the data file, and when the index file is constructed, a snapshot backup operation is performed for backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity.

[0121] In some optional implementations of this embodiment, the event monitoring module 301 is further used to:

[0122] Using a monitoring mechanism to monitor the key-value database in real time, capturing the change operation of the backup status field of the backup key-value pair in the key-value database;

[0123] When a change operation is detected in the backup status field, a change event is triggered and the changed backup status information is recorded.

[0124] By monitoring the change events of the key-value database, data changes can be obtained in real time, ensuring that the obtained change information is reliable, and the changes can be notified to the listener in real time to ensure the efficiency of monitoring.

[0125] In some optional implementations, the event writing module 302 includes:

[0126] A construction submodule, used to extract key information in the backup request and construct a backup request object according to the key information;

[0127] A conversion submodule, used for converting the backup request object into a standardized event object according to a predefined event type and attribute template;

[0128] A connection submodule, used for reading predefined connection information of the key-value database from a preset configuration file, and establishing a connection between the event object and the key-value database according to the connection information;

[0129] A generating submodule, used for generating a unique key for storing events according to the attributes of the event object, and serializing the event object into a preset data format to obtain a serialized event object as a value of a key-value pair;

[0130] A writing submodule is used to call the writing interface of the key-value database based on the connection, and write the unique key and the serialized event object into the backup key-value pair.

[0131] By storing backup request events in a key-value database, the stored event data can be made highly reliable. When the stored data changes, corresponding events can be triggered, improving the response speed and responding to data changes instantly.

[0132] In this embodiment, the data writing module 303 is further used for:

[0133] Determine whether there is currently an ongoing data operation through the data node;

[0134] When there is no ongoing data operation, all data in the message middleware is obtained as backup data according to a preset data persistence strategy, and a data file is formed.

[0135] By obtaining backup data from the message middleware through data nodes to form data files, and persisting the data files, while pausing the consumption of messages from the message middleware, data loss can be prevented, data consistency can be ensured, and data reliability can be improved; it can support fault recovery, improve fault tolerance, ensure business continuity, and meet business needs.

[0136] In some optional implementations, the index building module 305 is further used to:

[0137] Analyze the index information of the index request to obtain the target index node and index content to be indexed;

[0138] An index building instruction is called through the target index node, an index is built according to the index content, and an index file is obtained.

[0139] By building indexes based on metadata through index nodes, the efficiency of index building can be improved, which can improve the efficiency and accuracy of data retrieval.

[0140] In some optional implementations, the data backup and recovery device 300 further includes a verification module, which is used to:

[0141] Using a verification algorithm to verify the backup file to obtain a verification result;

[0142] When the verification result is that the verification is passed, the backup thread is called to record the metadata information of the backup file, and the metadata information of the backup file is written into a preset backup metadata management library.

[0143] By verifying the backup files, the integrity and accuracy of the data backup can be guaranteed, thereby improving the service stability of the vector database product and providing reliability assurance for downstream applications.

[0144] In some optional implementations, the data backup and recovery device 300 further includes a data update module, which is used to:

[0145] Compare the timestamp in the key-value database with the timestamp in the message middleware to determine the start consumption position;

[0146] Processing data from the start consumption position through the data node, and updating the processed data into a key-value database;

[0147] During the data update process, all key operations are recorded through the log system.

[0148] By comparing the timestamp to determine the start of consumption, the consistency and integrity of the data can be ensured, and the processed data can be synchronously updated to the key-value database to ensure the stability of the system.

[0149] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0150] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0151] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0152] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a data backup and recovery method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0153] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the data backup and recovery method.

[0154] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0155] By encapsulating the backup request as a backup request event and writing it into the backup key-value pair of the key-value database, and listening through the listening mechanism of the key-value database, that is, encapsulating and listening to the backup request through the key-value database, real-time status management of the backup process is realized; by calling the data write instruction to write the data file to the preset storage location, and pausing the message consumption of the message middleware at the same time, data changes during the backup can be avoided and data consistency can be ensured; after the data file is written, an index file is built according to the metadata of the data file, and when the index file is built, a snapshot backup operation is performed for backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity.

[0156] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data backup and recovery method as described above.

[0157] By encapsulating the backup request as a backup request event and writing it into the backup key-value pair of the key-value database, and listening through the listening mechanism of the key-value database, that is, encapsulating and listening to the backup request through the key-value database, real-time status management of the backup process is realized; by calling the data write instruction to write the data file to the preset storage location, and pausing the message consumption of the message middleware at the same time, data changes during the backup can be avoided and data consistency can be ensured; after the data file is written, an index file is built according to the metadata of the data file, and when the index file is built, a snapshot backup operation is performed for backup, which can ensure the consistency and integrity of the backup data and improve the backup efficiency and reliability; after the backup is completed, normal data processing is resumed, and the data recovery process does not need to rebuild the index, which can greatly shorten the data recovery time and improve the timeliness of data recovery, thereby ensuring business continuity.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0159] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data backup and recovery method, characterized in that: The steps include: Monitoring the change events of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database; receiving a backup request, encapsulating the backup request into a backup request event, and writing the backup request event into the backup key-value pair, triggering the change event; Obtain backup data from the message middleware through the data node to form a data file, and call the data write instruction to write the data file to a preset storage location, while pausing the consumption of messages from the message middleware; After the data file is written, the metadata of the data file is reported through the data node, and the metadata is written into a specified set of the key-value database; Generate an index request according to the specified set, and construct an index file for the specified set based on the index request through an index node; Monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built; Based on the snapshot backup operation, the data file, the index file and the key-value pairs of the specified set are backed up in sequence to obtain a backup file; After the snapshot backup is completed, a backup completion event is written to the key-value database, and the data node is notified to resume consuming messages from the message middleware.

2. The data backup and recovery method according to claim 1, characterized in that: The step of monitoring the change event of the backup key-value pair in the key-value database through the monitoring mechanism of the key-value database comprises: Using a monitoring mechanism to monitor the key-value database in real time, capturing the change operation of the backup status field of the backup key-value pair in the key-value database; When a change operation is detected in the backup status field, a change event is triggered and the changed backup status information is recorded.

3. The data backup and recovery method according to claim 1, characterized in that: The step of encapsulating the backup request into a backup request event and writing the backup request event into a backup key-value pair of a key-value database comprises: Extract key information from the backup request, and construct a backup request object according to the key information; Converting the backup request object into a standardized event object according to a predefined event type and attribute template; Reading predefined connection information of the key-value database from a preset configuration file, and establishing a connection between the event object and the key-value database according to the connection information; Generate a unique key for storing the event according to the attributes of the event object, and serialize the event object into a preset data format to obtain a serialized event object as a value of the key-value pair; The write interface of the key-value database is called based on the connection to write the unique key and the serialized event object into the backup key-value pair.

4. The data backup and recovery method according to claim 1, characterized in that: The step of obtaining backup data from the message middleware through the data node to form a data file includes: Determine whether there is currently an ongoing data operation through the data node; When there is no ongoing data operation, all data in the message middleware is obtained as backup data according to a preset data persistence strategy, and a data file is formed.

5. The data backup and recovery method according to claim 1, characterized in that: The step of constructing an index file for the specified collection based on the index request through the index node includes: Analyze the index information of the index request to obtain the target index node and index content to be indexed; An index building instruction is called through the target index node, an index is built according to the index content, and an index file is obtained.

6. The data backup and recovery method according to claim 1, characterized in that: After the step of sequentially backing up the data file, the index file, and the key-value pairs of the specified set based on the snapshot backup operation to obtain the backup file, the following step further includes: Using a verification algorithm to verify the backup file to obtain a verification result; When the verification result is that the verification is passed, the backup thread is called to record the metadata information of the backup file, and the metadata information of the backup file is written into a preset backup metadata management library.

7. The data backup and recovery method according to claim 1, characterized in that: After the step of notifying the data node to resume consuming the message of the message middleware, the method further includes: Compare the timestamp in the key-value database with the timestamp in the message middleware to determine the start consumption position; Processing data from the start consumption position through the data node, and updating the processed data into a key-value database; During the data update process, all key operations are recorded through the log system.

8. A data backup and recovery device, characterized in that: include: An event monitoring module, used for monitoring the change events of the backup key-value pairs in the key-value database through the monitoring mechanism of the key-value database; An event writing module, used for receiving a backup request, encapsulating the backup request into a backup request event, and writing the backup request event into the backup key-value pair to trigger the change event; A data writing module is used to obtain backup data from the message middleware through a data node to form a data file, and to call a data writing instruction to write the data file to a preset storage location, while suspending consumption of messages from the message middleware; A reporting module, configured to report metadata of the data file through the data node after the data file is written, and write the metadata into a specified set of the key-value database; An index building module, used to generate an index request according to the specified set, and build an index file for the specified set based on the index request through an index node; A status monitoring module is used to monitor the index building status of the index node by cyclically calling the application program interface, determine whether the index file is built according to the index building status, and call the backup thread to perform a snapshot backup operation when the index file is built; A backup module, used to sequentially back up the data file, the index file and the key-value pairs of the specified set based on the snapshot backup operation to obtain a backup file; The recovery module is used to write a backup completion event to the key-value database after the snapshot backup is completed, and to notify the data node to resume consuming the messages of the message middleware.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data backup and recovery method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data backup and recovery method according to any one of claims 1 to 7 are implemented.

Images