Computing power resource metadata storage method and device of intelligent computing center

Abstract

The invention provides a computing power resource metadata storage method and device of an intelligent computing center, and relates to the technical field of computing power infrastructure, and the method comprises the following steps: S1, sending a total backup request for target data to a coordination management module; s2, receiving a notification of executing a backup task sent by the coordination management module according to the full-amount backup request; s3, obtaining the target data from the coordination management module according to the backup task; s4, backup meta-information is generated according to the target data, the backup meta-information comprises at least one level of data information file of the target data, and digital fingerprint information recording a subordinate file is set in a superior file of each level of data information file; and S5, sending the backup meta-information to a storage module. According to the method and the device, the digital fingerprint information is set during backup, so that any subsequent file tampering can be found, and the file integrity of the whole directory is ensured.

Description

Technical Field

[0001] The present invention relates to the technical fields of intelligent computing centers, intelligent computing centers and computing power infrastructure technologies, and particularly relates to a method and device for storing metadata of computing power resources in an intelligent computing center. Background Art

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "intelligent computing centers" have emerged as the times require.

[0003] An "intelligent computing center" refers to a facility that uses large-scale heterogeneous computing power resources, including general computing power and intelligent computing power, and mainly provides the required computing power, data and algorithms for artificial intelligence applications (such as scenarios of artificial intelligence deep learning model development, model training and model inference, etc.). The intelligent computing center covers facilities, hardware, software, and can provide full-stack capabilities from underlying computing power to top-level application enabling.

[0004] The "intelligent computing center" includes, but is not limited to, the "intelligent computing center".

[0005] An "intelligent computing center", that is, an artificial intelligence computing center, is a type of computing power infrastructure that is based on artificial intelligence theory, adopts an artificial intelligence computing architecture, and provides computing power services, data services and algorithm services required for artificial intelligence applications.

[0006] "Computing power" is the core of "intelligent computing centers" and "intelligent computing centers". It is the ability of computer devices or computing / data centers to process information, the ability of computer hardware and software to cooperate to jointly execute a certain computing requirement, the computing ability to achieve the output of target results through processing information data, and a new type of productive force that integrates information computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] "Metadata of computing power resources" is a set of data used to describe and identify computing power resources. It is an identity card that details various attributes of this machine, this cluster, or this cloud instance, facilitating the management, scheduling and use of computing power resources.

[0008] For storing metadata of computing power resources, the corresponding disk file is very important. The disk file includes directories and files, and the directories are used for full backup and recovery. Existing backup and recovery of metadata of computing power resources do not rely on intelligent computing centers and computing power, and since the backup and recovery are not in the same cluster and may involve multiple operation and maintenance personnel, problems such as loss and tampering may occur during the process of file migration. Summary of the Invention

[0009] The present invention provides a method and device for storing computing power resource metadata of an intelligent computing center to solve the problems of loss and tampering of files during the migration of backup and recovery files of an existing database.

[0010] To solve the above technical problems, the present invention is implemented as follows:

[0011] In a first aspect, the present invention provides a method for storing computing power resource metadata of an intelligent computing center, which is applied to a backup / recovery module and includes:

[0012] Step S1: Send a full backup request for target data to a coordination management module;

[0013] Step S2: Receive a notification of executing a backup task sent by the coordination management module according to the full backup request;

[0014] Step S3: Obtain the target data from the coordination management module according to the backup task;

[0015] Step S4: Generate backup meta-information according to the target data, where the backup meta-information includes at least one-level data information files of the target data, and digital fingerprint information for recording lower-level files is set in the upper-level file of each level of the data information files;

[0016] Step S5: Send the backup meta-information to a storage module.

[0017] Optionally, after step S5, the method further includes:

[0018] Step S6: Send a full recovery request for the target data to the coordination management module;

[0019] Step S7: Receive a notification of executing a recovery task sent by the coordination management module according to the full recovery request;

[0020] Step S8: Obtain the backup meta-information in the storage module according to the recovery task;

[0021] Step S9: Perform step-by-step verification on the data information files of the backup meta-information; if the verification of a certain level of the data information files of the backup meta-information fails, stop the full recovery of the target data; if the verification of each level of the data information files of the backup meta-information is successful, complete the full recovery of the target data.

[0022] Optionally, the data information files of the backup meta-information further include at least one of the following: data column family, region identity identifier, backup path, file size, file name.

[0023] Optionally, step S9 includes:

[0024] Step S91: Gradually determine whether the data information file of the backup meta-information has changed. If the data information file of the backup meta-information at a certain level has changed, the verification fails; if the data information files of the backup meta-information at each level have not changed, the verification succeeds.

[0025] In a second aspect, the present invention provides a method for storing computing power resource metadata of an intelligent computing center, which is applied to a coordination management module and includes:

[0026] Step S1': Receive a full backup request sent by the backup / restore module;

[0027] Step S2': Determine whether the full backup request meets the preset backup requirements. If the full backup request meets the preset backup requirements, register a backup task according to the full backup request;

[0028] Step S3': Notify the backup / restore module to execute the backup task. The backup / restore module obtains the target data from the coordination management module according to the backup task; generates backup meta-information according to the target data, where the backup meta-information includes data information files at least at one level of the target data, and digital fingerprint information for recording lower-level files is set in the upper-level file of each level of the data information files; send the backup meta-information to the storage module.

[0029] Optionally, after step S3', it further includes:

[0030] Step S4': Receive a full restore request sent by the backup / restore module;

[0031] Step S5': Determine whether the full restore request meets the preset restore requirements. If the full backup request meets the preset restore requirements, register a restore task according to the full restore request;

[0032] Step S6': Notify the backup / restore module to execute the restore task. The backup / restore module obtains the backup meta-information in the storage module according to the restore task; perform hierarchical verification on the data information files of the backup meta-information; if the verification fails for a certain level of the data information files of the backup meta-information, stop the full restore of the target data; if the verification is successful for each level of the data information files of the backup meta-information, complete the full restore of the target data.

[0033] Optionally, the preset backup requirements include at least one of the following: there is no running backup task; the version and parameters of the target data meet the requirements; the backup time point of the current full backup request meets the preset backup safety point; there is no task for garbage collection in progress.

[0034] Optionally, the preset recovery requirements include at least one of the following: there is no running recovery task; the version and parameters of the target data meet the requirements; whether the storage location to be recovered currently is an empty library; there is no task for garbage collection in progress.

[0035] In a third aspect, the present invention provides a backup / recovery module, including:

[0036] A first sending module, configured to send a full backup request for target data to a coordination management module;

[0037] A first receiving module, configured to receive a notification of executing a backup task sent by the coordination management module according to the full backup request;

[0038] A first obtaining module, configured to obtain the target data from the coordination management module according to the backup task;

[0039] A first processing module, configured to generate backup meta-information according to the target data, where the backup meta-information includes data information files of at least one level of the target data, and digital fingerprint information for recording lower-level files is set in the upper-level file of each level of the data information files;

[0040] A second sending module, configured to send the backup meta-information to a storage module.

[0041] Optionally, it further includes:

[0042] A third sending module, configured to send a full recovery request for the target data to the coordination management module;

[0043] A second receiving module, configured to receive a notification of executing a recovery task sent by the coordination management module according to the full recovery request;

[0044] A second obtaining module, configured to obtain the backup meta-information in the storage module according to the recovery task;

[0045] A second processing module, configured to perform step-by-step verification on the data information files of the backup meta-information; if the verification of a certain level of the data information files of the backup meta-information fails, stop the full recovery of the target data; if the verification of each level of the data information files of the backup meta-information is successful, complete the full recovery of the target data.

[0046] Optionally, the data information file of the backup meta-information further includes at least one of the following: data column family, region identity identifier, backup path, file size, and file name.

[0047] Optionally, the second processing module includes:

[0048] A processing sub-module, configured to sequentially determine whether the data information file of the backup meta-information has changed. If the data information file of the backup meta-information at a certain level has changed, the verification fails; if the data information file of the backup meta-information at each level has not changed, the verification is successful.

[0049] In a fourth aspect, the present invention provides a coordination management module, including:

[0050] A third receiving module, configured to receive a full backup request sent by the backup / recovery module;

[0051] A third processing module, configured to determine whether the full backup request meets a preset backup requirement. If the full backup request meets the preset backup requirement, register a backup task according to the full backup request;

[0052] A first notification module, configured to notify the backup / recovery module to execute the backup task. The backup / recovery module obtains the target data from the coordination management module according to the backup task; generates backup meta-information according to the target data, where the backup meta-information includes data information files at at least one level of the target data, and digital fingerprint information for recording lower-level files is set in the upper-level file of each level of the data information file; and sends the backup meta-information to the storage module.

[0053] Optionally, it further includes:

[0054] A fourth receiving module, configured to receive a full recovery request sent by the backup / recovery module;

[0055] A fourth processing module, configured to determine whether the full recovery request meets a preset recovery requirement. If the full backup request meets the preset recovery requirement, register a recovery task according to the full recovery request;

[0056] A second notification module, configured to notify the backup / recovery module to execute the recovery task. The backup / recovery module obtains the backup meta-information in the storage module according to the recovery task; performs sequential verification on the data information files of the backup meta-information; if the verification fails for a certain level of the data information files of the backup meta-information, stop the full recovery of the target data; if the verification is successful for each level of the data information files of the backup meta-information, complete the full recovery of the target data.

[0057] Optionally, the preset backup requirements include at least one of the following: there is no running backup task; the version and parameters of the target data meet the requirements; the backup time point of the current full backup request meets the preset backup safety point; there is no task for garbage collection in progress.

[0058] Optionally, the preset recovery requirements include at least one of the following: there is no running recovery task; the version and parameters of the target data meet the requirements; whether the current storage location to be recovered is an empty library; there is no task for garbage collection in progress.

[0059] In a fifth aspect, the present invention provides an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements the method for storing metadata of computing power resources of the intelligent computing center as described in any one of the first aspects, or the steps in the method for storing metadata of computing power resources of the intelligent computing center as described in any one of the second aspects.

[0060] In a sixth aspect, the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements the method for storing metadata of computing power resources of the intelligent computing center as described in any one of the first aspects, or the steps in the method for storing metadata of computing power resources of the intelligent computing center as described in any one of the second aspects.

[0061] In a seventh aspect, the present invention provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, it implements the method for storing metadata of computing power resources of the intelligent computing center as described in any one of the first aspects, or the steps in the method for storing metadata of computing power resources of the intelligent computing center as described in any one of the second aspects.

[0062] In the present invention, a full backup request for target data is sent to the coordination management module; a notification for executing a backup task sent by the coordination management module according to the full backup request is received; the target data is obtained from the coordination management module according to the backup task; backup meta-information is generated according to the target data, where the backup meta-information includes at least one-level data information files of the target data, and digital fingerprint information for recording the lower-level files is set in the upper-level file of each level of the data information files; the backup meta-information is sent to the storage module. By setting digital fingerprint information during the backup of the metadata of computing power resources in the intelligent computing center, any subsequent file tampering can be detected, ensuring the integrity of the files in the entire directory, and solving the problems of loss and tampering of the files during the backup and recovery of the existing database. Description of the Drawings

[0063] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiments. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0064] Figure 1 is a flowchart of the application of the computing power resource metadata storage method of an intelligent computing center provided by the present invention to a backup / recovery module;

[0065] Figure 2 is a data backup flowchart of the computing power resource metadata storage method of an intelligent computing center provided by the present invention;

[0066] Figure 3 is a schematic diagram of the disk file directory structure of the computing power resource metadata storage method of an intelligent computing center provided by the present invention;

[0067] Figure 4 is a schematic diagram of the disk file directory structure of another computing power resource metadata storage method of an intelligent computing center provided by the present invention;

[0068] Figure 5 is a data recovery flowchart of the computing power resource metadata storage method of an intelligent computing center provided by the present invention;

[0069] Figure 6 is a flowchart of the application of the computing power resource metadata storage method of an intelligent computing center provided by the present invention to a coordination management module;

[0070] Figure 7 is a schematic diagram of the structure of a backup / recovery module provided by the present invention;

[0071] Figure 8 is a schematic diagram of the structure of a coordination management module provided by the present invention;

[0072] Figure 9 is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Embodiments

[0073] The technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the protection scope of the present invention.

[0074] First, the technical terms related to the present invention will be briefly described below.

[0075] The "computing power" described in the present invention refers to: the ability of computer devices or computing / data centers to process information, which is the ability of computer hardware and software to cooperate to jointly execute a certain computing requirement, the computing ability to achieve the output of the target result by processing information data, and a new type of productive force integrating information computing power, network carrying capacity, and data storage capacity, which mainly provides services to society through the computing power infrastructure.

[0076] The "computational power" (Computational Power, CP) described in the present invention refers to: the ability of the data center server to process data and achieve result output, which is a comprehensive index to measure the computing ability of the data center, including general computing ability, supercomputing ability, and intelligent computing ability. The commonly used measurement unit is the number of floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), and the larger the value, the stronger the comprehensive computing ability. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe 2A or 500,000 mainstream server CPUs or 2 million mainstream laptops. The calculation formula is: CP = CP_general + CP_intelligent + CP_super.

[0077] The "carrying capacity" (Network Power, NP) described in the present invention refers to: the performance of the data transmission ability of the computing power facility, which is a comprehensive ability including network architecture, network bandwidth, transmission delay, intelligent management and scheduling, etc., involving network transmission inside and between data centers, and is a comprehensive index to measure the network transmission scheduling ability.

[0078] The "storage power" (Storage Power, SP) described in the present invention refers to: the comprehensive ability of the data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon, which is a comprehensive index to measure the data storage ability of the data center, including external storage devices such as storage arrays and server internal storage devices. The commonly used measurement unit for storage capacity is exabyte (EB, 1 EB = 2^60 bytes), the commonly used measurement unit for performance is the number of read and write operations per second per unit capacity (IOPS / TB, Input / Output Operations Per Second / TB), and the disaster recovery ratio is an important manifestation of security and reliability.

[0079] The "computing power infrastructure" described in the present invention refers to: a new type of information infrastructure integrating information computing power, network carrying capacity, and data storage capacity, which can realize the centralized computing, storage, transmission, and application of information, presenting characteristics such as multi-element ubiquitous, intelligent and agile, secure and reliable, and green and low-carbon, and is of great significance for boosting industrial transformation and upgrading, empowering China's scientific and technological innovation, meeting people's beautiful life, and realizing efficient social governance.

[0080] The "new information infrastructure" described in the present invention refers to: mainly including network infrastructures such as 5G networks, fiber broadband networks, backbone networks, international communication networks, satellite Internet, etc., computing power infrastructures such as data centers, general computing power centers, intelligent computing centers, supercomputing centers, etc., and new technology facilities such as artificial intelligence, blockchain, and quantum computing. With the emergence and popularization of new general technologies, the form of the new information infrastructure will be more diverse.

[0081] The "computing power" described in the present invention includes: general computing power, intelligent computing power, and super computing power.

[0082] The "general computing power" described in the present invention refers to: the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0083] The "intelligent computing power" described in the present invention refers to: for various artificial intelligence innovation applications, a computing platform is deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit), such as natural language processing, machine vision, etc.

[0084] The "super computing power" described in the present invention refers to: mainly the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and processes extremely complex or data-intensive problems through a dedicated operating system. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, gene analysis, etc.

[0085] The "intelligent computing center" described in the present invention refers to: a facility that provides the required computing power, data, and algorithms mainly for artificial intelligence applications (such as scenarios like artificial intelligence deep learning model development, model training, and model inference) by using large-scale heterogeneous computing power resources, including general computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center covers facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enabling.

[0086] The "intelligent computing center" described in the present invention includes, but is not limited to, the "intelligent computing center".

[0087] The "intelligent computing center" described in the present invention, that is, the artificial intelligence computing center, is a type of computing power infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0088] The "computing power center" described in the present invention refers to: a facility mainly composed of infrastructure such as wind, fire, water, and electricity and IT software and hardware devices, with computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0089] The "supercomputing center" described in the present invention refers to: a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters, capable of providing functions such as large-scale computing, storage, and network services, and is widely used in application scenarios such as aerospace, national defense, oil exploration, climate modeling, and genome sequencing.

[0090] The "computing power resources" described in the present invention refer to: technologies and facilities with information computing, transmission, storage, and application capabilities required for the development of the digital society, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and support and guarantee resources such as wind, fire, water, and electricity.

[0091] The "model" described in the present invention includes but is not limited to "large language models" and "multimodal large models".

[0092] The "large language model" described in the present invention refers to a large language model (LLM), which is a language model with a relatively large number of parameters, aiming to understand and generate human language, trained with a large amount of text data, and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.

[0093] The "multimodal large model" (Multimodal Large Models) described in the present invention refers to: a model trained by jointly combining multimodal information such as text, images, videos, and audio, including but not limited to multimodal large language models.

[0094] The "computing power resource metadata" described in the present invention refers to a set of data used to describe and identify computing power resources, which is an ID card, recording in detail various attributes of this machine, this cluster, or this cloud instance, facilitating the management, scheduling, and use of computing power resources.

[0095] The "meta-information" described in the present invention refers to the information used to describe and manage the computing principle metadata of computing resources. The meta-information usually includes attributes, status, configuration, performance indicators, etc. of the resources, which helps in the monitoring, management, and optimization of the resources.

[0096] Please refer to Figure 1 , the present invention provides a method for storing computing power resource metadata of an intelligent computing center, which is applied to the backup / restore module and includes:

[0097] Step S1: Send a full backup request for the target data to the coordination management module;

[0098] In the present invention, the full backup is to back up the full amount of data at a certain point in time of the cluster, including tenants, databases, tables, and region databases. As Figure 2 shown, the backup / restore module interacts with the coordination management module, initiates a full backup request, and indicates the timestamp and storage address of the backup. The coordination management module will determine whether the full backup request meets the preset backup requirements. If the full backup request meets the preset backup requirements, a backup task will be registered according to the full backup request, and the backup / restore module will be notified to execute the backup task; if not, an error will be reported and the process will exit. By checking the parameters in advance and identifying potential problems, the possibility of errors and failures during the backup process can be reduced, the success rate of the full backup can be increased, time and computing resources can be saved, and data security can be improved.

[0099] Step S2: Receive the notification from the coordination management module to execute the backup task according to the full backup request;

[0100] In the present invention, the coordination management module will determine whether the full backup request meets the preset backup requirements. If the full backup request meets the preset backup requirements, a backup task will be registered according to the full backup request, and the backup / restore module will be notified to execute the backup task;

[0101] Optionally, the preset backup requirements include at least one of the following: there is no running backup task; the version and parameters of the target data meet the requirements; the backup time point of the current full backup request meets the preset backup safety point; there is no task for garbage collection in progress.

[0102] If there is a task for garbage collection in progress, it will be stopped. At the same time, before executing the backup task, the dingo_store Balance leader&Region and Region split&merge operations will also be paused. After the backup is completed, it will be restored to the original configuration. When the backup and garbage collection are carried out simultaneously, additional complexity and potential errors may be introduced. Stopping the garbage collection task and the above tasks before performing the full backup can ensure the smooth progress of the backup process, improve data consistency and integrity, optimize system performance, and reduce the risk of problems during recovery.

[0103] Step S3: Obtain the target data from the coordination management module according to the backup task;

[0104] In the present invention, as Figure 2As shown, the backup / recovery module calls the coordination management module to obtain the target data, that is, all data region information, and reasonably allocates it to the normal table module (store), the vector table module (index), and the file index module (document), and ensures balanced distribution. Each module scans its own region to obtain the specified key and value, thereby generating an sst (Sorted String Table) file and sending this sst file to the storage module, making the overall log format relatively simple.

[0105] Step S4: Generate backup meta-information according to the target data. Among them, the backup meta-information includes at least one-level data information files of the target data, and digital fingerprint information for recording lower-level files is set in the upper-level files of each level of the data information files.

[0106] In the present invention, when the backup / recovery module performs backups on the normal table module (store), the vector table module (index), and the file index module (document), backup meta-information is synchronously generated, that is, the disk file of the computing power resource metadata.

[0107] In the present invention, the directory structure of the disk file is as Figure 3 and Figure 4 shown, where backupmeta.encryption is the fingerprint information recorded by the top-level file backupmeta; backup.lock is the locking mark file during backup; backupmeta.debug is the debug file.

[0108] In the present invention, optionally, the data information files of the backup meta-information further include at least one of the following: data column family, region identity (ID), backup path, file size, file name.

[0109] In the present invention, for the directory structure, specifically, for example: in the file "80086_1-1_2a0edcc0162e5beeeb95f6c33872438486d77651_1737510054_write.sst", the index_cf_sst_meta_sdk_data.sst or index_cf_sst_meta_sql_data.sst files are recorded, and the recorded structure is:

[0110]

[0111]

[0112] For another example: "store_cf_sst_meta_sdk_data.sst" is managed by this backupmeta.datafile, and the recorded structure is as follows:

[0113]

[0114] In the present invention, digital fingerprint information of the lower-level file is set in the upper-level file of each level of the data information file of the backup meta-information; the digital fingerprint information can be, but is not limited to, set as a digital signature or a hash value. By this digital fingerprint information, it can be verified whether the backup data is tampered with or damaged during storage and transmission, ensuring the integrity of the data, ensuring that the backup data is generated by an authorized user or system, and preventing the introduction of forged or malicious data; when restoring data subsequently, the digital fingerprint information can help quickly locate and verify the required backup files, reducing the restoration time; and by generating unique digital fingerprint information for each backup meta-information, different versions of backup data can be easily managed and tracked, facilitating the backtracking and restoration of historical data, and can identify and avoid duplicate backup of the same data, thereby saving storage space and backup time.

[0115] Step S5: Send the backup meta-information to the storage module.

[0116] In the present invention, the backup / recovery module calls the coordination management module to send the backup meta-information to the storage module for use during subsequent data restoration; and resumes the operation of various tasks paused during the execution of the backup task.

[0117] In the present invention, a full backup request for the target data is sent to the coordination management module; a notification of executing the backup task sent by the coordination management module according to the full backup request is received; the target data is obtained from the coordination management module according to the backup task; backup meta-information is generated according to the target data, wherein the backup meta-information includes at least one level of data information files of the target data, and digital fingerprint information of the lower-level file is set in the upper-level file of each level of the data information file; the backup meta-information is sent to the storage module. By setting digital fingerprint information during backup, any subsequent file tampering can be detected, ensuring the integrity of the files in the entire directory, and solving the problems of loss and tampering of the backup and recovery files of the existing database during the moving process.

[0118] In the present invention, optionally, after step S5, it further includes:

[0119] Step S6: Send a full recovery request for the target data to the coordination management module;

[0120] In the present invention, full recovery refers to restoring the backed-up disk files to a brand-new empty library up to the directory of the playback data. For example, Figure 5 As shown, the backup / recovery module interacts with the coordination management module to initiate a full recovery request. The coordination management module will determine whether the full recovery request meets the preset recovery requirements. If the full backup request meets the preset recovery requirements, a recovery task will be registered according to the full recovery request; the backup / recovery module will be notified to execute the backup task; if not, an error will be reported and the process will exit. By checking parameters in advance and identifying potential problems, the possibility of errors and failures during the backup process can be reduced, the success rate of full backup can be increased, time and computing resources can be saved, and data security can be improved.

[0121] Step S7: Receive the notification from the coordination management module to execute the recovery task according to the full recovery request;

[0122] In the present invention, the coordination management module will determine whether the full recovery request meets the preset recovery requirements. If the full backup request meets the preset recovery requirements, a recovery task will be registered according to the full recovery request, and the backup / recovery module will be notified to execute the recovery task;

[0123] In the present invention, optionally, the preset recovery requirements include at least one of the following: there is no running recovery task; the version and parameters of the target data meet the requirements; whether the current storage location to be recovered is an empty library; there is no task for garbage collection in progress.

[0124] The full data recovery requirement is empty library import. If there is a task for garbage collection in progress, it will be stopped. During full recovery, it is crucial to ensure that critical data is not affected by the garbage collection mechanism. Stopping garbage collection can ensure the integrity of important data during recovery, improve data consistency and integrity, and optimize system performance.

[0125] Step S8: Obtain the backup meta-information in the storage module according to the recovery task;

[0126] In the present invention, for example, Figure 5 As shown, the backup / recovery module calls the coordination management module to obtain the backup meta-information in the storage module, and reasonably distributes it to the ordinary table module (store), vector table module (index), and file index module (document), and ensures balanced distribution. Each module recovers its own region and reports the recovery progress to the coordination management module for the backup / recovery module to call at any time.

[0127] Step S9: Gradually verify the data information file of the backup meta - information; if the verification fails for a certain level of the data information file of the backup meta - information, stop the full - volume recovery of the target data; if the verification is successful for each level of the data information file of the backup meta - information, complete the full - volume recovery of the target data.

[0128] In the present invention, optionally, the step S9 includes:

[0129] Step S91: Gradually determine whether the data information file of the backup meta - information has changed. If the data information file of the backup meta - information at a certain level has changed, the verification fails; if the data information file of the backup meta - information at each level has not changed, the verification is successful.

[0130] In the present invention, when the backup / recovery module performs backups in the general table module (store), the vector table module (index), and the file index module (document), it synchronously and gradually verifies the data information file of the backup meta - information. As Figure 3 and Figure 4 shown in the directory structure of the backup meta - information, by setting the digital fingerprint information of the lower - level file in the upper - level file of each level of the data information file of the backup meta - information and gradually verifying. If the verification fails for a certain level of the data information file of the backup meta - information, it means that the information in the sub - directory at this level has been tampered with or deleted, and the data is no longer trustworthy. Therefore, stop the full - volume recovery of the target data; if the verification is successful for each level of the data information file of the backup meta - information, it means that no data has been tampered with or deleted, and the full - volume recovery of the target data can be completed.

[0131] In the present invention, the digital fingerprint information can be but is not limited to being set as a digital signature or a hash value. By using this digital fingerprint information, verify whether the backup data has been tampered with or damaged during storage and transmission, ensure the integrity of the data, ensure that the backup data is generated by an authorized user or system, and prevent the introduction of forged or malicious data; when recovering data subsequently, the digital fingerprint information can help quickly locate and verify the required backup files, reducing the recovery time.

[0132] Please refer to Figure 6 , the present invention provides a method for storing computing power resource metadata in an intelligent computing center, which is applied to a coordination and management module and includes:

[0133] Step S1’: Receive a full - volume backup request sent by the backup / recovery module;

[0134] In the present invention, the full backup is to back up the full amount of data at a certain time point of the cluster, including tenants, databases, tables, and region databases. For example, Figure 2 As shown, the coordination management module interacts with the backup / recovery module, receives the full backup request sent by the backup / recovery module, and receives the timestamp and storage address of the backup, so as to subsequently determine whether the full backup request meets the preset backup requirements.

[0135] Step S2': Determine whether the full backup request meets the preset backup requirements. If the full backup request meets the preset backup requirements, register a backup task according to the full backup request;

[0136] In the present invention, the coordination management module will determine whether the full backup request meets the preset backup requirements. If the full backup request meets the preset backup requirements, register a backup task according to the full backup request, and notify the backup / recovery module to execute the backup task; if not, report an error and exit. By checking the parameters in advance and identifying potential problems, the possibility of errors and failures during the backup process can be reduced, the success rate of the full backup can be improved, time and computing resources can be saved, and data security can be enhanced.

[0137] In the present invention, optionally, the preset backup requirements include at least one of the following: there is no running backup task; the version and parameters of the target data meet the requirements; the backup time point of the current full backup request meets the preset backup safety point; there is no task for garbage collection in progress.

[0138] Specifically, parameter checking needs to be done first. The user determines whether the current parameters are legal through parameter checking, and checks whether there is another process backing up in the backup directory;

[0139] Check whether the versions of dingo_br and dingo_store are consistent. If they are not consistent, report an error and exit;

[0140] If there is already a task doing a full database backup task currently, report an error and exit;

[0141] Compare the backup timestamp to determine whether the backup time point of the current full backup request meets the preset backup safety point, that is, judge the safety point (safe_point_ts) of all tenants and the safety point (resolve_lock_safe_point_ts) where the locks have been resolved.

[0142] That is, the core comparison algorithm is:

[0143] 1) For a certain tenant, resolve_lock_safe_point_ts is greater than safe_point_ts;

[0144] 2) The timestamp of the backup must be greater than the safe_point_ts of all tenants;

[0145] 3) The timestamp of the backup must be less than or equal to the resolve_lock_safe_point_ts of all tenants;

[0146] If the garbage collection task GC is running and an emergency stop instruction is initiated, the dingo_store Balance leader & Region and Region split & merge operations will also be paused before the backup task is executed. After the backup is completed, it will be restored to the original configuration. When the backup and garbage collection are carried out simultaneously, additional complexity and potential errors may be introduced. Stopping the garbage collection task and the above-mentioned tasks before a full backup can ensure the smooth progress of the backup process, improve data consistency and integrity, optimize system performance, and reduce the risk of problems during recovery.

[0147] Step S3': Notify the backup / restore module to execute the backup task. The backup / restore module obtains the target data from the coordination management module according to the backup task; generates backup meta-information based on the target data, where the backup meta-information includes at least one level of data information files of the target data, and digital fingerprint information for recording the lower-level files is set in the upper-level files of each level of the data information files; and sends the backup meta-information to the storage module.

[0148] In the present invention, the backup / restore module is notified to execute the backup task. The backup / restore module obtains the target data from the coordination management module according to the backup task; generates backup meta-information based on the target data, where the backup meta-information includes at least one level of data information files of the target data, and digital fingerprint information for recording the lower-level files is set in the upper-level files of each level of the data information files; and sends the backup meta-information to the storage module. By setting digital fingerprint information during backup, any subsequent file tampering can be detected, ensuring the integrity of the files in the entire directory, and solving the problems of loss and tampering of the files during the backup and recovery of existing databases.

[0149] Optionally, after step S3', the following steps are further included:

[0150] Step S4': Receive a full recovery request sent by the backup / restore module;

[0151] In the present invention, full recovery means restoring the backup disk files to a brand-new empty database up to the directory where the data is played back, such as Figure 5As shown, the coordination management module interacts with the backup / restore module and receives a full recovery request, so that the coordination management module can subsequently determine whether the full backup request meets the preset backup requirements.

[0152] Step S5': Determine whether the full recovery request meets the preset recovery requirements. If the full backup request meets the preset recovery requirements, register a recovery task according to the full recovery request;

[0153] In the present invention, the coordination management module will determine whether the full recovery request meets the preset recovery requirements. If the full backup request meets the preset recovery requirements, register a recovery task according to the full recovery request, and notify the backup / restore module to execute the recovery task;

[0154] In the present invention, optionally, the preset recovery requirements include at least one of the following: there is no running recovery task; the version and parameters of the target data meet the requirements; whether the current storage location to be recovered is an empty library; there is no task for garbage collection in progress.

[0155] In the present invention, various checks should be performed first when starting the full database recovery task, such as checking whether there is a full database backup or full database recovery in progress, version comparison, and whether the current database is empty. And if there is a task for garbage collection in progress, it should be stopped. When performing a full recovery, it is crucial to ensure that critical data is not affected by the garbage collection mechanism. Stopping garbage collection can ensure the integrity of important data during recovery, improve data consistency and integrity, and optimize system performance.

[0156] Step S6': Notify the backup / restore module to execute the recovery task. The backup / restore module obtains the backup meta-information in the storage module according to the recovery task; perform hierarchical verification on the data information file of the backup meta-information; if the verification of a certain level of the data information file of the backup meta-information fails, stop the full recovery of the target data; if the verification of each level of the data information file of the backup meta-information is successful, complete the full recovery of the target data.

[0157] In the present invention, the coordination management module notifies the backup / restore module to execute the recovery task as Figure 5As shown, the backup / recovery module calls the coordination management module to obtain the backup meta-information in the storage module, and reasonably distributes it to the ordinary table module (store), the vector table module (index), and the file index module (document), and ensures balanced distribution. Each module restores its own region and reports the restoration progress to the coordination management module for the backup / recovery module to call at any time; the backup / recovery module performs step-by-step verification on the data information files of the backup meta-information; if the verification of a certain level of the data information file of the backup meta-information fails, the full restoration of the target data is stopped; if the verification of each level of the data information file of the backup meta-information is successful, the full restoration of the target data is completed. The digital fingerprint information is used to verify whether the backup data has been tampered with or damaged during storage and transmission, ensuring data integrity, ensuring that the backup data is generated by an authorized user or system, and preventing the introduction of forged or malicious data; when restoring data subsequently, the digital fingerprint information can help quickly locate and verify the required backup files, reducing the restoration time.

[0158] Please refer to Figure 7 , the present invention provides a backup / recovery module, including:

[0159] The first sending module 71 is used to send a full backup request for the target data to the coordination management module;

[0160] The first receiving module 72 is used to receive the notice of executing the backup task sent by the coordination management module according to the full backup request;

[0161] The first obtaining module 73 is used to obtain the target data from the coordination management module according to the backup task;

[0162] The first processing module 74 is used to generate backup meta-information according to the target data, wherein the backup meta-information includes data information files of at least one level of the target data, and digital fingerprint information for recording the lower-level files is set in the upper-level files of each level of the data information files;

[0163] The second sending module 75 is used to send the backup meta-information to the storage module.

[0164] In the present invention, optionally, it further includes:

[0165] The third sending module is used to send a full restoration request for the target data to the coordination management module;

[0166] The second receiving module is used to receive the notice of executing the restoration task sent by the coordination management module according to the full restoration request;

[0167] A second acquisition module, configured to acquire the backup meta-information in the storage module according to the restoration task;

[0168] A second processing module, configured to perform level-by-level verification on the data information file of the backup meta-information; if the verification at a certain level of the data information file of the backup meta-information fails, stop the full restoration of the target data; if the verification at each level of the data information file of the backup meta-information is successful, complete the full restoration of the target data.

[0169] In the present invention, optionally, the data information file of the backup meta-information further includes at least one of the following: a data column family, a region identity identifier, a backup path, a file size, and a file name.

[0170] In the present invention, optionally, the second processing module includes:

[0171] A processing sub-module, configured to determine level by level whether the data information file of the backup meta-information has changed. If the data information file of the backup meta-information at a certain level has changed, the verification fails; if the data information file of the backup meta-information at each level has not changed, the verification is successful.

[0172] The backup / restoration module provided by the present invention can implement Figure 1 each process implemented by the method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein again.

[0173] Please refer to Figure 8 , the present invention provides a coordination management module, including:

[0174] A third receiving module 81, configured to receive a full backup request sent by the backup / restoration module;

[0175] A third processing module 82, configured to determine whether the full backup request meets a preset backup requirement. If the full backup request meets the preset backup requirement, register a backup task according to the full backup request;

[0176] A first notification module 83, configured to notify the backup / restoration module to execute the backup task. The backup / restoration module acquires the target data from the coordination management module according to the backup task; generates backup meta-information according to the target data, where the backup meta-information includes data information files at at least one level of the target data, and digital fingerprint information for recording lower-level files is set in the upper-level file of each level of the data information file; and sends the backup meta-information to the storage module.

[0177] In the present invention, optionally, it further includes:

[0178] The fourth receiving module is configured to receive the full - volume recovery request sent by the backup / recovery module;

[0179] The fourth processing module is configured to determine whether the full - volume recovery request meets the preset recovery requirements. If the full - volume backup request meets the preset recovery requirements, a recovery task is registered according to the full - volume recovery request;

[0180] The second notification module is configured to notify the backup / recovery module to execute the recovery task. The backup / recovery module obtains the backup meta - information in the storage module according to the recovery task; performs step - by - step verification on the data information file of the backup meta - information; if the verification of a certain level of the data information file of the backup meta - information fails, the full - volume recovery of the target data is stopped; if the verification of each level of the data information file of the backup meta - information is successful, the full - volume recovery of the target data is completed.

[0181] In the present invention, optionally, the preset backup requirements include at least one of the following: there is no running backup task; the version and parameters of the target data meet the requirements; the backup time point of the current full - volume backup request meets the preset backup safety point; there is no task of performing garbage collection.

[0182] In the present invention, optionally, the preset recovery requirements include at least one of the following: there is no running recovery task; the version and parameters of the target data meet the requirements; whether the current storage location to be recovered is an empty library; there is no task of performing garbage collection.

[0183] The coordination management module provided by the present invention can implement Figure 6 the various processes implemented by the method embodiments and achieve the same technical effects. To avoid repetition, they are not described here again.

[0184] The present invention provides an electronic device 90. Refer to Figure 9 as shown, Figure 9 which is a schematic block diagram of the electronic device 90 of the present invention, including a processor 91, a memory 92, and a program or instruction stored on the memory 92 and executable on the processor 91. When the program or instruction is executed by the processor, the steps in any one of the methods for storing the computing power resource metadata of the intelligent computing center of the present invention are implemented.

[0185] The present invention provides a readable storage medium with a program or instruction stored thereon. When the program or instruction is executed by the processor, the various processes of the embodiments of the method for storing the computing power resource metadata of the intelligent computing center as described above are implemented, and the same technical effects can be achieved. To avoid repetition, they are not described here again.

[0186] The embodiments of the present application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the above-mentioned Figure 1 or Figure 6 each process of the method embodiment shown, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0187] Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0188] It should be noted that in the technical solutions of the present disclosure, in terms of the collection, acquisition, update, analysis, processing, use, transmission, storage, etc. of user personal information, they all comply with the provisions of relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and to safeguard the security of user personal information and network security.

[0189] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without further limitations, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0190] The above serial numbers of the present invention are only for description and do not represent the superiority or inferiority of the embodiments.

[0191] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a service classification device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0192] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for storing computing resource metadata of an intelligent computing center, applied to a backup / recovery module, characterized in that: include: Step S1: Sending a full backup request for target data to the coordination management module; Step S2: receiving a notification of executing a backup task sent by the coordination management module according to the full backup request; Step S3: acquiring the target data from the coordination management module according to the backup task; Step S4: Generate backup meta information according to the target data, wherein the backup meta information includes at least one level of data information files of the target data, and the upper level file of each level of the data information file is set to record the digital fingerprint information of the lower level file; Step S5: Send the backup meta-information to the storage module.

2. The method for storing computing power resource metadata of an intelligent computing center according to claim 1, characterized in that: After step S5, the following steps are also included: Step S6: Sending a full recovery request for the target data to the coordination management module; Step S7: receiving a notification of executing a recovery task sent by the coordination management module according to the full recovery request; Step S8: acquiring the backup meta-information in the storage module according to the recovery task; Step S9: verify the data information files of the backup meta-information level by level; if the verification of a certain level of the data information files of the backup meta-information fails, stop the full recovery of the target data; if the verification of each level of the data information files of the backup meta-information succeeds, complete the full recovery of the target data.

3. The method for storing computing power resource metadata of an intelligent computing center according to claim 1, characterized in that: The data information file of the backup meta-information also includes at least one of the following: data column cluster, area identification, backup path, file size, and file name.

4. The method for storing computing power resource metadata of an intelligent computing center according to claim 3, characterized in that: The step S9 comprises: Step S91: determining whether the data information files of the backup meta-information have changed level by level; if the data information files of the backup meta-information at a certain level have changed, the verification fails; if the data information files of the backup meta-information at each level have not changed, the verification succeeds.

5. A method for storing computing power resource metadata of an intelligent computing center, applied to a coordination management module, characterized in that: include: Step S1': receiving a full backup request sent by a backup / restore module; Step S2': determining whether the full backup request meets the preset backup requirement, and if the full backup request meets the preset backup requirement, registering a backup task according to the full backup request; Step S3': informing the backup / restore module to execute the backup task, and the backup / restore module obtains target data from the coordination management module according to the backup task; Generate backup meta-information according to the target data, wherein the backup meta-information includes at least one level of data information files of the target data, and the upper level file of each level of the data information file is set with digital fingerprint information of the lower level files; send the backup meta-information to the storage module.

6. The method for storing computing power resource metadata of an intelligent computing center according to claim 5, characterized in that: The step S3' further includes: Step S4': receiving a full restore request sent by the backup / restore module; Step S5': determining whether the full restore request meets the preset restore requirement, and if the full backup request meets the preset restore requirement, registering a restore task according to the full restore request; Step S6': notify the backup / restore module to execute the recovery task, and the backup / restore module obtains the backup meta-information in the storage module according to the recovery task; verifies the data information file of the backup meta-information level by level; if the verification of a certain level of the data information file of the backup meta-information fails, stop the full recovery of the target data; if the verification of each level of the data information file of the backup meta-information succeeds, complete the full recovery of the target data.

7. The method for storing computing resource metadata of an intelligent computing center according to claim 5, characterized in that: The preset backup requirements include at least one of the following: there is no backup task in progress; the version and parameters of the target data meet the requirements; the backup time point of the current full backup request meets the preset backup safety point; there is no garbage collection task in progress.

8. The method for storing computing power resource metadata of an intelligent computing center according to claim 6, characterized in that: The preset recovery requirement includes at least one of the following: there is no recovery task in progress; the version and parameters of the target data meet the requirements; whether the current storage location to be restored is an empty library; and there is no garbage collection task in progress.

9. A backup / restore module, characterized in that: include: A first sending module, used to send a full backup request for target data to the coordination management module; A first receiving module, configured to receive a notification of executing a backup task sent by the coordination management module according to the full backup request; A first acquisition module, used for acquiring the target data from the coordination management module according to the backup task; A first processing module is used to generate backup meta information according to the target data, wherein the backup meta information includes at least one level of data information files of the target data, and the upper level file of each level of the data information file is set with digital fingerprint information of the lower level file; The second sending module is used to send the backup meta-information to the storage module.

10. A coordination management module, characterized in that: include: A third receiving module is used to receive a full backup request sent by the backup / restore module; A third processing module is used to determine whether the full backup request meets the preset backup requirement, and if the full backup request meets the preset backup requirement, register a backup task according to the full backup request; A first notification module, used for notifying the backup / restore module to execute the backup task, and the backup / restore module obtains target data from the coordination management module according to the backup task; Generate backup meta-information according to the target data, wherein the backup meta-information includes at least one level of data information files of the target data, and the upper level file of each level of the data information file is set with digital fingerprint information of the lower level files; send the backup meta-information to the storage module.

11. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, it implements the computing power resource metadata storage method of the intelligent computing center as described in any one of claims 1 to 4, or the steps in the computing power resource metadata storage method of the intelligent computing center as described in any one of claims 5 to 8.

12. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, which, when executed by the processor, implement the computing power resource metadata storage method of the intelligent computing center as described in any one of claims 1 to 4, or the steps in the computing power resource metadata storage method of the intelligent computing center as described in any one of claims 5 to 8.

13. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the computing power resource metadata storage method of an intelligent computing center as described in any one of claims 1 to 4, or the steps in the computing power resource metadata storage method of an intelligent computing center as described in any one of claims 5 to 8.

Images