Directory quota information acquisition method and device based on database backup recovery BR

Abstract

The invention provides a directory quota information acquisition method and device based on database backup recovery BR, and relates to the technical field of data processing, in particular to the field of cloud computing and big data. According to the specific implementation scheme, BR operation is executed on a database so as to export a backup file containing metadata from the database; in the execution process of the BR operation, synchronously analyzing the exported backup file to obtain directory attribute information; and according to the directory attribute information, generating a directory full path, and according to the directory full path, obtaining quota information of each directory.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, in particular to the field of cloud computing and big data, and specifically to a method and device for obtaining directory quota information based on database backup and recovery BR. Background Art

[0002] The concept of quota comes from the local file system. Directory quota is based on the directory granularity, which identifies the total amount of space / inode available and the usage of the corresponding directory. When the user's usage of directory space / inode exceeds the set quota, the system usually refuses the user's subsequent writing or creation to avoid possible resource overruns or shortages.

[0003] There are two ways to count catalog quotas: incremental real-time statistics and scheduled analysis based on the full amount. However, the incremental real-time statistics method requires specific book restrictions and has poor usability, while scheduled analysis of the full amount will lose a certain degree of timeliness. Therefore, a catalog quota acquisition method that takes into account both timeliness and usability is needed. Summary of the invention

[0004] The present disclosure provides a method and device for obtaining directory quota information based on database backup and recovery BR.

[0005] According to one aspect of the present disclosure, a method for obtaining directory quota information based on database backup and recovery BR is provided, comprising:

[0006] Performing a BR operation on a database to export a backup file containing metadata from the database;

[0007] During the execution of the BR operation, the exported backup file is synchronously parsed to obtain directory attribute information;

[0008] A full directory path is generated according to the directory attribute information, and quota information of each directory is obtained according to the full directory path.

[0009] According to another aspect of the present disclosure, a directory quota information acquisition device based on database backup and recovery BR is provided, comprising:

[0010] An export module, used for performing a BR operation on a database to export a backup file containing metadata from the database;

[0011] A parsing module, used for synchronously parsing the exported backup file to obtain directory attribute information during the execution of the BR operation;

[0012] The acquisition module is used to generate a full directory path according to the directory attribute information, and to acquire quota information of each directory according to the full directory path.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method described in the embodiment of the first aspect.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the embodiment of the first aspect.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the steps of the method described in the embodiment of the first aspect are implemented.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 It is a schematic diagram of a method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure;

[0022] Figure 2 It is a schematic diagram of another method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure;

[0023] Figure 3 It is a schematic diagram of another method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure;

[0024] Figure 4 It is a schematic diagram of another method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure;

[0025] Figure 5It is a logic diagram of another method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure;

[0026] Figure 6 It is a block diagram of a directory quota information acquisition device based on database backup and recovery BR provided by an embodiment of the present disclosure;

[0027] Figure 7 A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] Data processing is the collection, storage, retrieval, processing, transformation and transmission of data. It runs through all areas of social production and social life. Its basic purpose is to extract and derive valuable and meaningful data for certain specific people from large amounts of disorganized and difficult to understand data.

[0030] Cloud computing is a type of distributed computing. It refers to breaking down huge data computing programs into countless small programs through the network "cloud". These small programs are then processed and analyzed by a system composed of multiple servers to obtain results and return them to users. It can complete the processing of tens of thousands of data in a very short time (a few seconds), thereby achieving powerful network services.

[0031] Big Data, or massive data, refers to large or complex data sets that are not adequately processed by traditional data processing applications. It can also be defined as large amounts of unstructured or structured data from various sources.

[0032] The present disclosure can be applied to the scenario of obtaining directory quota in a distributed file storage system of massive data. At present, the directory quota function support is incomplete or limited in use, which is manifested in limiting the directory renaming operation, the number of directory quota layers and the number of directory quotas, etc., increasing the complexity of file system design and implementation in an incremental real-time statistical manner, and causing file system performance degradation; performing regular full-volume analysis in an offline manner prolongs the directory quota statistics time, sacrificing timeliness, and is not friendly to delay-sensitive businesses.

[0033] The currently proposed directory quota acquisition method is based on incremental real-time statistics of input and output (IO), which is divided into binary log (binlog) export or IO-based accompanying statistics; based on metadata snapshot Snapshot, files are periodically transferred and parsed, and after outputting text that can be recognized by map-reduce (MR) jobs, directory quota statistics are performed in the form of MR jobs; in this acquisition method, the time spent on parsing snapshot files and MR jobs is difficult to estimate, and the timeliness performance is poor. In addition, a large amount of system resources will be consumed during the transfer or parsing process, and disk space may be exhausted, affecting the overall service stability.

[0034] Figure 1 is a schematic diagram of a method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure; Figure 1 As shown, the method includes:

[0035] S101: Perform a BR operation on a database to export a backup file containing metadata from the database.

[0036] In some implementations, the backup-restore (BR) operation is a process of copying the complete data in the database to another location so that it can be safely restored in the event of hardware, network, disk, etc. failures. Optionally, the master node can initiate a backup request, which includes a BR backup operation, so that the BR operation is performed on the database to obtain a backup file.

[0037] It can be understood that a database is a warehouse that organizes, stores and manages data according to a data structure, and is a collection of related data stored together in a certain organizational manner. Performing a BR operation on a database, that is, backing up and restoring data from the database, thereby exporting a backup file containing metadata from the database. In this embodiment, the BR operation obtains a full backup of the data in the database. Compared with the real-time incremental statistics method, each round of quota statistics can achieve full deviation correction to ensure the accuracy of directory quota acquisition.

[0038] Metadata is data about data. It provides information about the data's attributes, organization, domains, and relationships. It is mainly used to describe data and its environment. Metadata is not just information such as the data's type, name, value, etc., but a group of information used to describe the data. This information reflects certain characteristics of the data.

[0039] S102: During the execution of the BR operation, the exported backup file is synchronously parsed to obtain directory attribute information.

[0040] In some implementations, the exported backup files can be obtained regularly and analyzed synchronously. That is, during the data backup process performed by the BR operation, the exported backup data can be analyzed synchronously to reduce the time consumption of analysis after the backup is completed.

[0041] Optionally, the backup file to be exported can be divided into multiple sub-files, each sub-file is stored in the cloud storage when exported, and the sub-files in the cloud storage are checked regularly. When the sub-file backup is completed, the exported backup sub-file is obtained and parsed to obtain the parsed data, thereby reducing time consumption.

[0042] In some implementations, the parsed data includes attribute information of metadata, and directory-related attribute information is obtained from the attribute information to obtain directory attribute information; in some implementations, pre-configured required directory attributes can also be obtained, and the required directory attribute information can be determined from the parsed data, that is, the parsed data is filtered according to the required directory attributes to obtain the directory attribute information.

[0043] S103, generating a full directory path according to the directory attribute information, and obtaining quota information of each directory according to the full directory path.

[0044] In some implementations, path backtracking can be performed based on directory attribute information to generate the full directory path of each directory; in some implementations, directory attribute information can include information such as each directory index node inode, file type and size; inode is a unique identifier of a file and is used to quickly locate the location of a file or directory.

[0045] In some implementations, when tracing the path, the parent directory of the directory can be obtained, and the directories belonging to the same parent directory can be determined; based on the full path of the parent directory, the full path of the directories belonging to the same parent directory can be obtained; and the full path of the parent directory and the directories belonging to the parent directory can be added to the metadata list.

[0046] It can be understood that the parent directory is the parent directory of a directory, that is, the path is traced back according to the directory attribute information to determine the parent directory of the directory and the directories belonging to the same parent directory. The full path of the parent directory is determined according to the directory attribute information, and then the full path of the directories belonging to the same parent directory is determined according to the full path of the parent directory, so as to reduce the number of times the full path is spliced, and effectively reduce the time spent on path backtracing.

[0047] Furthermore, the quota information corresponding to each directory can be counted. For example, the file volume corresponding to the directory and its subdirectories in different full paths can be obtained, and the file volume of all subdirectories and the file volume of the directory can be accumulated to obtain the file volume of the current directory as the quota information corresponding to the current directory, that is, to obtain the quota information of the directory.

[0048] In this embodiment, a full backup file is obtained by executing a BR operation on the database. Compared with the existing real-time incremental statistics method, each round of directory quota statistics based on the backup file can achieve full correction to ensure the accuracy of the directory quota. During the execution of the BR operation, the exported backup file is synchronously parsed without waiting for the backup operation to be completed, which greatly shortens the time consumption of directory quota statistics. The full path of the directory is traced back according to the parsed target attribute information, and quota information statistics are performed according to the target full path, which reduces resource consumption and improves the accuracy of directory quota information acquisition.

[0049] Figure 2 is a schematic diagram of another method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure; Figure 2 As shown, the method includes:

[0050] S201: Perform a BR operation on a database to export a backup file containing metadata from the database.

[0051] In the embodiments of the present disclosure, the implementation method of step S201 can be implemented by any of the methods in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0052] S202, during the execution of the BR operation, the task execution status of the candidate node is obtained, and based on the task execution status, a plurality of target nodes are scheduled.

[0053] In some implementations, the task execution status of the candidate node refers to the file backup status, and multiple target nodes are scheduled according to the task execution status. For example, when the backup of some files is completed, multiple target nodes are scheduled to parse the exported backup files to improve the parsing efficiency.

[0054] S203, sending a backup parsing request to the target node, and the target node pulls the backup file for parsing.

[0055] Optionally, the backup parsing request can be a request sent by the Master node to the target node to instruct the target node to pull the backup file for parsing; the backup and parsing are performed simultaneously without waiting for the overall backup operation to be completed. Compared with the traditional serial directory quota statistics method based on fast transfer of Snapshot Dump and parsing, the overall time required for directory quota statistics can be greatly shortened.

[0056] In some implementations, the storage information of the backup file can be determined and the backup file can be stored based on the storage information; optionally, the storage information can be carried in the backup request sent by the Master node, and the exported backup file can be stored according to the storage information in the backup request, that is, the exported backup file is stored in the storage address in the storage information.

[0057] After the target node receives the backup resolution request, the target node can obtain the storage information of the backup file from the backup resolution request, that is, the storage information of the backup file storage location included in the backup resolution request, obtain the storage information of the backup file and pull the backup file based on the storage information of the backup file, so as to quickly locate and pull the backup file.

[0058] In some implementations, it is also possible to obtain a data storage standard file in the storage space where the storage information of the backup file is located, the data storage standard file includes the file identifier of the stored backup file; in response to the backup file being pulled, the pulling status of the backup file is marked; that is, the file identifier of the stored backup file is recorded in the data storage standard file, and after the backup file is pulled, the pulling status of the backup file is marked in the data storage standard file to determine that the current backup file has been pulled.

[0059] Furthermore, according to the pull status marks of the data storage standard files and the backup files, the unpulled backup files are determined, and the unpulled backup files stored in the storage space are pulled, thereby avoiding repeated parsing of the backup files and improving the parsing efficiency.

[0060] In some implementations, it is also possible to obtain a file list of parsed backup files maintained by the target node itself, and deduplicate the pulled backup files based on the file list. That is, the target node records the parsed backup files to obtain a file list, and when pulling and parsing the backup files, the backup files in the file list are removed to avoid repeated parsing of the backup files.

[0061] S204, obtaining directory attribute information.

[0062] In some implementations, the metadata in the backup file can be parsed to obtain a set of attribute information; in response to the end of the BR operation, the attribute information set is filtered according to the directory configuration information to obtain the target attribute information; that is, the attribute information set in all metadata in the backup file is determined by parsing the metadata in the backup file, and the attribute information can be data name, data size, data type, data structure (such as length, field or data column) and data association information (location or owner) and other information; the attribute information set is filtered according to the pre-set directory configuration information, and the directory configuration information is the attribute information required for the client configuration, so the required directory attribute information is filtered out from the attribute information set according to the directory configuration information to reduce unnecessary resource consumption.

[0063] In some other implementations, in response to the BR operation ending, a subset of attribute information parsed by the target node may be obtained; the attribute information subsets may be combined to obtain an attribute information set, and the attribute information set may be filtered according to the directory configuration information to obtain the directory attribute information.

[0064] That is, obtain the attribute information subset of the backup file parsed by each target node, combine the attribute information subsets parsed by all target nodes, obtain the attribute information set in all metadata in the backup file, and further filter the attribute information set according to the pre-set directory configuration information to screen out the required directory attribute information to avoid resource consumption on the critical path.

[0065] S205, generating a full directory path according to the directory attribute information, and obtaining quota information of each directory according to the full directory path.

[0066] In the embodiments of the present disclosure, the implementation method of step S205 can be implemented by any of the methods in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0067] In this embodiment, during the process of performing BR operation on the database, the file backup status is determined, and when the file backup is completed, multiple target nodes are scheduled to parse the exported backup files to improve the parsing efficiency. By sending a backup parsing request carrying storage information, the target node is instructed to pull the exported backup files in the storage location. During the pulling process, the backup files that have been parsed and processed can be screened by file identifiers or the file list of the target node itself, thereby avoiding repeated pulling and repeated parsing of the exported backup files, reducing resource consumption in the processing process, avoiding repeated operations and repeated processing, and obtaining attribute information through efficient parsing of multiple target nodes. Irrelevant attribute information is filtered and screened through pre-configured directory configuration information, and a full directory path is generated based on the filtered directory attribute information and the quota information of the directory is obtained, thereby improving the timeliness of obtaining the directory quota information.

[0068] Figure 3 is a schematic diagram of another method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure; Figure 3 As shown, the method includes:

[0069] S301: Perform a BR operation on a database to export a backup file containing metadata from the database.

[0070] In the embodiments of the present disclosure, the implementation method of step S301 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0071] S302: During the execution of the BR operation, the exported backup file is synchronously parsed to obtain directory attribute information.

[0072] In the embodiments of the present disclosure, the implementation method of step S302 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0073] S303, generating a full directory path according to the directory attribute information.

[0074] In the embodiments of the present disclosure, the implementation method of step S303 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0075] S304, split the full directory path to obtain the directory hierarchy relationship corresponding to the full directory path.

[0076] In some implementations, separators can be identified from the full path of the directory, and the full path of the directory can be segmented according to the separators, and the hierarchical directory can be extracted from the information between adjacent separators; the position order of the separators in the full path of the directory is obtained, and the level of the hierarchical directory is determined according to the position order of the separators; the hierarchical directories at each level are counted to generate a directory hierarchy relationship.

[0077] Optionally, the directory separator can be a separator symbol such as "\" or " / ". The separator is identified from the full path of the directory and split according to the separator, and the hierarchical directory is extracted from the information between adjacent separators. For example, if the full path of the directory is C:\Users\username, the full path of the directory is split into "C:", "Users" and "username" according to the separator, and the hierarchical directory extracted from the information between adjacent separators is "C:", "Users" and "username".

[0078] Furthermore, the level of the hierarchical directory is determined according to the position order of the separators in the target full path. For example, the hierarchical directory between the first separator is level 1, the hierarchical directory between the first separator and the second separator is level 2, and so on. That is, the level where "C:" is located is level 1, the level where "Users" is located is level 2, and the level where "username" is located is level 3; the hierarchical directories at each level are counted to generate the directory hierarchy relationship corresponding to the full path of the directory, and each directory level is more intuitively displayed and analyzed.

[0079] S305, obtaining quota information of each directory in the full path of the directory according to the directory hierarchy relationship.

[0080] In some implementations, the first directory attribute information of the hierarchical directory can be obtained from the directory attribute information; the number of files and the total size of the files included in the hierarchical directory can be determined based on the first directory attribute information; and the quota information of the hierarchical directory can be determined based on the number of files and the total size of the files. That is, the first directory attribute information corresponding to any hierarchical directory can be obtained from the directory attribute information, and the first directory attribute information can include the number of files and the total size of the files included in the hierarchical directory. Therefore, the number of files and the total size of the files included in the hierarchical directory can be obtained based on the first directory attribute information, and the number of files and the total size of the files are used as the quota information of the hierarchical directory. The quota information of the hierarchical directory is used as a statistical basis to improve the accuracy of obtaining quota information.

[0081] Furthermore, after determining the quota information of the hierarchical directories under each level, the quota information of the hierarchical directories is recursively applied according to the directory hierarchy relationship to obtain the quota information of the directory. For example, for the quota information of the hierarchical directory "C:" whose full directory path is C:\Users\username, the quota is recursively applied according to the directory hierarchy relationship, that is, "username", "Users" and "C:" are accumulated according to the directory hierarchy relationship to obtain the quota information of the directory "C:"; accordingly, for the directory "Users", its quota information is the accumulation of the quota information of "username" and the quota information of "Users", while the quota information of the directory "username" is the quota information of the directory of this level itself, and the quota information of the directory is obtained with higher accuracy.

[0082] In some implementations, any level directory "Users" may appear in the full path of other directories. When accumulating the quota information of the directory "Users", the quota information of all levels of directories under the "Users" level in the full path of all directories is accumulated, thereby obtaining the accurate quota information of the directory "Users".

[0083] Furthermore, the storage information of the quota information of the directory can be obtained, and the quota information of the directory is stored according to the storage information. Optionally, the storage information can include the location where the quota information is to be stored, and the quota information of the directory is stored according to the storage location in the storage information, so as to facilitate subsequent user acquisition and viewing.

[0084] In this embodiment, after obtaining the full path of the directory, the full path of the directory is segmented to obtain the hierarchical directories corresponding to the target full path and the hierarchical directories, so as to perform a more accurate analysis of the full path of the directory. By obtaining the quota information of each hierarchical directory, recursive accumulation is performed according to the directory hierarchy relationship to obtain the quota information of each directory in the target full path. The storage information of the quota information of the directory can also be obtained, and the information is stored based on the storage information to facilitate user acquisition and viewing. The process of obtaining the directory quota information does not involve disk reading and writing. Compared with traditional real-time incremental statistics and full-time scheduled statistics, it can take into account both timeliness and ease of use. By segmenting the full path of the directory, more detailed quota information statistics are performed to ensure the absolute accuracy of the directory quota statistics.

[0085] Figure 4 is a schematic diagram of another method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure; Figure 4 As shown, the method includes:

[0086] S401: Perform a BR operation on a database to export a backup file containing metadata from the database.

[0087] In the embodiments of the present disclosure, the implementation method of step S401 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0088] S402, during the execution of the BR operation, the task execution status of the candidate node is obtained, and based on the task execution status, a plurality of target nodes are scheduled.

[0089] In the embodiments of the present disclosure, the implementation method of step S402 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0090] S403, sending a backup parsing request to the target node, and the target node pulls the backup file for parsing.

[0091] In the embodiments of the present disclosure, the implementation method of step S403 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0092] S404, obtaining directory attribute information.

[0093] In the embodiments of the present disclosure, the implementation method of step S404 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0094] S405: Generate a full directory path according to the directory attribute information.

[0095] In the embodiments of the present disclosure, the implementation method of step S405 can be implemented by any of the methods in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0096] S406, split the full directory path to obtain the directory hierarchy relationship corresponding to the full directory path.

[0097] In the embodiments of the present disclosure, the implementation method of step S406 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0098] S407, obtaining quota information of each directory in the full path of the directory according to the directory hierarchy relationship.

[0099] In the embodiments of the present disclosure, the implementation method of step S407 may be implemented in any manner in the embodiments of the present disclosure, which is not limited here and will not be described in detail.

[0100] In this embodiment, a full backup file is obtained by performing a BR operation on the database. Compared with the existing real-time incremental statistics method, each round of directory quota statistics based on the backup file can achieve full correction to ensure the accuracy of the directory quota; in the process of performing a BR operation on the database, the file backup status is determined, and when the file backup is completed, multiple target nodes are scheduled to parse the exported backup files to improve the parsing efficiency. By sending a backup parsing request carrying storage information, the target node is instructed to pull the exported backup files in the storage location. During the pulling process, the file identifier or the file list of the target node itself can be used to filter the parsed backup files to avoid repeated pulling and repeated parsing of the exported backup files, thereby reducing Resource consumption during the processing process is avoided, repeated operations and repeated processing are avoided, irrelevant attribute information is filtered and screened through pre-configured directory configuration information, the full path of the directory is generated based on the screened directory attribute information, the full path of the directory is segmented to obtain the hierarchical directory corresponding to the target full path and the level of the hierarchical directory, so as to make a more accurate analysis of the full path of the directory, by obtaining the quota information of each hierarchical directory, recursively accumulating according to the directory hierarchy relationship, and obtaining the quota information of each directory in the target full path. The storage information of the quota information of the directory can also be obtained, and stored based on the storage information for user acquisition and viewing. In the process of obtaining the directory quota information, disk reading and writing are not involved, which can take into account timeliness and ease of use, and at the same time ensure the absolute accuracy of the directory quota statistics.

[0101] Figure 5 is a logic diagram of another method for obtaining directory quota information based on database backup and recovery BR provided by an embodiment of the present disclosure; Figure 5 As shown, when the task is initiated, the Master node initiates the directory quota statistics task, that is, the BR operation is performed on the database (Data Base, DB) to export the backup file, and the backup file includes metadata information. During the execution of the BR operation, the Master node sends a backup parsing request to multiple target nodes to instruct the target nodes to pull and parse the backup file to obtain all attribute information, filter all attribute information to obtain directory attribute information, further splice the entire directory based on the directory attribute information to obtain the full path of each directory, and after splicing the full path of the directory, the full path of the directory is segmented to obtain multiple hierarchical directories, and the quota information of the hierarchical directories is obtained and recursively accumulated to obtain the quota information of each directory, and further push and store the full directory quota information for users to obtain and view.

[0102] Figure 6 is a block diagram of a directory quota information acquisition device based on database backup and recovery BR provided by an embodiment of the present disclosure; Figure 6As shown, the directory quota information acquisition device 600 based on database backup recovery BR includes:

[0103] The export module 601 is used to perform a BR operation on the database to export a backup file containing metadata from the database;

[0104] The parsing module 602 is used to synchronously parse the exported backup file to obtain directory attribute information during the execution of the BR operation;

[0105] The acquisition module 603 is used to generate a full directory path according to the directory attribute information, and acquire the quota information of each directory according to the full directory path.

[0106] In some implementations, the acquisition module includes:

[0107] Split the full path of the directory to obtain the directory hierarchy relationship corresponding to the full path of the directory;

[0108] Get the quota information of each directory in the full path of the directory based on the directory hierarchy.

[0109] In some implementations, the acquisition module 603 includes:

[0110] Get the quota information of the hierarchical directories under each level;

[0111] For the quota information of the hierarchical directory, quota recursion is performed according to the directory hierarchy to obtain the quota information of the directory.

[0112] In some implementations, the acquisition module 603 includes:

[0113] Identify separators from the full path of the directory, split the full path of the directory according to the separators, and extract the hierarchical directory from the information between adjacent separators;

[0114] Get the position order of the separators in the full path of the directory, and determine the level of the hierarchical directory based on the position order of the separators;

[0115] Statistics of hierarchical directories are performed on each level to generate a directory hierarchy relationship.

[0116] In some implementations, the acquisition module 603 includes:

[0117] From the directory attribute information, obtain the first directory attribute information of the hierarchical directory;

[0118] Determine the number of files and the total size of the files included in the hierarchical directory according to the first directory attribute information;

[0119] Determine the quota information for the hierarchical directory based on the number of files and the total size of the files.

[0120] In some implementations, the apparatus 600 further includes:

[0121] The storage information of the quota information of the directory is obtained, and the quota information of the directory is stored according to the storage information.

[0122] In some implementations, the parsing module 602 includes:

[0123] Parse the metadata in the backup file to obtain a set of attribute information;

[0124] In response to the BR operation being completed, the attribute information set is filtered according to the directory configuration information to obtain the directory attribute information.

[0125] In some implementations, the parsing module 602 includes:

[0126] Obtain the task execution status of candidate nodes, and schedule multiple target nodes based on the task execution status;

[0127] A backup parsing request is sent to the target node, and the target node pulls the backup file for parsing.

[0128] In some implementations, the parsing module 602 includes:

[0129] In response to the BR operation ending, obtaining a subset of attribute information parsed by the target node;

[0130] The attribute information subsets are combined to obtain an attribute information set.

[0131] In some implementations, the apparatus 600 further includes:

[0132] Determine storage information of the backup file, and store the backup file based on the storage information;

[0133] The target node obtains the storage information of the backup file from the backup resolution request, and pulls the backup file based on the storage information of the backup file.

[0134] In some implementations, the parsing module 602 includes:

[0135] Obtaining a data storage standard file in the storage space where the storage information of the backup file is located, the data storage standard file including a file identifier of the stored backup file;

[0136] In response to the backup file being pulled, marking the pulling status of the backup file;

[0137] According to the pull status marks of the data storage standard files and the backup files, the unpulled backup files are determined, and the unpulled backup files stored in the storage space are pulled.

[0138] In some implementations, the apparatus 600 further includes:

[0139] Get the file list of parsed files maintained by the target node itself, and deduplicate the pulled backup files based on the file list.

[0140] In this embodiment, a full backup file is obtained by performing a BR operation on the database. Compared with the existing real-time incremental statistics method, each round of directory quota statistics based on the backup file can achieve full correction to ensure the accuracy of the directory quota; in the process of performing a BR operation on the database, the file backup status is determined, and when the file backup is completed, multiple target nodes are scheduled to parse the exported backup files to improve the parsing efficiency. By sending a backup parsing request carrying storage information, the target node is instructed to pull the exported backup files in the storage location. During the pulling process, the file identifier or the file list of the target node itself can be used to filter the parsed backup files to avoid repeated pulling and repeated parsing of the exported backup files, thereby reducing Resource consumption during the processing process is avoided, repeated operations and repeated processing are avoided, irrelevant attribute information is filtered and screened through pre-configured directory configuration information, the full path of the directory is generated based on the screened directory attribute information, the full path of the directory is segmented to obtain the hierarchical directory corresponding to the target full path and the level of the hierarchical directory, so as to make a more accurate analysis of the full path of the directory, by obtaining the quota information of each hierarchical directory, recursively accumulating according to the directory hierarchy relationship, and obtaining the quota information of each directory in the target full path. The storage information of the quota information of the directory can also be obtained, and stored based on the storage information for user acquisition and viewing. In the process of obtaining the directory quota information, disk reading and writing are not involved, which can take into account timeliness and ease of use, and at the same time ensure the absolute accuracy of the directory quota statistics.

[0141] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0142] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0143] Figure 7A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure is shown. Electronic device 700 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0144] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0145] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0146] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a directory quota information acquisition method based on database backup recovery BR. For example, in some embodiments, the directory quota information acquisition method based on database backup recovery BR may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the directory quota information acquisition method based on database backup recovery BR described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the directory quota information acquisition method based on database backup and recovery BR in any other appropriate manner (for example, by means of firmware).

[0147] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0149] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0150] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0151] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0152] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0153] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0154] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for obtaining directory quota information based on database backup and recovery BR, wherein: The method comprises: Performing a BR operation on a database to export a backup file containing metadata from the database; During the execution of the BR operation, the exported backup file is synchronously parsed to obtain directory attribute information; A full directory path is generated according to the directory attribute information, and quota information of each directory is obtained according to the full directory path.

2. The method according to claim 1, wherein: The step of obtaining quota information of each directory according to the full path of the directory includes: Segment the full directory path to obtain the directory hierarchy relationship corresponding to the full directory path; According to the directory hierarchy relationship, the quota information of each directory in the directory full path is obtained.

3. The method according to claim 2, wherein: The step of obtaining quota information of each directory in the full path of the directory according to the directory hierarchy relationship includes: Get the quota information of the hierarchical directories under each level; For the quota information of the hierarchical directory, quota recursion is performed according to the directory hierarchy relationship to obtain the quota information of the directory.

4. The method according to claim 2, wherein: The step of segmenting the full directory path to obtain a directory hierarchy relationship corresponding to the full directory path includes: Identify separators from the full path of the directory, segment the full path of the directory according to the separators, and extract hierarchical directories from information between adjacent separators; Obtaining the position sequence of separators in the full path of the directory, and determining the level of the hierarchical directory according to the position sequence of the separators; Statistics of the hierarchical directories are performed on each level to generate the directory hierarchical relationship.

5. The method according to claim 3, wherein: The step of obtaining the quota information of the hierarchical directory under each level includes: Acquire first directory attribute information of the hierarchical directory from the directory attribute information; Determine the number of files and the total size of files included in the hierarchical directory according to the first directory attribute information; The quota information of the hierarchical directory is determined according to the number of files and the total size of the files.

6. The method according to claim 1, wherein: After obtaining the quota information of the directory according to the full path of the directory, the method further includes: The storage information of the quota information of the directory is obtained, and the quota information of the directory is stored according to the storage information.

7. The method according to any one of claims 1 to 6, wherein: The synchronously parsing the continuously exported backup files to obtain directory attribute information includes: Parsing metadata in the backup file to obtain a set of attribute information; In response to the BR operation being completed, the attribute information set is filtered according to the directory configuration information to obtain the directory attribute information.

8. The method according to claim 7, wherein: The synchronously parsing the continuously exported backup files includes: Obtaining the task execution status of the candidate nodes, and scheduling the multiple target nodes based on the task execution status; A backup parsing request is sent to the target node, and the target node pulls the backup file for parsing.

9. The method according to claim 8, wherein: The step of parsing the metadata in the backup file to obtain a set of attribute information includes: In response to the BR operation ending, obtaining a subset of attribute information parsed by the target node; The attribute information subsets are combined to obtain an attribute information set.

10. The method according to claim 8, wherein: The method further comprises: Determining storage information of the backup file, and storing the backup file based on the storage information; The target node obtains the storage information of the backup file from the backup parsing request, and pulls the backup file based on the storage information of the backup file.

11. The method according to claim 8, wherein: The process of the target node pulling the backup file includes: Acquire a data storage standard file in the storage space where the storage information of the backup file is located, wherein the data storage standard file includes a file identifier of the stored backup file; In response to the backup file being pulled, marking a pulling status of the backup file; According to the data storage standard file and the pull status mark of the backup file, the unpulled backup file is determined, and the unpulled backup file stored in the storage space is pulled.

12. The method according to claim 8, wherein: The method further comprises: A file list of parsed files maintained by the target node itself is obtained, and the pulled backup files are deduplicated based on the file list.

13. A directory quota information acquisition device based on database backup and recovery BR, comprising: An export module, used for performing a BR operation on a database to export a backup file containing metadata from the database; A parsing module, used for synchronously parsing the exported backup file to obtain directory attribute information during the execution of the BR operation; The acquisition module is used to generate a full directory path according to the directory attribute information, and to acquire quota information of each directory according to the full directory path.

14. The device according to claim 13, wherein: The acquisition module comprises: Segment the full directory path to obtain the directory hierarchy relationship corresponding to the full directory path; According to the directory hierarchy relationship, the quota information of each directory in the directory full path is obtained.

15. The device according to claim 14, wherein: The acquisition module comprises: Get the quota information of the hierarchical directories under each level; For the quota information of the hierarchical directory, quota recursion is performed according to the directory hierarchy relationship to obtain the quota information of the directory.

16. The device according to claim 14, wherein: The acquisition module comprises: Identify separators from the full path of the directory, segment the full path of the directory according to the separators, and extract hierarchical directories from information between adjacent separators; Obtaining the position sequence of separators in the full path of the directory, and determining the level of the hierarchical directory according to the position sequence of the separators; Statistics of the hierarchical directories are performed on each level to generate the directory hierarchical relationship.

17. The device according to claim 15, wherein: The acquisition module comprises: Acquire first directory attribute information of the hierarchical directory from the directory attribute information; Determine the number of files and the total size of files included in the hierarchical directory according to the first directory attribute information; The quota information of the hierarchical directory is determined according to the number of files and the total size of the files.

18. The device according to claim 13, wherein: The device further comprises: The storage information of the quota information of the directory is obtained, and the quota information of the directory is stored according to the storage information.

19. The device according to any one of claims 13 to 18, wherein: The parsing module comprises: Parsing metadata in the backup file to obtain a set of attribute information; In response to the BR operation being completed, the attribute information set is filtered according to the directory configuration information to obtain the directory attribute information.

20. The device according to claim 19, wherein The parsing module comprises: Obtaining the task execution status of the candidate nodes, and scheduling the multiple target nodes based on the task execution status; A backup parsing request is sent to the target node, and the target node pulls the backup file for parsing.

21. The device according to claim 20, wherein: The parsing module comprises: In response to the BR operation ending, obtaining a subset of attribute information parsed by the target node; The attribute information subsets are combined to obtain an attribute information set.

22. The device according to claim 20, wherein: The device also includes: Determining storage information of the backup file, and storing the backup file based on the storage information; The target node obtains the storage information of the backup file from the backup parsing request, and pulls the backup file based on the storage information of the backup file.

23. The device according to claim 20, wherein: The parsing module comprises: Acquire a data storage standard file in the storage space where the storage information of the backup file is located, wherein the data storage standard file includes a file identifier of the stored backup file; In response to the backup file being pulled, marking a pulling status of the backup file; According to the data storage standard file and the pull status mark of the backup file, the unpulled backup file is determined, and the unpulled backup file stored in the storage space is pulled.

24. The device according to claim 20, wherein: The device also includes: A file list of parsed files maintained by the target node itself is obtained, and the pulled backup files are deduplicated based on the file list.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

Images