Database index generation method and device and data processing method and device

Abstract

The invention provides a database index generation method and device and a data processing method and device, and relates to the technical field of computers, in particular to the technical field of distributed systems, storage management and the like. According to the specific implementation scheme, the method comprises the steps of determining to-be-aggregated items related to file quotas based on metadata of a metadatabase in the distributed file system; based on the to-be-aggregated items, determining an item aggregation function; creating an aggregation index in the metadatabase; performing statistics on the to-be-aggregated items according to the item aggregation function to obtain statistical data; and storing the statistical data in the aggregation index as aggregation data.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, specifically to technical fields such as large models, deep learning, and image processing, and in particular to a database index generation method and device, a data processing method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] The quota function in the file system is used to count user usage. When the user usage exceeds the quota size, user access needs to be restricted to avoid over-issuance. Therefore, the quota function plays a billing role in the multi-tenant file system.

[0003] For file systems implemented on a single machine, since all the data in the directory tree is concentrated together, it is easy to count the user's usage. However, with the increasing size of user data, a single file system often requires hundreds of PB (Petabyte) of storage space. In this case, the traditional single-machine architecture is often unable to support it. Therefore, file systems based on distributed architecture, namely distributed file systems, came into being. In a distributed file system, since the directory tree data will be dispersed to multiple machines to achieve data sharding and load balancing, it is very difficult to calculate the quota function of the distributed architecture of the file system, and it is impossible to accurately calculate the user's usage. Summary of the invention

[0004] The present disclosure provides a database index generation method and device, a data processing method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect, a method for generating a database index is provided, the method comprising: determining items to be aggregated related to file quotas based on metadata of a metadata base in a distributed file system; determining an item aggregation function based on the items to be aggregated; creating an aggregation index in the metadata base; performing statistics on the items to be aggregated according to the item aggregation function to obtain statistical data; and saving the statistical data as aggregated data in the aggregation index.

[0006] According to a second aspect, a data processing method is provided, the method comprising: sending a request to a metadata repository of a distributed file system to obtain aggregated data, the aggregated data being data created by adopting a database index generation method as described in any implementation of the first aspect; receiving the aggregated data sent by the metadata repository; performing a quota check on the aggregated data to obtain a quota check result.

[0007] According to a third aspect, a database index generating device is provided, which includes: an item determination unit, configured to determine items to be aggregated related to file quotas based on metadata of a metadata base in a distributed file system; a function determination unit, configured to determine an item aggregation function based on the items to be aggregated; a creation unit, configured to create an aggregation index in the metadata base; an acquisition unit, configured to perform statistics on the items to be aggregated according to the item aggregation function to obtain statistical data; and a storage unit, configured to store the statistical data as aggregated data in the aggregation index.

[0008] According to a fourth aspect, a data processing device is provided, which includes: a sending unit, configured to send a request for obtaining aggregate data to a metadata repository of a distributed file system, where the aggregate data is data created by a database index generation device as described in any implementation method of the third aspect; a receiving unit, configured to receive the aggregate data sent by the metadata repository; and a detection unit, configured to perform a quota detection on the aggregate data to obtain a quota detection result.

[0009] According to the fifth aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect or the second aspect.

[0010] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.

[0011] According to a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any implementation of the first aspect or the second aspect.

[0012] The database index generation method and device provided by the embodiment of the present disclosure first determine the items to be aggregated related to the file quota based on the metadata of the metadata base in the distributed file system; secondly, determine the item aggregation function based on the items to be aggregated; thirdly, create an aggregation index in the metadata base; thirdly, count the items to be aggregated according to the item aggregation function to obtain statistical data; finally, save the statistical data as aggregated data in the aggregation index. Thus, with the aid of the aggregation index mechanism of the metadata base of the distributed file system, the items to be aggregated of the distributed file system can be aggregated, so that the usage related to the quota can be effectively calculated without damaging the user input and output path, and the usage can be made real-time and eventually consistent by aggregating the items to be aggregated, thereby improving the real-time and accuracy of the usage calculation of the distributed file system.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a flow chart of an embodiment of a method for generating a database index according to the present disclosure;

[0016] Figure 2 It is a structural diagram of the distributed file system disclosed in the present invention;

[0017] Figure 3 This is a schematic diagram of a structure for implementing aggregate indexes in the metadata database disclosed herein;

[0018] Figure 4 is a flow chart of an embodiment of a data processing method according to the present disclosure;

[0019] Figure 5 is a schematic diagram of the structure of the process of detecting aggregated data in the present disclosure;

[0020] Figure 6 is a structural schematic diagram of an embodiment of a database index generating device according to the present disclosure;

[0021] Figure 7 is a schematic diagram of the structure of an embodiment of a database processing device according to the present disclosure;

[0022] Figure 8 It is a block diagram of an electronic device used to implement the database index generating method or data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] The quota function of the distributed file system is used to count user usage, that is, the usage is quota-related usage, and user usage includes two dimensions: the number of files and the size of the files. In the traditional technology for calculating user usage, the quota function generally adopts a method based on IO associated statistics and a method based on metadata database snapshots.

[0025] Based on IO associated statistics: Track points on the IO path requested by the user to generate IO operation logs. The operation log usually records the following fields, including file system identifier, operation type, file index number, offset, data length, etc., and synchronizes the operation log to a central node. The node is responsible for parsing, processing and analyzing the above IO operation logs, increasing and decreasing the usage data in real time, and finally obtaining complete usage data for use by the upper-level quota module. This method has the following disadvantages:

[0026] 1) Low accuracy. The IO operation log needs to be synchronized between the directory tree node and the quota center node, and this process is often asynchronous with the user IO, which makes the two likely to be inconsistent. For example, when the directory tree node is changed online, the user IO operation may be executed successfully, but the IO operation log synchronization fails. If this IO operation is a large file deletion, it will lead to a large deviation in user usage.

[0027] 2) Average performance. An IO operation log needs to go through multiple links from generation to effectiveness, some of which have large overhead, including: IO operation log storage, IO operation log synchronization via RPC, etc. These inefficient operations increase the overhead on the disk and network, and reduce the timeliness of the quota to a certain extent.

[0028] 3) Insufficient scalability, because the IO operation logs need to be aggregated from each directory tree node to the quota center node, and the quota center node is a single centralized node and is not scalable. If the metadata operation or file read and write interface throughput of the file system is large, a single point bottleneck is likely to occur, which limits the overall performance of the system.

[0029] Based on the metadata database snapshot method: Utilize the metadata database's storage transfer capability to regularly export metadata snapshots in full, and then parse, process and analyze the snapshots to obtain the number of files in each file system and the file size, and then sum them up to obtain the respective usage data. Usually, for scalability considerations, the metadata database will have multiple data nodes; after these data nodes transfer the snapshots, they will be aggregated to a central node, and then the subsequent aggregation tasks will be performed. This method has the following disadvantages:

[0030] 1) Poor timeliness. Due to the large amount of data, a storage transfer process usually takes hours. If the failed storage transfer process is considered, the time consumption will be further magnified. The user quota function usually requires a delay of no more than minutes, so the metadata database snapshot solution cannot meet the timeliness requirements.

[0031] 2) Poor data consistency. Typical file systems will try to evenly distribute data among several metadata database nodes. In snapshot-based solutions, snapshots of various nodes need to be aggregated together. Considering that the storage transfer process is time-consuming and prone to failure, when there are many metadata nodes, the time points of these node snapshots are likely to differ by several hours. If these data are forcibly merged together, anomalies or even errors will occur due to inconsistent data views.

[0032] In view of the defects in the traditional technology, the present invention proposes a database index generation method, which provides a reliable implementation method for the calculation of quotas in a distributed file system by constructing an aggregate index and recording the items to be aggregated related to the quota. Figure 1 A process 100 according to an embodiment of a method for generating a database index according to the present disclosure is shown. The method for generating a database index includes the following steps:

[0033] Step 101: Determine items to be aggregated related to file quotas based on metadata of a metadatabase in a distributed file system.

[0034] In this embodiment, the execution subject on which the database index generation method runs can be a storage engine of a metadata base, and the storage engine is also a directory tree node, wherein the metadata base is a database of a distributed file system, and the metadata base mainly stores the directory tree and file attributes of the distributed file system, such as Figure 2 As shown, in a distributed file system, Figure 2 In the , the directory tree node provides metadata services of file semantics to the client. Since the metadata in the distributed file system is sunk to the underlying source database, the directory tree node is a stateless service, which is easy to load balance and scale horizontally. In addition, the metadata database is also based on a distributed architecture and has multiple metadata database shards. The metadata database shards store single table data on multiple machines and jointly provide data services to the outside world.

[0035] In this embodiment, the file quota includes: a name quota and a space quota. The name quota is a hard limit on the number of file and directory names in the current directory tree. If the quota is exceeded, the file and directory creation fails. The space quota is a hard limit on the number of bytes used by files in the tree with the directory as the root. If the quota does not allow writing of a complete block, the block allocation fails.

[0036] In this embodiment, the item to be aggregated is at least one key field in the directory item of the metadata database, and the items to be aggregated related to the file quota in the metadata database include key fields related to user usage such as file owner, file size, number of recorded files, etc. The item to be aggregated can be a key field related to the name quota and / or space quota.

[0037] In this embodiment, after obtaining the metadata of the metadata database of the distributed file system, at least one key field can be selected from all key fields related to the file quota as an item to be aggregated based on the quota calculation requirements, and the item to be aggregated is a field to be aggregated in the aggregation index.

[0038] In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of metadata and items to be aggregated are carried out after authorization and in compliance with relevant laws and regulations. The information related to the user in the metadata and items to be aggregated is information obtained after the user's permission.

[0039] Step 102: Determine an item aggregation function based on the items to be aggregated.

[0040] In this embodiment, the item aggregation function is a function that calculates the value of the item to be aggregated, and the item aggregation function belongs to the user-defined function in the aggregation index. Since the metadatabase aggregation index is a mechanism for automatically maintaining user-defined aggregation functions, the item aggregation function is a function to be automatically maintained by the metadatabase aggregation index.

[0041] In this embodiment, the item aggregation function may be determined based on the quota calculation requirement of the current items to be aggregated. For example, if the quota requirement of the items to be aggregated is cumulative, the item aggregation function is a value function of accumulating the items to be aggregated.

[0042] Step 103: Create an aggregate index in the metadata database.

[0043] In this embodiment, the aggregate index in the metadata database is a mechanism for automatically maintaining user-defined aggregate functions, supporting grouping and calculating the number of records according to a given key and accumulating the key value of the key. For example, when metadata is inserted, deleted, or updated, the aggregate value can be automatically maintained according to the aggregate index settings and stored in the aggregate index. The implementation principle is as follows: Figure 3 As shown:

[0044] Figure 3 In the above, Proxy represents the front-end node of the metadata database, which is mainly responsible for receiving the user's structured query statements, parsing, analyzing, optimizing, generating and executing physical plans, and finally converting them into key-value operations on the BE node; BE represents the storage engine node of the source database, which is mainly responsible for managing key-value records and implementing operations such as adding, deleting, modifying, and querying key-value records; Primarytablet represents the primary index, Sec Index tablet represents the secondary index, and Aggr Index tablet represents the clustered index; CDC is a component located in BE, and its full name is change data capture, which is used to receive record change messages and perform corresponding processing, such as updating the secondary index.

[0045] An aggregate index is similar to a secondary index, but the difference is that it maintains an aggregate function for a given key. When BE completes the key value operation, it generates a record change message according to the aggregate index settings and sends it to CDC. When CDC processes the message, it applies the changes to the aggregate index, thereby updating the aggregate value.

[0046] In this embodiment, creating an aggregate index mainly involves creating an index data table, which stores data of quota-related items to be aggregated, that is, aggregated data. The values ​​in the index data table change as the values ​​calculated by the item aggregation function after the client accesses the metadata library change.

[0047] In this embodiment, an aggregate index is created, and the aggregate index is used to pre-count and save the results of subsequent commonly used statistical queries, so that data can be provided conveniently and quickly during subsequent statistical queries. On the one hand, the processing efficiency of statistical queries is improved by virtue of the orderliness of the index; on the other hand, the aggregate index is updated accordingly as the data changes, ensuring the real-time nature of the data. The aggregated data of the aggregate index is obtained by counting all relevant data of the indexed relationship (which can be a single table or a result table of a subquery), ensuring the accuracy of the statistical data.

[0048] Step 104, statistics are performed on the items to be aggregated according to the item aggregation function to obtain statistical data.

[0049] In this embodiment, the statistical data is the data obtained after the function operation corresponding to the item aggregation function is performed on the items to be aggregated, and the statistical data includes the key representing the items to be aggregated and the actual value of the items to be aggregated calculated by the item aggregation function.

[0050] In this embodiment, performing statistics on the items to be aggregated according to the item aggregation function means using the item aggregation function to calculate the value of the item to be aggregated to obtain the actual value of the item to be aggregated; for example, if the item aggregation function is an accumulation function, after obtaining the value of the item to be aggregated, the value of the item to be aggregated is accumulated to obtain the actual value of the item to be aggregated.

[0051] Optionally, after the client performs operations such as adding, deleting, modifying, and querying metadata related to the items to be aggregated in the metadata database, statistics can be collected on the items to be aggregated according to the item aggregation function to obtain statistical data.

[0052] Step 105: Save the statistical data as aggregated data in an aggregate index.

[0053] In this embodiment, the aggregated data is the data stored in the aggregated index. The aggregated index generally stores data in the form of a table. The aggregated data represents the fields of the table in the aggregated index and the actual values ​​of the fields, wherein the fields are the items to be aggregated and the actual values ​​of the fields are the actual values ​​of the items to be aggregated.

[0054] In this embodiment, the above 105 includes: using the items to be aggregated in the statistical data as fields of the aggregated data, and storing the actual values ​​of the items to be aggregated in the statistical data as actual values ​​of the corresponding fields in the aggregated data in the aggregated index.

[0055] The database index generation method provided by the embodiment of the present disclosure first determines the items to be aggregated related to the file quota based on the metadata of the metadata base in the distributed file system; secondly, determines the item aggregation function based on the items to be aggregated; thirdly, creates an aggregation index in the metadata base; thirdly, performs statistics on the items to be aggregated according to the item aggregation function to obtain statistical data; finally, saves the statistical data in the aggregation index as aggregate data. Thus, with the aid of the aggregation index mechanism of the metadata base of the distributed file system, the items to be aggregated of the distributed file system can be aggregated, so that the usage can be effectively calculated without damaging the user input and output path, and the usage can be made real-time and ultimately consistent, thereby improving the real-time and accuracy of the usage calculation of the distributed file system.

[0056] In some embodiments of the present disclosure, the metadata of the metadata database of the distributed file system is used to determine the items to be aggregated related to the file quota, including: obtaining calculation requirement data of the file quota; obtaining key fields in the directory item table and the target item table of the metadata database of the distributed file system; matching the key fields with the calculation requirement data to obtain the items to be aggregated related to the file quota.

[0057] In this optional implementation, the calculation demand data of the file quota includes: name quota calculation data and / or space quota calculation data, the name quota calculation data refers to the file and directory names in the current distributed file system directory tree not exceeding the corresponding first quota; the space quota calculation data refers to the second quota of the number of bytes used by the files in the root tree of the current distributed file system. The first quota and the second quota are both values ​​set based on the actual needs of the distributed file system.

[0058] In this optional implementation, the directory entry table of the metadata database is used to record the memory structure of the management files and directories of the distributed file system. Each file and directory has a memory structure in the distributed file system, which records the file name, parent directory, child directory and other information; these memory structures are displayed and processed in the form of tables, and the fields in the table are the key fields of the memory structure.

[0059] In this optional implementation, the key fields can be compared with the calculation demand data for similarity, and the key fields with similarity greater than the similarity threshold are the items to be aggregated related to the file quota. The similarity threshold can be set based on development requirements, for example, the similarity threshold is 80%.

[0060] Optionally, the above-mentioned matching of key fields with calculation demand data to obtain items to be aggregated related to file quotas includes: vectorizing the key fields to obtain field vectors; vectorizing the calculation demand data to obtain demand vectors; comparing the field vectors with the demand vectors for similarity, and taking the key fields corresponding to the field vectors whose similarity is greater than the similarity threshold as items to be aggregated.

[0061] The method for determining the items to be aggregated provided by this optional implementation obtains the calculation requirement data of the file quota; obtains the key fields in the directory item table and the target item table of the metadata database of the distributed file system; matches the key fields with the calculation requirement data to obtain the items to be aggregated related to the file quota, thereby correlating the acquisition of the items to be aggregated with the calculation requirement data of the file quota, and can effectively obtain the items to be aggregated, thereby improving the reliability of obtaining the items to be aggregated.

[0062] Optionally, the determining the items to be aggregated related to the file quota based on the metadata of the metadata base of the distributed file system includes: screening all key fields related to the quota in the metadata of the metadata base of the distributed file system to obtain the items to be aggregated.

[0063] In some embodiments of the present disclosure, determining the item aggregation function based on the items to be aggregated includes: selecting a function from a preset function set based on the items to be aggregated to obtain a selected function set; and obtaining the item aggregation function based on file quota calculation demand data and the selected function set.

[0064] In this optional implementation, the function set includes multiple functions, each function has a corresponding processing function, such as summing, finding the maximum, finding the sum value in a set time period, etc.

[0065] In this optional implementation, based on the items to be aggregated, a function is selected from a pre-set function set to obtain the selected function set, including: selecting a function that can process the items to be aggregated from the function set to obtain a selected function; based on the file quota calculation data, determining the key operation information, determining the function related to the key operation information from the selected function, and obtaining the item aggregation function.

[0066] The method for obtaining an item aggregation function provided in this embodiment selects functions from a pre-set function set based on the items to be aggregated to obtain a selected function set; obtains the item aggregation function based on the file quota calculation requirement data and the selected function set, thereby selecting functions related to the file quota calculation requirement from the function set to obtain the item aggregation function, thereby improving the reliability and accuracy of obtaining the item aggregation function.

[0067] In some embodiments of the present disclosure, the above-mentioned creation of the aggregate index includes: creating an index data table; and recording the correspondence between the item aggregation function, the item to be aggregated, and the index data table in the metadata database of the distributed file system.

[0068] In this optional implementation, the index data table is a table that records the relationship between item aggregation functions and items to be aggregated. The fields of the index data table include items to be aggregated, and the values ​​of each field in the index data table are calculated by the item aggregation function.

[0069] In this optional implementation, recording the correspondence between item aggregation functions, items to be aggregated, and index data tables in the metadata database of the distributed file system can effectively represent that the index data table is a table that records items to be aggregated and item aggregation functions.

[0070] In this optional implementation, the clustered index is a newly created clustered index mechanism, which is similar to the conventional clustered index in implementation, but different in implementation operation. It is updated asynchronously. After the index data table is created, the update of the data in the index data table does not block the update process of the main table.

[0071] In this optional implementation, the index data table is a table that records all data of the aggregate index. The content in the table is updated asynchronously without affecting the update of other tables in the metadata database. That is, the index data table is updated asynchronously relative to other tables in the metadata database.

[0072] The method for creating an aggregate index provided in this optional implementation creates an index data table; and records the correspondence between the item aggregation function, the item to be aggregated, and the index data table in the metadata database of the distributed file system. Thus, after the index data table is created, the correspondence between the index data table, the item aggregation function, and the item to be aggregated is recorded, thereby improving the accuracy of the data relationship of the created aggregate index.

[0073] In some optional implementations of the present disclosure, the above-mentioned creation of the aggregate index further includes: after creating the index data table, determining the data length of the statistical data; and allocating space to the index data table according to the data length.

[0074] In this optional implementation, the data length of the statistical data can be calculated by a length calculation algorithm, or the data length of the statistical data can be determined by directly counting the number of bytes of the statistical data.

[0075] In this optional implementation, the statistical data is stored in the data structure of the aggregate index, that is, for each index key value in the statistical data, additional space is allocated in the data structure of the aggregate index to store the statistical data.

[0076] The creation of an aggregate index provided by this optional implementation method includes: creating an index data table; determining the data length of the statistical data; allocating space for the index data table according to the data length, and recording the correspondence between the item aggregation function, the item to be aggregated, and the index data table in the metadata database of the distributed file system.

[0077] The method for creating an aggregate index provided by this optional implementation determines the data length of the statistical data and allocates space to the index data table according to the data length, which can ensure that the statistical data is effectively recorded and improve the accuracy of the creation of the aggregate index.

[0078] In some optional implementations of the present disclosure, the above-mentioned items to be aggregated include: the number of files and the file size under different file owners in the directory entry table of the metadata database; the item aggregation function includes: a sum function; statistics are performed on the items to be aggregated according to the item aggregation function to obtain statistical data including: using the sum function to sum the number of files under different file owners to obtain the number of files; using the sum function to sum the file sizes under different file owners to obtain the capacity value; and using the number of files and the capacity value as statistical data.

[0079] In this optional implementation, in order to store the directory tree data of the distributed file system, the metadata library needs to store the directory entry table, and the key fields of the directory entry table are: parent_inode, name, inode, type, ugi, size, etc. Among them, parent_inode and name represent the primary key, inode represents the index node, type represents the file type, ugi represents the file owner, and size represents the file size.

[0080] In order to implement the configuration of ugi's quota function on the aggregate index mechanism with the help of the aggregate index mechanism of the metadata database, it is necessary to set an index data table in the aggregate index. The format of the index data table is as follows: ugi, aggr_num, size; among them, ugi is the primary key of the index data table, aggr_num and size are other fields in the index data table except the primary key, aggr_num indicates the number of records. After the index data table is grouped according to ugi, aggr_num is used to record the number of files, and size is used to record the file size, thereby realizing the usage statistics in the quota function of ugi.

[0081] It should be noted that the aggregate index is a general capability that can be used to implement aggregate calculations of any key combination in the database. The quota function of ugi is a special type of aggregation. In order to use the aggregate index to implement the quota function of ugi, it is necessary to configure it, indicating which keys need to be aggregated and how the functions that need to be aggregated are defined. The quota function of ugi is the usage statistics of the user dimension. Each user has a capacity and usage. When the usage exceeds the capacity, the user needs to be restricted from writing data to avoid insufficient system resources.

[0082] The method for obtaining statistical data provided by this optional implementation adopts a summation function to sum the number of files of different file owners to obtain the number of files; adopts a summation function to sum the file sizes of different file owners to obtain the capacity value; uses the number of files and the capacity value as statistical data, which provides a reliable implementation method for calculating file usage and improves the reliability of obtaining statistical data.

[0083] In some optional implementations of the present disclosure, the above-mentioned items to be aggregated also include: a modulus value of the index node of the directory item table of the metadata database modulo a set value; statistics are performed on the items to be aggregated according to the item aggregation function to obtain statistical data, including: before calculating the statistical data, controlling the modulus value to be a positive integer.

[0084] In this optional implementation, the format of the index data table is: ugi, inode_mod, aggr_num, size. Among them, ugi, inode_mod are the primary keys of the index data table, aggr_num, size are other fields in the index data table except the primary key. Inode is the index node in the directory entry table, inode_mod represents the value after N modulo the index node in the directory entry table. By performing N modulo on the index node in the directory entry table and keeping the modulo value as a positive integer, the number of index nodes can be made greater than N, that is, inode_mod = inode%N, where N is 1000.

[0085] In this optional implementation, the significance of adding inode_mod based on the primary key ugi of the index data table is to avoid the hot spot problem caused by frequent updates of aggregate index records of all records under the same ugi.

[0086] In this optional implementation, the set value is a preset integer value, for example, the set value is 1000, and the items to be aggregated also include: the modulus value of the index node of the directory entry table of the metadata library modulo the set value, and the modulus value is controlled to be a positive integer, which can disperse the pressure to the set value records, thereby avoiding database hotspots caused by frequent updates of the same record.

[0087] The method for obtaining statistical data provided by this optional implementation method, the items to be aggregated also include: the modulus value of the index node of the directory entry table of the metadata database modulo a set value, and before calculating the statistical data, the modulus value is controlled to be a positive integer, which can ensure that the access pressure of the metadata database is dispersed to the set value records, avoiding metadata database hotspots caused by updating the same record, and improving the reliability of distributed file system access.

[0088] In order to realize the use of quota, the present disclosure also provides a data processing method. Specifically, Figure 4 A process 400 according to an embodiment of a data processing method of the present disclosure is shown. The data processing method comprises the following steps:

[0089] Step 401: Send a request for obtaining aggregated data to a metadata repository of a distributed file system.

[0090] In this embodiment, the aggregated data is data created by the database index generation method of the above embodiment. After the aggregated index is created, all aggregated data of the aggregated index is stored in the corresponding area of ​​the aggregated index, such as the index data table, and the aggregated data can be obtained by accessing the area, such as accessing the index data table.

[0091] In this embodiment, the request to obtain aggregated data refers to a request to obtain aggregated data from an aggregated index. The aggregated data in the aggregated index is data including a key and a value, wherein the key is the item to be aggregated and the value is the actual value corresponding to the item to be aggregated.

[0092] In this embodiment, the quota server in the metadata database is used to manage the aggregate index. Specifically, the quota server manages multiple quota table shards. The quota server provides the following service capabilities: update in seconds; support for tens of billions of files in a single cluster; eventual consistency; no data loss.

[0093] like Figure 5 As shown, the directory tree server is a component that provides directory tree services in the metadata database. The metadata related to the directory tree is stored in the directory entry table, and multiple directory entry tables are stored in different directory entry table slices. The client accesses the metadata interface of the directory tree server and modifies the data in the directory entry table slice at intervals. The modification of the data in the directory entry table slice will synchronize the changes to the quota table slice of the quota server through the CDC mechanism. The quota table slice is used to store the quota table; the execution subject (such as Figure 5 The client in the quota table obtains the aggregate data in the current quota table shard by accessing the quota server. The aggregate data is the quota information and usage of the distributed file system. The quota server directly accesses the quota table shard and provides services to the execution entity.

[0094] Step 402: Receive aggregated data sent by the metadata repository.

[0095] In this embodiment, upon receiving a request for obtaining aggregated data, the quota server obtains the aggregated data from the quota table shards, and sends the aggregated data to the execution subject on which the data processing method runs.

[0096] Step 403: perform quota detection on the aggregated data to obtain a quota detection result.

[0097] In this embodiment, detecting the aggregated data includes: detecting whether the aggregated data includes the item to be aggregated and the actual value of the item to be aggregated; in response to detecting that the aggregated data includes the item to be aggregated and the actual value of the item to be aggregated, determining that the format of the aggregated data is qualified; detecting whether the actual value of the item to be aggregated is greater than a preset quota value, if the actual value of the item to be aggregated is greater than the preset quota value, generating a quota detection result that the item to be aggregated exceeds the quota.

[0098] The data processing method provided in this embodiment has the advantages of accuracy, real-time and high efficiency, and meets the user's demand for quota function under massive data scale. In specific implementation, since the metadata is persisted using a distributed database, the quota function is realized with the help of the database's aggregate index mechanism. When the user request operation is completed, the aggregate index will asynchronously maintain the usage information, which is not only lossless to the user's IO path, but also has second-level real-time performance and final consistency.

[0099] The data processing method provided in this embodiment sends a request for obtaining aggregated data to the metadata database of the distributed file system; receives the aggregated data sent by the metadata database; performs quota detection on the aggregated data to obtain a quota detection result, which provides a reliable implementation method for realizing the quota function of the distributed file system and ensures the reliability of the usage calculation of the distributed file system.

[0100] In some optional implementations of the present disclosure, the above-mentioned quota detection on the aggregated data to obtain the quota detection result includes: obtaining a preset quota value; detecting whether the aggregated data is greater than the preset quota value, and obtaining a quota detection result including that the aggregated data is greater than the preset quota value or the aggregated data is less than the preset quota value or the aggregated data is equal to the preset quota value.

[0101] In this optional implementation, the preset quota value is a quota limit preset for the distributed file system. For example, if the preset quota value is 50 files, the preset quota value is a quota for the number of files.

[0102] The method for obtaining the quota detection result provided by this optional implementation method obtains the preset quota value; detects whether the aggregated data is greater than the preset quota value, and obtains the quota detection result including that the aggregated data is greater than the preset quota value or the aggregated data is less than the preset quota value or the aggregated data is equal to the preset quota value. By obtaining the preset quota value, the limit of the aggregated data can be flexibly set, thereby improving the flexibility of quota detection.

[0103] In some optional implementations of the present disclosure, the data processing method further includes: in response to a quota detection result that the aggregated data is greater than a preset quota value, issuing an alarm message.

[0104] In this optional implementation, the alarm information may be presented in a variety of forms and transmitted through a variety of channels, such as sending a message related to the alarm information to a pre-set alarm channel, or sending a message related to the alarm information to a specific server of the distributed file system, so that the distributed file system determines that the current file usage has exceeded a preset quota value.

[0105] The data processing method provided by this optional implementation sends out an alarm message when it is detected that the quota detection result is that the aggregated data is greater than the preset quota value, which can effectively warn of the quota over-limit situation and improve the reliability and accuracy of the quota setting of the distributed file system.

[0106] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a database index generation device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically implemented in various electronic devices.

[0107] like Figure 6 As shown, the database index generation device 600 provided in this embodiment includes: an item determination unit 601, a function determination unit 602, a creation unit 603, a obtaining unit 604, and a storage unit 605. Among them, the above-mentioned item determination unit 601 can be configured to determine the items to be aggregated related to the file quota based on the metadata of the metadata library in the distributed file system. The above-mentioned function determination unit 602 can be configured to determine the item aggregation function based on the items to be aggregated. The above-mentioned creation unit 603 can be configured to create an aggregation index in the metadata library. The above-mentioned obtaining unit 604 can be configured to perform statistics on the items to be aggregated according to the item aggregation function to obtain statistical data. The above-mentioned storage unit 605 can be configured to save the statistical data as aggregated data in the aggregation index.

[0108] In this embodiment, in the database index generating device 600, the specific processing of the item determining unit 601, the function determining unit 602, the creating unit 603, the obtaining unit 604, and the saving unit 605 and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101, step 102, step 103, step 104, and step 105 in the corresponding embodiments are not repeated here.

[0109] In some optional implementations of this embodiment, the above-mentioned item determination unit 601 is configured to: obtain calculation requirement data for file quotas; obtain key fields in the directory item table and target item table of the metadata database of the distributed file system; match the key fields with the calculation requirement data to obtain items to be aggregated related to the file quota.

[0110] In some optional implementations of this embodiment, the function determination unit 602 is configured to: select functions from a preset function set based on the items to be aggregated to obtain a selected function set; and obtain an item aggregation function based on file quota calculation demand data and the selected function set.

[0111] In some optional implementations of this embodiment, the creation unit 603 is configured to: create an index data table; and record the correspondence between the item aggregation function, the item to be aggregated, and the index data table in the metadata database of the distributed file system.

[0112] In some optional implementations of this embodiment, the creation unit 603 is further configured to: determine the data length of the statistical data; and allocate space for the index data table according to the data length.

[0113] In some optional implementations of the present embodiment, the above-mentioned items to be aggregated include: the number of files and the file size under different file owners in the directory entry table of the metadata database; the item aggregation function includes: a summation function; the above-mentioned obtaining unit 604 is configured to: use the summation function to sum the number of files under different file owners to obtain the number of files; use the summation function to sum the file sizes under different file owners to obtain the capacity value; and use the number of files and the capacity value as statistical data.

[0114] In some optional implementations of this embodiment, the above-mentioned items to be aggregated also include: a modulus value of the index node of the directory entry table of the metadata database modulo a set value; the above-mentioned obtaining unit 604 is configured to: before calculating the statistical data, control the modulus value to be a positive integer.

[0115] The database index generation device provided by the embodiment of the present disclosure is as follows: first, the item determination unit 601 determines the items to be aggregated related to the file quota based on the metadata of the metadata database in the distributed file system; secondly, the function determination unit 602 determines the item aggregation function based on the items to be aggregated; thirdly, the creation unit 603 creates an aggregation index in the metadata database; then, the obtaining unit 604 performs statistics on the items to be aggregated according to the item aggregation function to obtain statistical data; finally, the saving unit 605 saves the statistical data as aggregated data in the aggregation index. Thus, with the aid of the aggregation index mechanism of the metadata database of the distributed file system, the items to be aggregated of the distributed file system are aggregated, so that the usage can be effectively calculated without damaging the user input and output path, and the usage can be made real-time and ultimately consistent, thereby improving the real-time and accuracy of the usage calculation of the distributed file system.

[0116] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0117] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data processing device, which is similar to Figure 4Corresponding to the method embodiment shown, the device can be specifically implemented in various electronic devices.

[0118] like Figure 7 As shown, the data processing device 700 provided in this embodiment includes: a sending unit 701, a receiving unit 702, and a detection unit 703. The sending unit 701 can be configured to send a request for obtaining aggregated data to the metadata repository of the distributed file system, and the aggregated data is data created by the database index generation device described in the above embodiment. The receiving unit 702 can be configured to receive the aggregated data sent by the metadata repository. The detection unit 703 can be configured to perform quota detection on the aggregated data to obtain a quota detection result.

[0119] In this embodiment, in the data processing device 700, the specific processing of the sending unit 701, the receiving unit 702, and the detection unit 703 and the technical effects thereof can be referred to in Figure 4 The relevant descriptions of step 401, step 402, and step 403 in the corresponding embodiment are not repeated here.

[0120] In some optional implementations of this embodiment, the above-mentioned detection unit 703 is configured to obtain a preset quota value; detect whether the aggregated data is greater than the preset quota value, and obtain a quota detection result including that the aggregated data is greater than the preset quota value or the aggregated data is less than the preset quota value or the aggregated data is equal to the preset quota value.

[0121] In some optional implementations of this embodiment, the apparatus further includes: an alarm unit (not shown in the figure), which is configured to issue an alarm message in response to a quota detection result indicating that the aggregated data is greater than a preset quota value.

[0122] In the data processing device provided by the embodiment of the present disclosure, a sending unit 701 sends a request for obtaining aggregated data to the metadata library of the distributed file system; a receiving unit 702 receives the aggregated data sent by the metadata library; and a detection unit 703 performs quota detection on the aggregated data to obtain a quota detection result, which provides a reliable implementation method for realizing the quota function of the distributed file system and ensures the reliability of the usage calculation of the distributed file system.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0124] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0125] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0126] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0127] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a database index generation method or a data processing method. For example, in some embodiments, the database index generation method or the data processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the database index generation method or the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the database index generating method or the data processing method in any other appropriate manner (for example, by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable database index generating device or a data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a database index, the method comprising: Based on metadata of a metadata database in a distributed file system, determine items to be aggregated related to file quotas; Determining an item aggregation function based on the items to be aggregated; Creating an aggregate index in the metadata database; Performing statistics on the items to be aggregated according to the item aggregation function to obtain statistical data; The statistical data is stored as aggregated data in the aggregated index.

2. The method according to claim 1, wherein: The step of determining the items to be aggregated related to the file quota based on the metadata of the metadata base in the distributed file system includes: Get the calculation demand data of file quota; Obtaining key fields in the directory entry table and the target entry table of the metadata database of the distributed file system; The key fields are matched with the calculation requirement data to obtain items to be aggregated related to the file quota.

3. The method according to claim 1, wherein: The determining of the item aggregation function based on the items to be aggregated includes: Based on the items to be aggregated, select a function from a preset function set to obtain a selected function set; Based on the file quota calculation requirement data and the selected function set, an item aggregation function is obtained.

4. The method according to claim 1, wherein: The creating of the aggregate index comprises: Create an index data table; The corresponding relationship among the item aggregation function, the items to be aggregated and the index data table is recorded in the metadata database of the distributed file system.

5. The method according to claim 4, wherein: After creating the index data table, the method further includes: Determine the data length of statistical data; Allocate space for the index data table according to the data length.

6. The method according to any one of claims 1 to 5, wherein: The items to be aggregated include: The number and size of files under different file owners in the directory entry table of the metadata database; the item aggregation function includes: a sum function; The statistical data obtained by counting the items to be aggregated according to the item aggregation function include: using a summation function to sum the number of files under different file owners to obtain the number of files; using a summation function to sum the file sizes under different file owners to obtain a capacity value; and using the number of files and the capacity value as statistical data.

7. The method according to claim 6, wherein: The items to be aggregated also include: The index node of the directory item table of the metadata database obtains a modulus value modulo a set value; and the items to be aggregated are counted according to the item aggregation function to obtain statistical data, including: before calculating the statistical data, controlling the modulus value to be a positive integer.

8. A data processing method, the method comprising: Sending a request to a metadata repository of a distributed file system to obtain aggregated data, wherein the aggregated data is data created by the database index generation method according to any one of claims 1 to 7; Receiving aggregated data sent by the metadata repository; A quota check is performed on the aggregated data to obtain a quota check result.

9. The method according to claim 8, wherein: The performing quota detection on the aggregated data to obtain the quota detection result includes: Get the preset quota value; Detect whether the aggregated data is greater than the preset quota value, and obtain a quota detection result including that the aggregated data is greater than the preset quota value, or the aggregated data is less than the preset quota value, or the aggregated data is equal to the preset quota value.

10. The method according to claim 8, further comprising: In response to the quota detection result that the aggregated data is greater than a preset quota value, an alarm message is issued.

11. A database index generating device, the database index generating device comprising: an item determination unit, configured to determine items to be aggregated related to the file quota based on metadata of a metadata base in the distributed file system; A function determination unit, configured to determine an item aggregation function based on the items to be aggregated; A creating unit, configured to create an aggregate index in the metadata database; an obtaining unit, configured to perform statistics on the items to be aggregated according to the item aggregation function to obtain statistical data; The saving unit is configured to save the statistical data as aggregated data in the aggregated index.

12. The device according to claim 11, wherein The item determination unit is configured to: obtain calculation requirement data of file quota; obtain key fields in the directory item table and the target item table of the metadata library of the distributed file system; match the key fields with the calculation requirement data to obtain items to be aggregated related to the file quota.

13. The device according to claim 11, wherein: The function determination unit is configured to: based on the items to be aggregated, select a function from a preset function set to obtain a selected function set; based on the file quota calculation requirement data and the selected function set, obtain an item aggregation function.

14. The device according to claim 11, wherein: The creation unit is configured to: create an index data table; and record the corresponding relationship among the item aggregation function, the item to be aggregated, and the index data table in the metadata database of the distributed file system.

15. The device according to claim 14, wherein: The creation unit is further configured to: determine the data length of the statistical data; and allocate space for the index data table according to the data length.

16. The device according to any one of claims 11 to 15, wherein: The items to be aggregated include: the number of files and the file sizes under different file owners in the directory entry table of the metadata database; the item aggregation function includes: a summation function; the obtaining unit is configured to: use the summation function to sum the number of files under different file owners to obtain the number of files; use the summation function to sum the file sizes under different file owners to obtain the capacity value; and use the number of files and the capacity value as statistical data.

17. The device according to claim 16, wherein: The items to be aggregated further include: a modulo value of an index node of a directory entry table of the metadata library modulo a set value; and the obtaining unit is configured to: before calculating statistical data, control the modulo value to be a positive integer.

18. A data processing device, comprising: A sending unit, configured to send a request to a metadata repository of a distributed file system to obtain aggregated data, wherein the aggregated data is data created by the database index generating device according to any one of claims 11 to 17; a receiving unit, configured to receive the aggregated data sent by the metadata repository; The detection unit is configured to perform a quota detection on the aggregated data to obtain a quota detection result.

19. The device according to claim 18, wherein: The detection unit is configured to obtain a preset quota value; detect whether the aggregated data is greater than the preset quota value, and obtain a quota detection result including that the aggregated data is greater than the preset quota value or the aggregated data is less than the preset quota value or the aggregated data is equal to the preset quota value.

20. The apparatus according to claim 18, further comprising: The alarm unit is configured to issue an alarm message in response to the quota detection result that the aggregated data is greater than a preset quota value.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 10.

Images