Metadata access method and apparatus, device, storage medium, and program product

Abstract

The present application relates to the field of cloud technology, and discloses a metadata access method and apparatus, a device, and a storage medium. According to the method, a global metadata cache scheme is used, and relates to a three-layer architecture, i.e., a global cache file, a cache description array, and a memory block; storage areas can be divided according to the data length of metadata in a shared memory; and when target metadata needs to be acquired for a target process, metadata can be queried layer by layer according to the three-layer architecture until target address information of target metadata that has been cached is obtained, and the target address information is returned to the target process for realizing the access of the target metadata. In this way, the metadata required by all the working processes can be accessed by means of the method, so that metadata caches of the working processes do not need to be respectively maintained by the working processes, thereby reducing the repeated metadata cache between the working processes, reducing the occupied memory resources, reducing the transmission resource occupation of the repeated metadata, and improving the resource utilization rate.

Description

Metadata access method, device, equipment, storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 28, 2023, with application number 202311086103X and application name “Metadata Access Method, Device, Equipment and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to metadata access in the field of cloud technology. Background Art

[0003] With the development of cloud technology, cloud computing has become the new standard for enterprise applications. Cloud-native architecture provides an efficient way to develop, deploy, and manage applications in the cloud. Through continuous practice and exploration, the cloud-native model has gradually become recognized as the best practice for building and deploying cloud-native applications. Cloud-native databases, a type of cloud-native data infrastructure, leverage technologies such as resource decoupling and resource pooling to achieve high elasticity, high availability, and scalability. They support the high-performance, high-concurrency, and on-demand data management requirements of applications across various business domains.

[0004] In cloud-native mode, metadata is not stored on the computing cluster. When metadata is needed, it must be requested from the metadata server. Therefore, in order to reduce the frequency of metadata requests, it is necessary to maintain a metadata cache in the computing cluster.

[0005] Summary of the Invention

[0006] Embodiments of the present application provide a metadata access method, apparatus, device, and storage medium for reducing duplicate resources occupied by metadata cache and improving resource utilization.

[0007] In one aspect, a metadata access method is provided, the method comprising:

[0008] In response to a metadata query request from a target process, a matching operation is performed on a global cache file based on target index information of target metadata carried in the metadata query request; wherein the global cache file includes at least one record, each record including: index information of cached metadata and a slot identifier of a cache description array corresponding to the metadata, each cache description array corresponding to at least one memory block for caching metadata;

[0009] When the match is successful, accessing the target cache description array corresponding to the target metadata based on the target slot identifier associated with the target index information;

[0010] When it is determined based on the target cache description array that the target metadata is currently in a valid state, target address information of a target memory block that caches the target metadata is obtained from the target cache description array; wherein the target address information is used to indicate location information of the target memory block in a shared memory, and the shared memory is divided into storage areas for metadata according to different data lengths;

[0011] The target address information is returned to the target process.

[0012] In one aspect, a metadata access device is provided, the device comprising:

[0013] a global matching unit, configured to respond to a metadata query request from a target process and perform a matching operation on a global cache file based on target index information of target metadata carried in the metadata query request; wherein the global cache file includes at least one record, each record including: index information of cached metadata and a slot identifier of a cache description array corresponding to the metadata, each cache description array corresponding to at least one memory block for caching metadata;

[0014] An array access unit, configured to access a target cache description array corresponding to the target metadata based on a target slot identifier associated with the target index information when a match is successful;

[0015] An address obtaining unit is configured to obtain target address information of a target memory block that caches the target metadata from the target cache description array when determining that the target metadata is currently in a valid state based on the target cache description array; wherein the target address information is used to indicate location information of the target memory block in a shared memory, and the shared memory divides the storage area of ​​the metadata according to different data lengths; and return the target address information to the target process.

[0016] In one aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0017] In one aspect, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0018] In one aspect, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the steps of any of the above methods.

[0019] In the embodiment of the present application, a global metadata cache solution is adopted. In this cache solution, a three-layer architecture of global cache file, cache description array and memory block is adopted. The global cache file is used to store the index information of the cached metadata and the slot identifier of the corresponding cache description array to index the cache description array of the lower layer. The cache description array is used to store the cache-related description information of the metadata, and the memory block is used to actually cache the metadata. When the target process needs to obtain the target metadata, it can query downward layer by layer according to the above three-layer architecture until the target address information of the cached target metadata is obtained, and the target address information is returned to the target process to achieve access to the target metadata. In this way, all working processes can access the metadata they need according to this method, so there is no need to maintain their own metadata cache, which reduces the duplication of metadata caches between working processes, reduces the occupied memory resources, and also reduces the transmission resource occupation of duplicate metadata, thereby improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0021] FIG2 is a schematic diagram of an architecture related to metadata access provided in an embodiment of the present application;

[0022] FIG3 is a flow chart of a metadata access method provided in an embodiment of the present application;

[0023] FIG4 is a schematic diagram of a matching process according to an embodiment of the present application;

[0024] FIG5 is a schematic diagram of a comparison process provided in an embodiment of the present application;

[0025] FIG6 is a schematic diagram of a file description array set provided in an embodiment of the present application;

[0026] FIG7 is a schematic diagram of partitioning of a global cache file provided in an embodiment of the present application;

[0027] FIG8 is a schematic diagram of an example process of metadata access provided by an embodiment of the present application;

[0028] FIG9 is another schematic diagram of a flow chart of a metadata access method provided in an embodiment of the present application;

[0029] FIG10 is a schematic diagram of a free memory list provided in an embodiment of the present application;

[0030] 11A and 11B are another exemplary flowchart of metadata access provided by an embodiment of the present application;

[0031] FIG12 is another flowchart of the metadata access method provided in an embodiment of the present application;

[0032] 13A and 13B are schematic diagrams of another exemplary process of metadata access provided by an embodiment of the present application;

[0033] FIG14 is a schematic structural diagram of a metadata access device provided in an embodiment of the present application;

[0034] FIG15 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0035] FIG16 is a schematic diagram of the composition structure of another computing device using an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other in any way. In addition, although a logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0037] It is understandable that in the following specific implementation methods of this application, user data, such as user voice data, may be involved. Therefore, when the various embodiments of this application are applied to specific products or technologies, relevant licenses or consents need to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0038] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0039] Metadata: Metadata is information about the organization of data, data domains and their relationships. Simply put, metadata is data used to describe data, and contains descriptive information about data and information resources. For example, it refers to the meta-information used to manage business object storage data. The data in any file system is divided into data and metadata. Data refers to the actual data in ordinary files, while metadata refers to system data used to describe the characteristics of a file, such as access rights, file owners, and distribution information of file data blocks (inode...), etc. In a cluster file system, distribution information includes the location of the file on the disk and the location of the disk in the cluster. If a business object needs to operate a file, it must first obtain its metadata before it can locate the file and obtain the file's content or related attributes. In the embodiment of the present application, metadata includes but is not limited to: the name of the data set, various functional attribute settings, the address where the data copy Replica is stored, etc.

[0040] Global cache file: The global cache refers to the metadata cache shared between worker processes in a computing cluster. Each worker process maintains the same metadata cache through a certain mechanism, which can effectively merge requests for the same metadata between different worker processes and avoid frequent requests for data from the metadata server.

[0041] Database Definition Language (DDL): It is a language of Structured Query Language (SQL) that is responsible for defining data structures and database objects. It usually consists of three operation syntaxes: create (CREATE), modify (ALTER), and delete (DROP).

[0042] The embodiments of the present application involve cloud technology and are mainly designed based on metadata caching in cloud technology.

[0043] Cloud technology refers to a managed technology that unifies hardware, software, and network resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network, information technology, integration technology, management platform technology, and application technology, all based on the cloud computing business model. It can form a resource pool for on-demand, flexible, and convenient use. Cloud computing will become a crucial support. Backend services for technical network systems, such as those for video sites, image sites, and more portals, require significant computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Data of varying levels will be processed separately, and data from various industries will require robust system support, which can only be achieved through cloud computing.

[0044] With the development of cloud technology, it can often be implemented in conjunction with other technologies, often applied in the field of artificial intelligence (AI). AI refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0045] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Therefore, the implementation of AI is often accompanied by the application of cloud technologies, such as cloud computing, cloud storage, and databases.

[0046] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, these resources appear infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a pay-per-use basis.

[0047] As a provider of cloud computing infrastructure, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0048] Based on logical functional divisions, the Platform as a Service (PaaS) layer can be deployed on top of the Infrastructure as a Service (IaaS) layer, and Software as a Service (SaaS) can be deployed on top of the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software execution, such as databases and web containers. SaaS is a variety of business software, such as web portals and text messaging apps. Generally speaking, SaaS and PaaS are layers above IaaS.

[0049] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0050] Simply put, a database can be thought of as a digital filing cabinet, a place where electronic files are stored, allowing users to add, query, update, and delete data. A database is a collection of data that is stored in a specific way, can be shared with multiple users, has minimal redundancy, and is independent of applications.

[0051] The embodiments of the present application relate to a metadata caching solution in a cloud-native mode. Metadata refers to system data used to describe the characteristics of a file, such as access rights, file owners, and distribution information of file data blocks, etc. For example, in a cluster file system, the distribution information includes the location of the file on the disk and the location of the disk in the cluster. If a business object needs to operate a file, it must first obtain its metadata before it can locate the location of the file and obtain the file's content or related attributes. Generally speaking, cloud-native databases do not store metadata information on the computing cluster and need to request the required metadata from the metadata server. In order to reduce the frequency of requests, it is necessary to maintain a metadata cache in the computing cluster.

[0052] In the related art, the metadata caching solution commonly used is the Postgresql (PG) caching solution. In this solution, the working process that executes the query maintains a local metadata cache by itself. When the metadata expires, it requests new metadata from the metadata server. In this way, each working process in the computing cluster needs to request metadata from the metadata server. The same metadata may need to be obtained repeatedly. Not only does the same metadata need to repeatedly occupy transmission resources, but it also occupies memory resources in the computing cluster for repeated storage, which obviously greatly reduces resource utilization.

[0053] Furthermore, when executing a DDL operation, the changed metadata is organized into a set of invalidation messages and sent to other worker processes. Therefore, according to the PG caching solution, all caches of all worker processes need to be invalidated. Therefore, when a DDL operation occurs, invalidation messages need to be sent to all these worker processes, which obviously consumes more transmission resources. Furthermore, if a single data invalidation invalidates all metadata caches, it will obviously consume more resources to create new metadata caches, resulting in resource waste.

[0054] Based on this, an embodiment of the present application provides a metadata access method, in which a new metadata global caching solution is adopted. In this caching solution, a three-layer architecture of global cache file-cache description array-memory block is adopted. The global cache file is used to store the index information of the cached metadata and the slot identifier of the corresponding cache description array to index to the cache description array of the lower layer. The cache description array is used to store the cache-related description information of the metadata, and the memory block is used to actually cache the metadata.

[0055] Then, when the target process needs to obtain target metadata, it matches the target index information carried in its metadata query request in the global cache file. If a match is successful, it indicates that the target metadata has been cached. Then, the target cache description array is accessed through the target slot identifier associated with the target index information. When the target metadata is confirmed to be currently valid, the target address information of the target memory block caching the target metadata is obtained from the target cache description array and the target address information is returned to the target process, thereby achieving access to the target metadata. It can be seen that according to the above three-layer architecture, querying layer by layer until the target address information of the cached target metadata is obtained, the target metadata can be accessed. In this way, all working processes can access the metadata they need according to this method, eliminating the need to maintain their own metadata cache, reducing the duplication of metadata caches between working processes, reducing the memory resources occupied, and at the same time, reducing the number of requests made by the computing cluster to the metadata server. Repeated data requests from working processes are merged, thereby reducing the transmission resource usage of duplicate metadata and improving resource utilization.

[0056] In addition, in an embodiment of the present application, by partitioning the metadata, and the partitions corresponding to the corresponding partition version information, when a DDL operation occurs, it is not necessary to update the entire metadata cache, but only part of the metadata needs to be updated, and the validity of the metadata data status is described through version information and verification of the version information, thereby avoiding the invalidation of all metadata caches due to the invalidation of one data, and reducing the degree of resource waste.

[0057] In the embodiments of the present application, a complete set of access processes are designed for various situations involved in metadata caching, including situations where the accessed metadata is cached, not cached, and requires data replacement. This meets the global metadata access requirements of each work process in various situations, effectively merges requests for the same metadata information between different work processes, and then puts some information that can be protected by locking into the local cache, which can effectively avoid frequent requests for data from the metadata server due to lease expiration.

[0058] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios described below are only used to illustrate the embodiments of the present application and are not limiting. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0059] The solution provided by the embodiment of the present application can be applied to computing cluster scenarios, such as computing cluster scenarios using cloud native mode and non-cloud native mode. As shown in Figure 1, an application scenario diagram provided by the embodiment of the present application can include a metadata server 101 and a computing device set 102.

[0060] In one possible implementation, the computing device set 102 may be a computing cluster.

[0061] The metadata server 101 is used to store metadata. It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited to these.

[0062] A computing cluster is used for actual computing or application deployment. It can include multiple computing nodes, each of which can be used to deploy computing tasks or applications based on user deployment requests. Each computing node can be a physical server or any computer device with sufficient computing power, such as a mobile phone, tablet computer (PAD), laptop, desktop computer, smart TV, smart car device, smart wearable device, smart TV, aircraft, etc.

[0063] It should be noted that the metadata access method in the embodiments of the present application can be executed by computing nodes in a computing cluster. The computing nodes may include one or more processors, memory, and interactive I / O interfaces. The memory of the computing nodes may store the program instructions required to execute the metadata access method provided in the embodiments of the present application. When executed by the processor, these program instructions can be used to implement the metadata access process provided in the embodiments of the present application.

[0064] For example, when the target process on the computing node needs to access the target metadata, it can match the target index information carried in its metadata query request in the global cache file. If the match is successful, it indicates that the target metadata has been cached. Then, the target cache description array is accessed through the target slot identifier associated with the target index information. When it is confirmed that the target metadata is currently in a valid state, the target address information of the target memory block that caches the target metadata is obtained from the target cache description array, and the target address information is returned to the target process to achieve access to the target metadata.

[0065] In one possible implementation, computing device set 102 may be a terminal device set. The terminal device set may include multiple terminal devices, each of which may be a tenant's device. In a multi-tenant architecture, all tenants have their own metadata, and all tenant-related operations are driven by metadata. During metadata usage, each time a terminal device is used, queries and calculations must be performed from the database. Therefore, to reduce the frequency of metadata requests, the terminal device can employ the methods of the embodiments of the present application to maintain a global metadata cache. When metadata is needed, it can be queried from the local cache.

[0066] In the embodiment of the present application, the metadata server 101 and the computing device set 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network, for example, a mobile cellular network or a Wireless Fidelity (WIFI) network. Of course, other possible networks are also possible, and the embodiment of the present application does not limit this.

[0067] Below, in combination with the application scenarios described above, the method provided by the exemplary embodiment of the present application is described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this respect.

[0068] FIG2 is a schematic diagram of an architecture related to metadata access provided in an embodiment of the present application. Taking a computing cluster as an example, multiple worker processes may be deployed thereon, namely, worker processes 1 to N shown in FIG2 . Multiple worker processes share a set of metadata global caches. The metadata global cache uses the global cache data structure shown in FIG2 , which is composed of the following parts:

[0069] (1) Global cache file

[0070] The global cache file is used to record global cache information, specifically, to identify which metadata is cached. As shown in Figure 2, the global cache file can include multiple record locations, such as record locations 1 through M. Each record location can be used to store at least one record, and each record can be used to store at least one metadata-related index and the slot identifier of the cache description array associated with that metadata. Therefore, when querying for target metadata, one can query the global cache file to see if the corresponding index information is included. If a match is found, the associated slot identifier is obtained and the query can be continued.

[0071] In a possible implementation, the global cache file may be implemented in the form of a hash table, and each entry in the hash table may be used to store a record.

[0072] (2) Cache description array (cache desc)

[0073] As shown in Figure 2, the second layer of the data structure is the cache description array, which includes multiple slots, such as slot 1 to slot K as shown in Figure 2. Each slot can be used to store one or more cache description arrays. The cache description array contains control information of the cache metadata, describes the usage of the corresponding memory block, and information such as the memory offset corresponding to the metadata information.

[0074] Among them, each record in the global cache file will uniquely point to a slot identifier of a cache description array, so that when the match is successful in the global cache file, the slot identifier of the cache description information of the corresponding target metadata can be found, thereby accessing the cache description array corresponding to the slot identifier. As shown in Figure 2, the slot identifier found is "id=0", which points to the first slot (such as slot 1 in Figure 2), indicating that the cache description array of the currently required target metadata is stored in this slot, and the relevant information of the target metadata can be obtained by accessing the cache description array of this slot.

[0075] (3) Continuous memory

[0076] Continuous memory is used to cache metadata obtained from the metadata server. Since the data length of the metadata value varies, the continuous memory can be divided into memory blocks according to the different upper limits of the data length. Memory blocks of different sizes can be used to store metadata of different data lengths. For example, the memory can be divided according to 128, 256, 512, 1024, 2048, 4096, and 8192 bytes, and used as needed. The proportion of memory of each size can be set according to the proportion of commonly used metadata lengths. When metadata needs to be cached, the metadata server can be remotely called to obtain the corresponding metadata and cached in its own cache for subsequent direct local query retrieval.

[0077] Refer to the global cache data structure shown in Figure 2. In the hash table, the corresponding desc subscript will be found according to the index information of the matching metadata. According to the subscript, it can be mapped to the address information in the memory to obtain the corresponding metadata. The metadata cached in the memory comes from the metadata server.

[0078] Based on the above-mentioned global cache data structure, an embodiment of the present application provides a metadata access method. FIG3 is a flowchart illustrating the metadata access method provided by an embodiment of the present application. The method can be executed by a computer device, which can be the terminal device or computing node mentioned in FIG1 . The specific implementation process of the method is as follows:

[0079] Step 301: In response to a metadata query request from a target process, a matching operation is performed on a global cache file based on target index information of target metadata carried in the metadata query request.

[0080] In an embodiment of the present application, as shown in Figure 2, the global cache file includes at least one record, each record includes: index information of cached metadata, and a slot identifier of a cache description array corresponding to the corresponding metadata, and each cache description array corresponds to at least one memory block for caching metadata.

[0081] In one possible implementation, the metadata access process in the embodiment of the present application can be executed by each work process itself, that is, when each work process needs a certain target metadata, it can obtain the corresponding target metadata by executing the metadata access method provided in the embodiment of the present application.

[0082] In one possible implementation, a proxy process can be added to each computer device to implement metadata access. Each worker process can then access metadata by calling the proxy process. This allows each worker process to maintain a local metadata cache, all of which can be accessed through the proxy process. Consequently, when repeated metadata requests occur, the proxy process can consolidate them, reducing resource waste.

[0083] In one possible implementation, the implementation process of the embodiments of the present application can also be performed collaboratively by individual worker processes and proxy processes. For example, the worker process itself can perform metadata access, while the proxy process can perform tasks related to metadata cache maintenance. Of course, in actual applications, this configuration can be tailored to specific needs, and the embodiments of the present application do not impose any restrictions on this.

[0084] Specifically, when the target process needs to use the target metadata, the target process can initiate a metadata query request, which carries the target index information of the target metadata to be queried. Since the index information of the cached metadata is stored in the global cache file, by matching the target index information with it, it can be determined whether the target metadata has been cached locally.

[0085] Among them, metadata is usually stored through metadata files. For example, for common databases, metadata can be stored in the form of metadata tables. Because different tables of metadata services have different unique indexes, a metadata file identifier is maintained based on this information, which can be called cacheid. A cacheid uniquely corresponds to a metadata file. Each metadata file can contain multiple data columns. Different data columns are identified by different column index values, which are called keys.

[0086] When the target process performs a metadata query, the target index information passed in may include the metadata file identifier cacheid and the data column index value key corresponding to the target metadata in the metadata file. When matching, it is necessary to match based on the specific values ​​of cacheid and key. Only when both match can the match be considered successful.

[0087] In one possible implementation, the global cache file can be implemented using a hash table. A hash table, also known as a hash table, is a data structure that is directly accessed based on a key value. That is, records are accessed by mapping the key value to a location in the table to speed up the search. This mapping function is called a hash function, and the array that stores records is called a hash table.

[0088] Furthermore, in an embodiment of the present application, for the hash table maintenance process, that is, when caching target metadata, the data column index value key in the incoming target index information is calculated through a certain target mapping method to obtain the corresponding hash identifier hash value, and the hash value + cacheid is used as the hash key, and then the above-mentioned target index information and hash identifier are stored in the entry corresponding to the hash key, that is, the cache id, hash value and the specific value of the index column key are stored.

[0089] Then, when performing a query, referring to FIG4 , which is a schematic diagram of the matching process provided by an embodiment of the present application, the matching process is similar to the above-mentioned process of storing records, namely, first, based on the data column index value key in the input target index information, according to the target mapping method, a corresponding hash identifier hash value is generated, and based on the hash identifier hash value and the metadata file identifier cacheid, a query record identifier hash key corresponding to the target metadata is generated. The hash key can be used to locate the corresponding record position from the hash table, thereby determining the target record corresponding to the target metadata from the global cache file. Finally, each piece of information is matched one by one with the corresponding value in the target record. This is to take into account the possibility of hash collisions in the hash key calculated in the above process. Multiple records may be stored at the corresponding position of the same hash value. Therefore, to avoid matching errors, it is necessary to further check whether each piece of information is correct. For example, as shown in FIG4 , when the target record of the target metadata is determined to be stored at record position 3 according to the hash key, the metadata file identifier cacheid, the data column index value key, and the hash identifier hash value are matched with the corresponding values ​​included in the target record.

[0090] In a possible implementation, referring to FIG5 , which is a schematic diagram of a comparison process provided by an embodiment of the present application, when comparing a target record with the input target index information, the metadata file identifier cacheID may be compared first, that is, the input metadata file identifier 1 (cacheID 1) is compared with the metadata file identifier 2 (cacheID 2) in the target record (1 and 2 here are used to refer to different names and do not limit the difference in actual values ​​between the two) to determine whether the cacheID hits. If not, the match fails. If so, the hash identifier hashValue is compared, that is, the hash identifier 1 (hash value 1) calculated by the input data column index value key is compared with the hash identifier 2 (hash value 2) in the target record to determine whether the hash value hits. If not, the match fails. If so, the data column index value key is compared, that is, the input data column index value 1 (key 1) is compared with the data column index value 2 (key 2) in the target record to determine whether the key hits. If not, the match fails. If so, the match succeeds.

[0091] In the embodiment of the present application, different data types can use different mapping methods. Therefore, when mapping the hash value, it is necessary to find the corresponding target mapping method based on the type of the data column index value key.

[0092] Specifically, considering that the same key may have different data types in different metadata files, in order to facilitate the acquisition of each key type in each metadata file, a file description array set is maintained in the embodiment of the present application, as shown in FIG6 . The file description array set uses cacheid as an index, and each file description array defines information such as the key type under different cacheids, that is, it indicates the data type of each data column in the corresponding metadata file. Then, using cacheid as the identifier of the file description array for indexing, the corresponding file description array can be found through cacheid. By inputting the metadata file identifier cacheid, the corresponding target file description array can be obtained from multiple file description arrays in the file description array set. As shown in FIG6 , the corresponding file description array can be obtained through metadata file identifier 1 (cacheid 1), which contains the data type of each key in the metadata file of cacheid 1. Of course, in addition to the data type, other possible information can also be included, and the embodiment of the present application does not limit this.

[0093] Based on the target file description array, the target data type associated with the data column index value key is obtained. Based on this, the target mapping method corresponding to the target data type is determined. This method is then called to perform hash mapping on the data column index value key, obtaining a hash value. As shown in Figure 6, when the key is data column index value 3 (key 3), its data type is obtained as data type 3, and its corresponding target mapping method is mapping function 3. Mapping function 3 is then called to calculate the hash value.

[0094] Similarly, when comparing key values, considering that different data types have different comparison methods, the target data type associated with the key is also required to determine the comparison function used for comparison, and then call the determined comparison function to compare the key values.

[0095] Therefore, on the basis of using hash identifiers to speed up the search, taking into account the possible existence of hash collisions, metadata file identifiers, data column index values ​​and hash identifiers are used for secondary matching to ensure the accuracy of matching the target cache description array.

[0096] Step 302: When the match is successful, the target cache description array corresponding to the target metadata is accessed based on the target slot identifier associated with the target index information.

[0097] In the embodiment of the present application, when the index information stored in the target record is exactly the same as the target index information, it is considered that the match is successful.

[0098] In an embodiment of the present application, in addition to the index information, each record also stores the second layer in the global cache data structure, that is, the slot identifier of the cache description array. Then, after a successful match, the target slot identifier associated with the target index information can be obtained from the target record, and then the target cache description array corresponding to the target metadata can be accessed based on the target slot identifier.

[0099] Continuing with the hash table example above, the entry also stores an index corresponding to the cache description array at the next level. If the metadata file identifier cacheid, the data column index key, and the hash value all match, the target index information is considered a match. The index corresponding to the current entry is then retrieved, allowing the corresponding cache description array to be indexed.

[0100] In the embodiment of the present application, since it is a global cache data structure shared by multiple work processes, multiple work processes can access it. Therefore, in order to avoid conflicts when accessing between multiple work processes, it is necessary to lock the accessed data.

[0101] Specifically, when the target process accesses the global cache file, it needs to perform a first locking operation to add a shared lock to the global cache file. The presence of a shared lock indicates that a working process is currently accessing the global cache file. When some operations need to be performed on the global cache file, such as modifying or deleting the global cache file, it is necessary to wait for the access to end. When the target process finishes accessing the global cache file, for example, after obtaining the slot identifier of the cache description array, it needs to perform a corresponding unlocking operation to release the shared lock of the global cache file.

[0102] Considering that a single working process may cause lock conflicts after locking the entire global cache file, for example, during the period of its access, other working processes cannot write to the global cache file, and the waiting time is too long, which may cause task blocking. In order to improve the lock conflict situation, the embodiment of the present application adopts a partitioned global cache file, that is, the global cache file can include multiple file partitions, each file partition is used to store at least one record. As shown in Figure 7, a partition diagram of the global cache file provided by the embodiment of the present application is shown, wherein Figure 7 shows two partitions, partition A and partition B, each partition can contain multiple record locations, such as partition A contains record locations A1 to An, and partition B contains record locations B1 to Bn.

[0103] When accessing a global cache file, the target file partition containing the target record is determined based on the query record identifier hash key calculated above. A first lock operation is then performed on the target file partition to add a shared lock to the target file partition, indicating that a worker process is currently accessing the target file partition. Other partitions will not be locked. Similarly, after accessing the target file partition, an unlock operation is also required to release the shared lock on the target file partition.

[0104] For example, when the calculated hash key is located in partition A, partition A is locked. When partition B is not locked, partition B can be written by other processes, avoiding lock conflicts caused by locking the entire global cache file.

[0105] In the embodiments of the present application, a locking operation refers to setting a corresponding flag position to information representing a corresponding lock. For example, a target file partition may include its corresponding shared lock flag position. When the target file partition is locked, the shared lock flag position is set to a locked state, or information about the target process is added to the flag position to indicate that the target process has locked it. Alternatively, a locking operation may also refer to applying for a shared lock on the target file partition when accessing the target file partition. When a shared lock is issued to the target process, the locking is considered successful.

[0106] It should be noted that the target process access mentioned in the embodiments of the present application may include two ways, namely, the target process itself accesses data, or it may refer to the target process accessing data through a proxy process. The embodiments of the present application do not impose any restrictions on this.

[0107] Step 303: Determine the current data status of the target metadata based on the target cache description array.

[0108] In the embodiment of the present application, the cache description array is used to store control information of the cached metadata, describing the usage of the corresponding memory block, and information such as the memory offset corresponding to the metadata information. Each cache description array may contain one or more of the following information:

[0109] (1) The state field is the data description of the current cache description array, which can include information such as the desc lock status, reference count (refcount) parameter, access count (usecount) parameter, status, etc. For example, the above information can be carried by 32 bits, that is, the above information can rely on a 32-bit data description.

[0110] Among them, the desc lock state represents whether it is locked. The lock is implemented by atomic operation. When performing some operations on the cache description array, the lock state of the state field needs to be updated. For example, when modifying the bit status on refcount and usecount, it is necessary to hold the lock.

[0111] The refcount parameter is used for reference counting, that is, when accessing the cache description array, refcount is increased by 1, and after the access is completed, refcount is reduced by 1; usecount describes the number of accesses after loading into the cache, and it can usually be configured with a maximum value, such as 5, which is mainly used for detection during memory replacement.

[0112] Status: When the cache is in progress, the process representation is used to indicate that the data has been read and is being copied to the cache, waiting for other work processes to complete before being read. The valid status indicates that the current metadata is valid and can be read.

[0113] (2) The metadata partition identifier (part_num) field represents the metadata partition information. Based on this partition division, only the information of some partitions needs to be invalidated each time, without invalidating all caches.

[0114] (3) Partition version information (part_version) field, which represents the version number of the current metadata partition. Usually, part_num and part_version need to be used in combination. Different part_nums correspond to different version numbers. They need to be compared with the globally stored version information to determine whether the current metadata is valid.

[0115] (4) Metadata validity expiration time (lease) field. Lease is a periodic time used to represent the expiration time of metadata. Metadata is valid before this expiration time. Expired data is unreliable and needs to be retrieved from the metadata server for caching.

[0116] (5) Memory address (offset) field, which represents the address information of the metadata cache and can be represented by a memory offset address.

[0117] (6) Data lock (content_lock) field: When writing metadata obtained from the metadata server, an exclusive lock is required, and when reading metadata, a share lock is required.

[0118] (7) Next field: When the metadata is too long to be cached in a single memory block, for example, the 8KB space in a single memory block cannot store it, or when the input is not a unique index information and the returned result is a list, this field can be used to store the index of the next value.

[0119] Furthermore, when accessing the cache description array, the target metadata partition identifier corresponding to the target metadata can be obtained from it, and then it can be determined whether the version information of the partition identifier is consistent with the global storage. The global storage can refer to a separate set of version information maintained in the current device to store the latest version information of each metadata partition; alternatively, the global storage can also obtain the latest version information from the metadata server; alternatively, the metadata server's DDL operations can be subscribed to. When a DDL operation is executed, the metadata partition targeted by the operation can be known, and the version information of the corresponding metadata partition can be modified.

[0120] In the cache description array, the partition version information corresponding to the target metadata partition identifier is also stored. By comparing the two, it can be determined whether the versions are consistent and whether the target metadata is valid.

[0121] Specifically, based on the target metadata partition identifier corresponding to the target metadata, the latest partition version information of the target metadata partition can be obtained, and then the latest partition version information can be compared with the storage partition version information in the target cache description array. If the two are consistent, it indicates that the currently cached target metadata has expired and needs to be re-acquired; if the two are consistent, and based on the lease in the cache description array, it is determined that the target metadata is currently valid; if the two are consistent, and based on the lease in the cache description array, it is determined that the target metadata is currently expired, it is also determined that the target metadata is currently invalid.

[0122] Before returning the target address information, by identifying whether the target metadata is valid or not, the correctness of the response to the metadata query request can be greatly improved, avoiding the target progress from obtaining incorrect or invalid target metadata, and improving the response accuracy.

[0123] In this embodiment, to assist in determining version information, an auxiliary process can be added to periodically obtain GTS services from the Global Transaction Service (GTS) server, convert them into timestamps using a specific conversion method, and the backend process compares this timestamp with the cached lease to determine whether the current cache is invalid. Simultaneously, the auxiliary process checks the etcd middleware to obtain the version numbers corresponding to each part of the metadata information for use in the aforementioned comparison.

[0124] Step 304: When the data state is valid, obtain the target address information of the target memory block that caches the target metadata from the target cache description array, and return the target address information to the target process.

[0125] Among them, the target address information is used to indicate the location information of the target memory block in the shared memory. The shared memory is the continuous memory mentioned above, which is used for metadata shared cache. In order to facilitate the storage and management of metadata, the metadata storage area can be divided according to different data lengths, and then different metadata can be stored in memory blocks of corresponding sizes when caching metadata.

[0126] Specifically, when the target metadata is currently in a valid state, the specific value of the offset field can be provided to the target process from the target cache description array, so that the target process can access the target metadata based on the memory address.

[0127] In an embodiment of the present application, for each cache description array, when accessing the cache description array, it is necessary to update the reference count parameter refcount and the access count parameter usecount in the corresponding cache description array, and when the access ends, it is also necessary to update the reference count parameter refcount in the corresponding cache description array. For example, when accessing the cache description array with ID 0, it is necessary to perform a desc lock operation on it, add a desc lock to it, and then perform a refcount++ operation and a usecount++ operation, that is, add one to both refcount and usecount. When the access ends, for example, after the target address information has been obtained and returned to the target process, a refcount-- operation is performed, that is, refcount is subtracted by one, and the access is exited.

[0128] The following describes the technical solution of the embodiment of the present application using a specific example of metadata access. FIG8 is a flow chart of accessing cached metadata provided by the embodiment of the present application, which includes the following steps:

[0129] Step (1): The backend process passes in the cache ID and key of the target metadata to be queried. Based on the passed in cache ID, the hash value calculation function is found, and the hash value is calculated based on the key value. Based on the calculated hash value, the share lock of the corresponding file partition is obtained, that is, the share lock is added to the corresponding file partition, that is, the share lock is added to the partition where record positions 1 to 3 are located.

[0130] Step (2): Find the corresponding desc subscript based on the passed key.

[0131] Specifically, when searching, first compare cacheid, then compare hashvalue, and finally compare according to the value of the passed key. If all match, it is determined that the target metadata of the current query has been cached and corresponds to the record in the hash table, and then obtain the desc subscript, which is the slot identifier of the cache description array.

[0132] Since hash collisions may occur, multiple records may be stored at the corresponding position of the same hash value. As shown in Figure 8, two records corresponding to metadata are stored at record position 2 at the same time, namely "id=0" and "id=3" as shown in Figure 8. Therefore, when matching, after finding the corresponding record position through the hash value or hash key, it is necessary to further match the value of the passed key with each record at the record position to determine whether a match is found. As shown in Figure 8, the record with "id=3" at record position 2 is finally hit. The slot identifier of the cache description array represented by "id=3" points to slot 3.

[0133] Step (3): According to the subscript "id=3", access cache desc on slot 3 to determine the data status of the corresponding target metadata. Cache desc stores the memory offset. When the metadata is valid, the memory offset address of the target metadata can be obtained from it.

[0134] Specifically, if the part_version corresponding to the part_num cached in the current cache description array is consistent with the global storage, and the lease has not expired, the data status is valid, then the corresponding desc lock is added to the cache description array, and refcount++ and usecount++ operations are performed.

[0135] Among them, part_version will change when the corresponding metadata changes, and it is necessary to verify whether it is the latest. Lease is a regular time. Expired data is unreliable and needs to be retrieved. Therefore, the target metadata is determined to be valid only when both conditions are met.

[0136] Step (4): Release the share lock of the corresponding file partition in the hash table.

[0137] Step (5): Return the memory offset address of the target metadata to the backend process.

[0138] In an embodiment of the present application, when querying target metadata, it is possible that the target metadata is not cached locally. In this case, it is necessary to obtain the target metadata and cache it locally before returning the corresponding memory address.

[0139] FIG9 is another flow chart of the metadata access method provided in an embodiment of the present application, which includes the following steps:

[0140] Step 901: In response to a metadata query request from a target process, a matching operation is performed on a global cache file based on target index information of target metadata carried in the metadata query request.

[0141] The process of step 901 is similar to the process of step 301 shown in FIG. 3 , so please refer to the introduction in the aforementioned part and will not be repeated here.

[0142] Step 902: When no match is successful, a metadata acquisition request is sent to the metadata server.

[0143] Specifically, when matching is performed based on the input target index information, if any information contained therein does not match, it means that the match is not successful.

[0144] For example, when the target index information includes cacheid and key, the hash value calculation function is found according to the incoming cacheid, and the hashvalue is calculated using the calculation function based on the key value. The hashvalue and cacheid are combined to obtain the hashkey, and then the corresponding target record location is determined. The share lock of the target file partition where the target record location is located is added, and then the cacheid, hashvalue and key values ​​are compared in turn. If any of them does not match, it is determined that the match is not successful.

[0145] If no match is found, indicating that the target metadata has not yet been cached, the shared lock on the target file partition is released, and a second lock operation is performed on the target file partition where the target record is located, thereby adding an exclusive lock to the target file partition. The exclusive lock prohibits other processes except the target process from accessing the target file partition, thus avoiding conflicts with other processes when writing to the target record location.

[0146] After adding the exclusive lock, the target record can be updated based on the target index information and the target record status can be marked as cache in progress. For example, the cacheid, hashvalue, and key values ​​are first stored in the target record location and the status is marked as process, while the slot identifier of the cache description number is not set yet.

[0147] After the writing of the above information is completed, the target file partition is unlocked to release the exclusive lock of the target file partition, and a metadata acquisition request is sent to the metadata server to obtain the target metadata.

[0148] Step 903: Receive the target metadata returned by the metadata server in response to the metadata acquisition request, and cache the target metadata in a free memory block.

[0149] In an embodiment of the present application, if there is a free memory block, the obtained target metadata can be directly stored in the free memory block.

[0150] In actual applications, a free list may be maintained, and when it is necessary to determine whether there is a free memory block, the free list may be queried.

[0151] In a possible implementation, the free list may be a free memory list for storing information about free memory blocks. It may then be queried whether the free memory list is empty to determine whether a free memory block exists.

[0152] Specifically, since memory blocks are divided into different sizes, the free memory list can also be stored according to the memory block size partition. As shown in Figure 10, according to the upper limit of the storage data, the memory blocks can be divided into 128, 256, 512, 1024 and 2048, etc., and the free memory blocks of each size can be maintained separately and stored in their corresponding sublists. When determining whether there are free memory blocks, the data length of the target metadata can be used to start traversing from the appropriate memory block size, and traverse in sequence to determine whether there are free memory blocks. For example, if the data length of the target metadata is 489k, it can start from the memory block size of 512 and first traverse to see if there are free memory blocks of 512. If not, continue traversing to see if there are free memory blocks of 1024 until the traversal is completed or a free memory block is found.

[0153] In a possible embodiment, the free list can be a free slot list, and each slot corresponds to a memory block in a one-to-one manner. Therefore, it is also possible to determine whether there are free slots to determine whether there are free memory blocks. Similar to the structure of Figure 10, the process of determining free slots can also be similar to the process of determining free memory blocks above, so it will not be described in detail.

[0154] In an embodiment of the present application, the free list can be maintained by adding an auxiliary process, which can regularly organize cache desc and maintain multiple free lists. Different memory sizes are placed in different free lists, and are initially placed in the corresponding free list.

[0155] Specifically, the auxiliary process may traverse the cache description array set at a preset interval to determine whether the slot identifier of the currently traversed cache description array is in the free slot list. If the slot identifier of the currently traversed cache description array is not in the free slot list, the auxiliary process obtains the refcount parameter and the usecount parameter of the cache description array and sorts them according to the following conditions:

[0156] (1) If the obtained refcount parameter and usecount parameter are both zero, indicating that no process has accessed the cache description array for a long time, the slot identifier of the cache description array is added to the free slot list.

[0157] (2) If the obtained reference count parameter is zero and the access count parameter is not zero, it indicates that a process has recently accessed the cache description array, but no process is currently accessing it, then the access count parameter is decremented by one.

[0158] In actual applications, the auxiliary process can periodically scan the cache desc array, for example, first scanning the shortest free list slots. If a slot has both a refcount of 0 and a usecount of 0, it is placed on the free list. If a slot has a refcount of 0 but a non-zero usecount, the usecount is decremented by 1. During this process, the hash entry corresponding to the slot does not need to be removed in advance; the decision can be made when the slot is actually used.

[0159] Accordingly, when a worker process needs to acquire a free memory block, it prioritizes slots from the free list of the corresponding length. If the free list is empty, the worker process must be woken up to sort the list. Slot selection is still based on the refcount and usecount requirements of 0. If no slots of the specified length have a refcount of 0 after two iterations, the worker process selects the next slot of the specified length. If a slot has a refcount of 0 but a non-zero usecount, the worker process continues the traversal, decreasing the usecount until a slot that meets the requirements is found.

[0160] Step 904: Update the cache description array corresponding to the free memory block, and update the target record based on the slot identifier of the cache description array.

[0161] Accordingly, after storing the target metadata in the free memory block, the cache description array needs to be updated accordingly.

[0162] Specifically, when updating and writing the cache description array, it is necessary to add a desc lock to the cache description array, and then perform refcount++ and usecount++ operations to set its state from the process state to the valid state, indicating that it has been cached and can be read, and fill in part_num, part_version and other information with the current latest information.

[0163] It should be noted that when no metadata is stored in a free memory block, the corresponding cache description array does not store data, or the cache description information stored is related to the previously stored metadata and has not been reset. Therefore, after storing the target metadata in the free memory block, the information related to the target metadata needs to be filled in.

[0164] Similarly, after the cache description array is updated, the corresponding target record also needs to be updated, that is, the slot identifier of the cache description array is stored in the target record, so that the cache description array can be correctly found based on the target record when the target metadata is subsequently queried.

[0165] Specifically, when writing the target record, in order to avoid conflicts with operations of other work processes, it is necessary to perform a second locking operation on the target file partition where the target record is located, that is, after adding an exclusive lock, update the slot identifier of the cache description array to the target record.

[0166] Step 905: Return the target address information of the free memory block to the target process.

[0167] Specifically, after completing the caching of the target metadata, the target address information of the free memory block can be returned to the target process, so that the target process can obtain the target metadata from the free memory block (here used to refer to the memory block storing the target metadata, and is not limited to the real free memory block) based on the target address information.

[0168] The following describes the technical solution of the embodiment of the present application using a specific example of metadata access. FIG. 11A and FIG. 11B show a flowchart of accessing uncached metadata provided by the embodiment of the present application, where there is currently free memory to store the metadata. The flowchart includes the following steps:

[0169] Step (1) (not shown in the figure): the backend process passes in the cache ID and key of the target metadata to be queried, finds the hash value calculation function based on the passed-in cache ID, and calculates the hash value based on the key value. According to the calculated hash value, the share lock of the corresponding file partition is obtained, that is, the share lock is added to the corresponding file partition. For example, if the record position corresponding to the hash value or hash key calculated in Figure 11A is record position 5, then the share lock is added to the hash partition where record positions 4 to 6 are located.

[0170] Step (2) (not shown in the figure): a match is performed in record position 5. No match is successful, that is, the target metadata is not found, indicating that the target metadata has not been cached, and the share lock of the hash partition is released.

[0171] Step (3): Add an exclusive lock to the hash partition and prepare to write data.

[0172] Step (4): Store the cacheid, hash value, and key values ​​in the entry of the hash table, specifically the record position corresponding to the above hashvalue or hashkey, that is, record position 5, and mark the state in the target record in record position 5 as process. The cache description array has not yet been determined, so the slot identifier of cache desc is not set yet, corresponding to "id=?" in Figure 11A, indicating that the slot identifier is still unknown.

[0173] Step (5): Release the exclusive lock of the hash partition.

[0174] Step (6): Request the target metadata from the metadata server. According to the data length of the target metadata, obtain a free memory block to store the target metadata, add a desc lock to the cache description array corresponding to the free memory block, perform refcount++ and usecount++ operations, set the status in the cache description array to valid, and fill in part_num, part_version and other information.

[0175] As shown in Figure 11A, based on the data length, the memory block corresponding to slot 7 is determined as a free memory block for storing the target metadata, and the metadata service is remotely called, that is, the target metadata is obtained from the metadata server. After the target metadata is stored, the information in the cache description array on slot 7 is updated.

[0176] Step (7): To add the slot identifier of the cache description array to the target record, an exclusive lock on the hash partition is required, indicating that data needs to be written. That is, an exclusive lock is added to the hash partition where record position 5 is located.

[0177] Step (8): The slot identifier of the cache description array is filled into the target record. Referring to FIG11B , the desc subscript, that is, slot 7, is filled into the entry. As shown in FIG11B , after filling in "id=7", subsequent queries can be indexed to slot 7 based on the target record at record position 5.

[0178] Step (9): Release the exclusive lock of the corresponding hash partition in the hash table.

[0179] Step (10): Return the memory offset address of the target metadata to the backend process.

[0180] In the embodiment of the present application, when querying the target metadata, there may be a situation where the target metadata is not cached locally. In this case, it is necessary to obtain the target metadata and cache it locally before returning the corresponding memory address. However, it is possible that when storing the metadata, there is no free memory block to directly store the metadata. In this case, it is necessary to replace it, that is, to overwrite the previously stored metadata to store the metadata currently needed. Referring to Figure 12, another flowchart of the metadata access method provided by the embodiment of the present application is shown, which includes the following steps:

[0181] Step 1201: In response to a metadata query request from a target process, a matching operation is performed on a global cache file based on target index information of target metadata carried in the metadata query request.

[0182] The process of step 1201 is similar to the process of step 301 shown in FIG. 3 , so please refer to the introduction in the aforementioned part and will not be repeated here.

[0183] Step 1202: When no match is found, a metadata acquisition request is sent to the metadata server.

[0184] Step 1203: If there is no free memory block, determine a target cache description array that meets a preset replacement condition from the cache description array set.

[0185] When all memory blocks are full, there are no free memory blocks. At this time, replacement is required. The meaning of replacement is to replace the memory block that originally stored other metadata with the target metadata of the current query.

[0186] Specifically, since the memory block corresponds to the cache description array, it is possible to determine whether the replacement condition is met based on the cache description array. Generally speaking, the memory block that can be replaced should not have been accessed for a long time, and the popularity of these metadata is not high, so it can be replaced. For each cache description array, when it is accessed by the target process, the refcount parameter and usecount parameter therein will be updated. Therefore, the refcount parameter and usecount parameter in the cache description array can be used to characterize the popularity of the metadata and whether there is a process accessing it. Therefore, the preset replacement condition can be that the values ​​of the refcount parameter and the usecount parameter are both 0.

[0187] Therefore, when looking for a replaceable memory block, a target cache description array whose reference count parameter and access count parameter are both zero can be determined from the cache description array set.

[0188] In one possible implementation, since memory blocks can be divided into different partitions, the cache description arrays can also be grouped in a corresponding manner, so that the cache description array set can be divided into multiple array partitions, each array partition including: a cache description array of a corresponding type of memory block, and different types of memory blocks have different upper limits on the cache data length.

[0189] Then, when a target cache description array that meets the preset replacement conditions is determined, the corresponding target array partition can be determined based on the data length of the target metadata, and a target cache description array that meets the preset replacement conditions can be determined from the target array partition, so that the upper limit of the data length of the determined target memory block can be adapted to the target metadata to avoid exceeding the storage size. If a target cache description array that meets the conditions does not exist in the target array partition, the system can wait and, after the wait times out, determine a target cache description array that meets the conditions from other types of partitions.

[0190] For example, slots are prioritized from the free list of the corresponding length. If the free list is empty, the auxiliary process needs to be activated to sort the list. The conditions for selecting a slot are still 0 refcount and 0 usecount. If no slots of the corresponding length have 0 refcount after two traversals, the next slot can be selected for traversal. If a slot has 0 refcount but not 0 usecount during the traversal, the traversal must continue, decreasing the usecount until a slot that meets the conditions is found.

[0191] It should be noted that, in actual application, you can first request the target metadata from the metadata server, and then determine whether there are free memory blocks; or, you can also determine whether there are free memory blocks, and if not, determine the target memory blocks that can be replaced, and then request the target metadata from the metadata server. The embodiments of the present application do not impose any restrictions on this.

[0192] Step 1204: Generate an original record identifier corresponding to the target cache description array based on the original index information in the target cache description array.

[0193] In actual applications, even if the target cache description array is not currently being accessed by a process, it is impossible to determine whether the corresponding record is currently being accessed by a process. If the record is being accessed by another process, the current target cache description array cannot be replaced. Since the found target cache description array stores the original description information of the original metadata, the original record identifier corresponding to the target cache description array can be generated based on the original index information stored therein, thereby finding the original record identifier corresponding to the original metadata.

[0194] Step 1205: When the original record corresponding to the original record identifier meets the clearing condition, the information stored in the original record is cleared.

[0195] The clearing condition is similar to the above. It is necessary to ensure that no other process is currently accessing the data. Therefore, the clearing condition can mean that no other process is currently accessing the data. When the original record meets the clearing condition, since the original metadata corresponding to the record has been replaced with other metadata, the information stored in the original record can be cleared to avoid subsequent incorrect access. The confirmation of whether there are other processes accessing the data can be confirmed by the refcount parameter in the cache description array. Since the current target process is accessing the data, the refcount parameter should be 1. If the refcount parameter is greater than 1, it indicates that another process is accessing the data.

[0196] Step 1206: Cache the target metadata into the target memory block corresponding to the target cache description array, and update the target record determined based on the target index information based on the slot identifier of the target cache description array.

[0197] When it is determined that replacement is possible, the obtained target metadata can be cached in the target memory block corresponding to the target cache description array just determined, and based on the slot identifier of the target cache description array, the target record determined based on the target index information is updated so that the corresponding target metadata can be queried based on the target record in subsequent queries.

[0198] Step 1207: Return the target address information of the target memory block to the target process.

[0199] Specifically, after the target metadata is cached, the target address information of the target memory block may be returned to the target process, so that the target process obtains the target metadata from the target memory block based on the target address information.

[0200] When there is no free memory block, by determining the appropriate target cache description array, the target metadata currently needed is replaced into the target memory block through memory block replacement. This not only ensures a timely response to the current metadata query request, but also the replaced information will not affect the overall situation, effectively improving system efficiency.

[0201] The following describes the technical solution of the embodiment of the present application using a specific example of metadata access. FIG. 13A and FIG. 13B illustrate a flowchart of accessing uncached metadata when there is currently no free memory to store the metadata, provided by the embodiment of the present application. The flowchart includes the following steps:

[0202] Step (1) (not shown): The root backend process passes in the cache ID and key of the target metadata to be queried. Based on the passed-in cache ID, it finds the hash value calculation function and calculates the hash value based on the key value. Based on the calculated hash value, it obtains the share lock of the corresponding file partition, that is, adds a share lock to the corresponding file partition. For example, if the record position corresponding to the calculated hash value or hash key in Figure 11A is record position 5, then a share lock is added to the hash partition where record positions 4 to 6 are located. If the target metadata cannot be found and there is no free memory, the share lock of the hash partition is released.

[0203] Step (2): Request data through the Remote Procedure Call Protocol (RPC). According to the data length, select the cache desc array of the corresponding length area, find a slot with refcount = 0 and usecount = 0, add a desc lock to it, and perform a refcount++ operation. See Figure 13A, where slot 7 corresponding to the 7th memory block is found.

[0204] Step (3): Calculate the old hash value based on the key stored in the metadata corresponding to slot 7, and calculate the new hash value based on the key of the target metadata to be queried.

[0205] Step (4): Because the record corresponding to the original data in slot 7 needs to be cleared and the record for the current target metadata needs to be filled, exclusive locks need to be added to the old hash table partition and the new hash table partition. As shown in Figure 13A, the old hash value points to record position 5, so it is necessary to lock the partitions 4 to 6 where it is located. The new hash value points to record position 9, so it is necessary to lock the partitions 8 to 10 where it is located.

[0206] Step (5): Fill in the entry corresponding to the new hash value calculated in the current target metadata. Do not fill in the cache desc id first. Mark the state as process. That is, fill in the relevant information of the current target metadata except the cache desc id in record position 9.

[0207] Step (6): Check if the refcount of desc is 1, indicating that no other process has accessed it except this worker process. If so, remove the old entry pointed to by the old hash value. If not, find an available slot again.

[0208] Step (7): Release the partition lock corresponding to the old entry, that is, the partition lock corresponding to partitions 4 to 6.

[0209] Step (8): Fill the target metadata read from the metadata server into the corresponding memory.

[0210] Step (9): Update the cache desc id of the new entry pointed to by the new hash value, as shown in Figure 13B. After the update, the record stored at position 9 points to slot 7, which is used to store the target metadata related information of the current query.

[0211] Step (10): Release the hash table partition lock, that is, the partition lock corresponding to partitions 8 to 10.

[0212] In the embodiment of the present application, when the work process reads metadata, all work processes that read metadata will add the corresponding read lock, and the DDL operation needs to add the corresponding write lock. At this time, wait for the read lock to be released. Under normal circumstances, it is necessary to wait for the lease time (the cache of other processes will automatically expire after the lease time) before updating. However, the implementation of this application introduces optimization, adds a version number, and manages metadata partitions. The metadata partition involved in DDL has a version number + 1. The partition number and partition version number are stored in the cache of the computing cluster. When the version numbers are inconsistent, it is necessary to obtain data from the metadata server again. In this way, the metadata server does not need to wait for the lease time before doing DDL, but only needs to see that the version number of each computing cluster node has been updated. In the embodiment of the present application, the work process maintains a local cache, and can also merge repeated metadata requests from multiple processes by adding a proxy process.

[0213] To sum up, a new metadata caching solution is provided in the embodiment of the present application. By maintaining a global cache, requests for the same metadata information between different working processes are effectively merged, and then some information that can be protected by locking is placed in the local cache, which can effectively avoid frequent requests for data from the metadata server due to lease expiration. It can be used in cloud-native mode or non-cloud-native mode. When the PG in multi-process mode uses the global cache, it can effectively reduce memory usage and avoid the working process reading the same metadata from the disk multiple times. The invalidation message does not need to be sent to all processes. The working process that executes DDL can invalidate the data corresponding to the current global cache.

[0214] Referring to FIG. 14 , based on the same inventive concept, an embodiment of the present application further provides a metadata access device 140 , which includes:

[0215] The global matching unit 1401 is configured to respond to a metadata query request from a target process and perform a matching operation on a global cache file based on target index information of target metadata carried in the metadata query request. The global cache file includes at least one record, each record including index information of cached metadata and a slot identifier of a cache description array corresponding to the metadata, each cache description array corresponding to at least one memory block for caching metadata.

[0216] The array access unit 1402 is configured to access the target cache description array corresponding to the target metadata based on the target slot identifier associated with the target index information when the match is successful;

[0217] The address obtaining unit 1403 is used to obtain the target address information of the target memory block that caches the target metadata from the target cache description array when determining that the target metadata is currently in a valid state based on the target cache description array; wherein the target address information is used to indicate the location information of the target memory block in the shared memory; and return the target address information to the target process.

[0218] In a possible implementation, the target index information includes a metadata file identifier and a data column index value corresponding to the target metadata in the metadata file; the global matching unit 1401 is specifically configured to:

[0219] Based on the data column index value, according to the target mapping method, a corresponding hash identifier is generated, and based on the hash identifier and the metadata file identifier, a query record identifier corresponding to the target metadata is generated;

[0220] Based on the query record identifier, determine the target record from the global cache file, and match the metadata file identifier, data column index value, and hash identifier with the corresponding values ​​included in the target record;

[0221] The array access unit 1402 is specifically configured to:

[0222] When the metadata file identifier, the data column index value, and the hash identifier all match successfully, the target cache description array corresponding to the target metadata is accessed based on the target slot identifier associated with the target index information.

[0223] In a possible implementation, the global matching unit 1401 is specifically configured to:

[0224] Based on the metadata file identifier, obtaining a corresponding target file description array from a plurality of file description arrays, wherein each file description array indicates: a data type of each data column in the corresponding metadata file;

[0225] Based on the target file description array, obtain the target data type associated with the data column index value;

[0226] Call the target mapping method corresponding to the target data type, perform hash value mapping on the data column index value, and obtain a hash identifier.

[0227] In a possible implementation, the global cache file includes multiple file partitions, each file partition includes at least one record; the global matching unit 1401 is further configured to:

[0228] Based on the query record identifier, determine the target file partition where the target record is located;

[0229] Performing a first locking operation on the target file partition to add a shared lock to the target file partition; wherein the shared lock indicates that a working process is accessing the target file partition;

[0230] When the match is successful, based on the target slot identifier associated with the target index information, after accessing the target cache description array corresponding to the target metadata, the target file partition is unlocked to release the shared lock of the target file partition.

[0231] In a possible implementation, the device further includes a data cache unit 1404, configured to:

[0232] If no match is found, a metadata acquisition request is sent to the metadata server;

[0233] receiving target metadata returned by the metadata server in response to the metadata acquisition request, and caching the target metadata in a free memory block;

[0234] Update the cache description array corresponding to the free memory block, and update the target record based on the slot identifier of the cache description array;

[0235] Returns the target address information of the free memory block to the target process.

[0236] In a possible implementation, the data cache unit 1404 is specifically configured to:

[0237] If the match fails, a second locking operation is performed on the target file partition where the target record is located, so as to add an exclusive lock to the target file partition; wherein the exclusive lock indicates that other processes except the target process are prohibited from accessing the target file partition;

[0238] Based on the target index information, the target record is updated and the status of the target record is marked as cached;

[0239] Unlock the target file partition to release the exclusive lock of the target file partition, and send a metadata acquisition request to the metadata server to obtain the target metadata;

[0240] After performing the second locking operation on the target file partition, the slot identifier of the cache description array is updated to the target record.

[0241] In a possible implementation, the data cache unit 1404 is further configured to:

[0242] If there is no free memory block, then determining a target cache description array that meets a preset replacement condition from the cache description array set;

[0243] Generate an original record identifier corresponding to the target cache description array based on the original index information in the target cache description array;

[0244] When the original record corresponding to the original record identifier meets the clearing condition, clear the information stored in the original record;

[0245] Cache the target metadata into the target memory block corresponding to the target cache description array, and update the target record determined based on the target index information based on the slot identifier of the target cache description array;

[0246] Returns the target address information of the target memory block to the target process.

[0247] In one possible implementation, the cache description array set includes a plurality of array partitions, each array partition including: a cache description array of a corresponding type of memory block, wherein the upper limit of the cache data length of different types of memory blocks is different;

[0248] The data cache unit 1404 is specifically configured to:

[0249] Based on the data length of the target metadata, determine the corresponding target array partition;

[0250] A target cache description array that meets a preset replacement condition is determined from the target array partition.

[0251] In a possible implementation, the cache description array includes a reference count parameter and an access count parameter; the apparatus includes a timing scanning unit 1405 configured to:

[0252] For each cache description array, when accessing the cache description array, updating the reference count parameter and access count parameter in the corresponding cache description array;

[0253] The data cache unit 1404 is specifically configured to:

[0254] A target cache description array having a reference count parameter and an access count parameter both being zero is determined from the cache description array set.

[0255] In a possible implementation, the target cache description array includes a metadata partition identifier, partition version information, and metadata validity expiration time;

[0256] The address obtaining unit 1403 is specifically configured to:

[0257] Based on the target metadata partition identifier corresponding to the target metadata, obtain the latest partition version information of the target metadata partition;

[0258] When the latest partition version information is consistent with the storage partition version information in the target cache description array, and the target metadata is determined to be not currently expired based on the metadata validity expiration time, it is determined that the target metadata is currently in a valid state.

[0259] In a possible implementation, the timing scanning unit 1405 is configured to:

[0260] At every preset interval, the cache description array collection is traversed and the following operations are performed respectively:

[0261] If the slot identifier of the currently traversed cache description array is not in the free slot list, obtain the reference count parameter and the access count parameter of the cache description array;

[0262] If the obtained reference count parameter and access count parameter are both zero, the slot identifier of the cache description array is added to the free slot list;

[0263] If the obtained reference count parameter is zero and the access count parameter is not zero, the access count parameter is decremented by one.

[0264] Through the above device, a metadata global caching solution is adopted. For all working processes, they can access the metadata they need according to this method, so there is no need to maintain their own metadata cache, which reduces the duplication of metadata cache between working processes and reduces the occupied memory resources. At the same time, it also reduces the transmission resource occupation of duplicate metadata and improves resource utilization.

[0265] The device can be used to execute the methods shown in the various embodiments of the present application. Therefore, for the functions that can be implemented by the various functional modules of the device, please refer to the description of the aforementioned embodiments and no further details will be given.

[0266] Referring to FIG15 , based on the same technical concept, an embodiment of the present application further provides a computing device. In one embodiment, the computing device may be the computing node or terminal device shown in FIG1 . As shown in FIG15 , the computing device includes a memory 1501 , a communication module 1503 , and one or more processors 1502 .

[0267] Memory 1501 is used to store computer programs executed by processor 1502. Memory 1501 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and programs required to run the functions of the embodiments of the present application; the data storage area may store various instant messaging information and operating instruction sets.

[0268] Memory 1501 may be a volatile memory; a non-volatile memory; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1501 may be a combination of the aforementioned memories.

[0269] The processor 1502 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1502 is configured to implement the above metadata access method when calling the computer program stored in the memory 1501 .

[0270] The communication module 1503 is used to communicate with computing nodes, metadata servers, terminal devices, etc.

[0271] The specific connection medium between the memory 1501, communication module 1503, and processor 1502 is not limited in the embodiments of the present application. In Figure 15, the embodiment of the present application shows that the memory 1501 and the processor 1502 are connected via a bus 1504. The bus 1504 is depicted in bold in Figure 15. The connection methods between other components are merely schematic and are not intended to be limiting. The bus 1504 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 15 only uses a single bold line, but does not represent a single bus or a single type of bus.

[0272] The memory 1501 stores a computer storage medium, which stores computer-executable instructions. The computer-executable instructions are used to implement the metadata access method of the embodiments of the present application. The processor 1502 is used to execute the metadata access method of the above-mentioned embodiments, as shown in Figure 3, Figure 9 or Figure 12.

[0273] The following describes a computing device 160 according to this embodiment of the present application with reference to Figure 16. The computing device 160 in Figure 16 is merely an example and should not limit the functionality and scope of use of the embodiment of the present application.

[0274] As shown in Figure 16 , computing device 160 is implemented as a general-purpose computing device. Components of computing device 160 may include, but are not limited to, at least one processing unit 161 , at least one storage unit 162 , and a bus 163 connecting various system components (including storage unit 162 and processing unit 161 ).

[0275] Bus 163 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0276] The storage unit 162 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1621 and / or a cache memory unit 1622 , and may further include a read-only memory (ROM) 1623 .

[0277] The storage unit 162 may also include a program / utility 1625 having a set (at least one) of program modules 1624, such program modules 1624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0278] Computing device 160 may also communicate with one or more external devices 164 (e.g., a keyboard, pointing device, etc.), one or more devices that enable a user to interact with computing device 160, and / or any device that enables computing device 160 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication may occur via input / output (I / O) interface 165. Furthermore, computing device 160 may communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 166. As shown, network adapter 166 communicates with other modules of computing device 160 via bus 163. It should be understood that, although not shown, other hardware and / or software modules may be used in conjunction with computing device 160, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0279] Based on the same inventive concept, an embodiment of the present application further provides a storage medium storing a computer program. When the computer program runs on a computer, the computer executes the steps of the metadata access method according to various exemplary embodiments of the present application described above in this specification.

[0280] In some possible implementations, various aspects of the metadata access method provided in the present application may also be implemented in the form of a computer program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to cause the computer device to execute the steps of the metadata access method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the steps of each embodiment.

[0281] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0282] The program product of the embodiment of the present application may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and can be run on a computer device. However, the program product of the present application is not limited thereto. In the present application, a readable storage medium may be any tangible medium containing or storing a program, and the computer program included therein may be used by or in conjunction with a command execution system, apparatus, or device.

[0283] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0284] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0285] Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages.

[0286] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0287] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0288] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0289] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0290] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A metadata access method, the method being executed by a computer device, the method comprising: In response to a metadata query request of a target process, a matching operation is performed on a global cache file based on target index information of target metadata carried in the metadata query request; wherein the global cache file includes at least one record, each record including: index information of cached metadata, and a slot identifier of a cache description array corresponding to the corresponding metadata, each cache description array corresponding to at least one memory block for caching metadata; When the match is successful, accessing the target cache description array corresponding to the target metadata based on the target slot identifier associated with the target index information; When it is determined based on the target cache description array that the target metadata is currently in a valid state, the target address information of the target memory block that caches the target metadata is obtained from the target cache description array; wherein the target address information is used to indicate the location information of the target memory block in the shared memory; The target address information is returned to the target process.

2. The method according to claim 1, wherein the target index information includes a metadata file identifier and a data column index value corresponding to the target metadata in the metadata file; then the response to the metadata query request of the target process, based on the target index information of the target metadata carried in the metadata query request, performing a matching operation on the global cache file, comprises: Based on the data column index value, according to the target mapping method, a corresponding hash identifier is generated, and based on the hash identifier and the metadata file identifier, a query record identifier corresponding to the target metadata is generated; Based on the query record identifier, determine the target record from the global cache file, and match the metadata file identifier, the data column index value, and the hash identifier with corresponding values ​​included in the target record respectively; When the match is successful, based on the target slot identifier associated with the target index information, accessing the target cache description array corresponding to the target metadata includes: When the metadata file identifier, the data column index value, and the hash identifier all match successfully, the target cache description array corresponding to the target metadata is accessed based on the target slot identifier associated with the target index information.

3. The method according to claim 2, wherein generating a corresponding hash identifier based on the data column index value according to a target mapping method comprises: Based on the metadata file identifier, a corresponding target file description array is obtained from a plurality of file description arrays, wherein each file description array indicates: a data type of each data column in a corresponding metadata file; Based on the target file description array, obtaining the target data type associated with the data column index value; The target mapping method corresponding to the target data type is called to perform hash value mapping processing on the data column index value to obtain the hash identifier.

4. The method according to claim 2, wherein the global cache file includes a plurality of file partitions, each file partition includes at least one record; after generating a query record identifier corresponding to the target metadata based on the hash identifier and the metadata file identifier, the method further includes: Based on the query record identifier, determining the target file partition where the target record is located; Performing a first locking operation on the target file partition to add a shared lock to the target file partition; wherein the shared lock indicates that a working process is accessing the target file partition; When the match is successful, after accessing the target cache description array corresponding to the target metadata based on the target slot identifier associated with the target index information, the method further includes: An unlock operation is performed on the target file partition to release the shared lock of the target file partition.

5. The method according to any one of claims 2 to 4, after determining the target record from the global cache file based on the query record identifier, the method further comprises: When no match is successful, a metadata acquisition request is sent to the metadata server; receiving the target metadata returned by the metadata server in response to the metadata acquisition request, and caching the target metadata in a free memory block; Updating the cache description array corresponding to the free memory block, and updating the target record based on the slot identifier of the cache description array; The target address information of the free memory block is returned to the target process.

6. The method according to claim 5, when no match is successful, sending a metadata acquisition request to a metadata server comprises: When the match fails, a second locking operation is performed on the target file partition where the target record is located, so as to add an exclusive lock to the target file partition; wherein the exclusive lock indicates that other processes except the target process are prohibited from accessing the target file partition; Based on the target index information, the target record is updated, and the state of the target record is marked as a cache progress state; Performing an unlock operation on the target file partition to release the exclusive lock of the target file partition, and sending a metadata acquisition request to a metadata server to obtain the target metadata; Then the updating of the target record based on the slot identifier of the cache description array includes: After performing the second locking operation on the target file partition, the slot identifier of the cache description array is updated to the target record.

7. The method according to claim 5, after receiving the target metadata returned by the metadata server in response to the metadata acquisition request, the method further comprises: If there is no free memory block, a target cache description array that meets a preset replacement condition is determined from the cache description array set; Based on the original index information in the target cache description array, generating an original record identifier corresponding to the target cache description array; When the original record corresponding to the original record identifier meets the clearing condition, clearing the information stored in the original record; Cache the target metadata into a target memory block corresponding to the target cache description array, and update the target record determined based on the target index information based on the slot identifier of the target cache description array; The target address information of the target memory block is returned to the target process.

8. The method of claim 7, wherein the cache description array set comprises a plurality of array partitions, each array partition comprising: The cache description array of the corresponding type of memory block. The upper limit of the cache data length of different types of memory blocks is different; Then, determining a target cache description array that meets a preset replacement condition from the cache description array set includes: Determine a corresponding target array partition based on the data length of the target metadata; A target cache description array that meets a preset replacement condition is determined from the target array partitions.

9. The method according to claim 7, wherein the cache description array includes a reference count parameter and an access count parameter; the method further comprises: For each cache description array, when accessing the cache description array, updating a reference count parameter and an access count parameter in the corresponding cache description array; Then, determining a target cache description array that meets a preset replacement condition from the cache description array set includes: From the cache description array set, the target cache description array whose reference count parameter and access count parameter are both zero is determined.

10. The method according to any one of claims 1 to 4, wherein the target cache description array includes a metadata partition identifier, partition version information, and metadata effective expiration time; The method further comprises: Based on the target metadata partition identifier corresponding to the target metadata, obtaining the latest partition version information of the target metadata partition; When the latest partition version information is consistent with the storage partition version information in the target cache description array, and it is determined based on the metadata validity expiration time that the target metadata is not currently expired, it is determined that the target metadata is currently in a valid state.

11. The method according to any one of claims 1 to 4, further comprising: At each preset interval, the cache description array collection is traversed and the following operations are performed respectively: If the slot identifier of the currently traversed cache description array is not in the free slot list, obtaining a reference count parameter and an access count parameter of the cache description array; If the obtained reference count parameter and access count parameter are both zero, adding the slot identifier of the cache description array to the free slot list; If the obtained reference count parameter is zero and the access count parameter is not zero, the access count parameter is decremented by one.

12. A metadata access device, the device comprising: A global matching unit, configured to respond to a metadata query request of a target process and perform a matching operation on a global cache file based on target index information of target metadata carried in the metadata query request; wherein the global cache file includes at least one record, each record including: index information of cached metadata, and a slot identifier of a cache description array corresponding to the corresponding metadata, each cache description array corresponding to at least one memory block for caching metadata; An array access unit, configured to access a target cache description array corresponding to the target metadata based on a target slot identifier associated with the target index information when a match is successful; An address obtaining unit is used to obtain target address information of a target memory block that caches the target metadata from the target cache description array when it is determined that the target metadata is currently in a valid state based on the target cache description array; wherein the target address information is used to indicate location information of the target memory block in the shared memory; and, return the target address information to the target process.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer storage medium having a computer program stored thereon, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program which, when run on a computer, causes the computer to execute the steps of the method according to any one of claims 1 to 11.

Images