Metadata processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a metadata processing method and device, electronic equipment and a readable storage medium, and belongs to the field of data storage. The method comprises the following steps: acquiring an operation request message sent by a client, wherein the operation request message comprises a target object and a target operation; determining a plurality of key value operation requests according to the target object and the target operation; according to a pre-configured logic function, performing aggregation processing on at least part of the key value operation requests in the plurality of key value operation requests to obtain a target calling request; and according to the target calling request, processing corresponding to each key value operation request in the target calling request is executed on the target metadata. In this way, calling of at least part of the key value operation requests to the KV storage system is converted into one-time calling of the KV storage system through the target calling request, the number of calling times of the KV storage system can be reduced, and metadata operation efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data storage, and in particular, to a method, apparatus, electronic device, and readable storage medium for metadata processing. Background Art

[0002] KV storage (also known as key-value storage) is a non-relational database that uses keys to uniquely identify and access data. The basic operation of KV storage is to map a key to a value, and the corresponding value can be retrieved, updated, or deleted through the key.

[0003] Due to the high availability and scalability of KV storage, some distributed file systems (such as HDFS of Hadoop) or distributed database systems (such as Cassandra, Redis, etc.) usually adopt KV storage as the metadata storage base to meet the requirements of efficient storage and fast access to metadata. However, general KV storage is not designed for file system metadata, and metadata operations usually require multiple interactions with KV storage. For example, path lookup needs to recursively search from the root directory, and deleting an empty directory needs to recursively delete subdirectories, etc. The operations are cumbersome and the metadata operation efficiency is not high. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, electronic device, and readable storage medium for metadata processing, which are used to at least solve the problem of low metadata operation efficiency in related technologies.

[0005] In a first aspect, the embodiments of the present application provide a method for metadata processing, including: obtaining an operation request message sent by a client, where the operation request message includes a target object and a target operation; determining a plurality of key-value operation requests according to the target object and the target operation; obtaining a target call request by aggregating at least some of the key-value operation requests in the plurality of key-value operation requests according to a pre-configured logical function; and performing processing corresponding to each key-value operation request in the target call request on the target metadata according to the target call request.

[0006] In a second aspect, the embodiments of the present application provide a metadata processing apparatus, including: an obtaining module, configured to obtain an operation request message sent by a client, where the operation request message includes a target object and a target operation; a determining module, configured to determine a plurality of key-value operation requests according to the target object and the target operation; an aggregating module, configured to obtain a target call request by aggregating at least some of the key-value operation requests in the plurality of key-value operation requests according to a pre-configured logical function; and a processing module, configured to perform processing corresponding to each key-value operation request in the target call request on the target metadata according to the target call request.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The memory stores a program or instructions that can run on the processor. When the program or instructions are executed by the processor, the steps of the method described in the first aspect above are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. A program or instructions are stored on the computer-readable storage medium. When the program or instructions are executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] The technical solution provided by the embodiment of the present application at least brings the following beneficial effects:

[0010] Obtain an operation request message sent by a client. The operation request message includes a target object and a target operation; determine a plurality of key-value operation requests according to the target object and the target operation; obtain a target call request by aggregating at least some of the key-value operation requests among the plurality of key-value operation requests according to a pre-configured logical function; perform processing corresponding to each key-value operation request in the target call request on the target metadata according to the target call request. In this embodiment, by aggregating at least some of the key-value operation requests among the plurality of key-value operation requests through a logical function, the call of at least some of the key-value operation requests to the KV storage can be converted into a one-time call to the KV storage through the target call request. Performing a one-time batch processing on the target metadata according to the target call request can reduce the number of calls to the KV storage, reduce the latency of metadata operations, and improve the efficiency of metadata operations.

[0011] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0013] Figure 1 Shows a schematic diagram of a possible application scenario provided by an embodiment of the present application;

[0014] Figure 2 Shows a schematic flowchart of the metadata processing method provided by an embodiment of the present application;

[0015] Figure 3 Shows an example diagram of partial reading and updating provided by an embodiment of the present application;

[0016] Figure 4Shows the schematic flow diagram of file creation provided by the embodiments of the present application;

[0017] Figure 5 Shows the schematic flow diagram of directory deletion provided by the embodiments of the present application;

[0018] Figure 6 Shows the schematic flow diagram of file renaming operation provided by the embodiments of the present application;

[0019] Figure 7 Shows the schematic flow diagram of modifying directory group ownership provided by the embodiments of the present application;

[0020] Figure 8 Shows the schematic structural diagram of the metadata processing device provided by the embodiments of the present application;

[0021] Figure 9 Shows the schematic structural diagram of the electronic device provided by the embodiments of the present application. Detailed Description of the Invention

[0022] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0023] KV storage (Key-Value storage) is a non-relational database that stores data in the form of key-value pairs, where each key uniquely corresponds to a value. KV storage typically provides high throughput and low-latency data access, which makes it widely used in scenarios where fast storage and retrieval of large amounts of data are required. In a distributed file system, KV storage can be used to store metadata or small files to provide fast access and retrieval. In scenarios such as big data processing and machine learning, KV storage is often used to store temporary data, intermediate results, or parameter configurations, etc., to support distributed computing and data processing tasks.

[0024] Figure 1FIG. 0 shows a schematic diagram of a possible application scenario provided by an embodiment of the present application. As shown in the figure, the distributed block storage system includes a metadata service 110 (MetaData Service, MDS), a Key-Value storage system 120 (KVStore), a block storage service 130 (Chunk Store), and a client 140 (Client); among them, the metadata service 110 is used for configuration and metadata management; the Key-Value storage system 120 is used for metadata storage; the block storage service 130 is used for block data storage. In block storage, a file or object is divided into fixed-size chunks and stored in the storage system respectively. Each chunk has a unique identifier, and the complete file or object can be retrieved and assembled according to the identifier; the client 140 is used to provide an access interface for block storage, which can be in user mode or kernel mode. General KV storage is not designed for file system metadata. Operations such as recursive search of the file directory tree usually require multiple KV operations to complete. For example, for path resolution, it is necessary to search step by step according to the parent directory ID and file name; for deeper directories, multiple KV operations are required to complete one path resolution; there is also a creation operation, which requires querying KV once to see if the record exists, and if not, performing a KV put operation to insert a record. Its operations are rather cumbersome and the metadata operation efficiency is not high.

[0025] In view of the problems existing in the above metadata processing process, an embodiment of the present application provides a metadata processing method. This method aggregates at least some of the key-value operation requests among multiple key-value operation requests through a logical function, so as to convert the calls to the KV storage by at least some of the key-value operation requests into a one-time call to the KV storage through a target call request, and perform one-time batch processing on the target metadata according to the target call request, thereby reducing the number of calls to the KV storage and improving the metadata operation efficiency.

[0026] Please refer to Figure 2 , Figure 2 FIG. 10 shows a schematic flowchart of the metadata processing method provided by an embodiment of the present application. This method can be applied to the application scenario as shown in Figure 1 FIG. 12. This application scenario may include a metadata service 110 and a client 140. Among them, the client 140 can communicate with the metadata service 110 through a network, and the client can generate an operation request message and send the operation request message to the metadata service 110. After the metadata service 110 processes the received operation request message, it triggers the Key-Value storage system 120 to perform relevant operations.

[0027] Among them, the client 140 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. Or, it can also be one or more applications in the above devices. The metadata service 110 can be applied to a server. The server can be implemented by an independent server or a server cluster composed of multiple servers. It can have a corresponding Key-Value storage system 120. The Key-Value storage system 120 can store data that the server needs to process, such as an operation request message to be processed from the client 140. In actual applications, the Key-Value storage system can be integrated on the server, or placed on the cloud or other network servers. As Figure 2 shown, taking the method 200 applied to Figure 1 the metadata service 110 in it as an example, it can include the following steps:

[0028] Step S201: Obtain an operation request message sent by the client.

[0029] In step S201, the client 140 sends an operation request message to the metadata service 110. The operation request message is used to request the storage location information of the metadata of the target object and operate on the target metadata according to the storage location information. Among them, the operation request message includes the target object and the target operation. For example, the create file message create / dir1 / dir2 / A includes the target object dir1 / dir2 / A and the target operation create; the delete directory message delete / dir1 / dir2 includes the target object / dir1 / dir2 and the target operation delete; the move or rename file mv / src / A / dat includes the target object / src / A / dat and the target operation mv; the modify directory group message chown / dir1 / dir2 gid uid includes the target object dir1 / dir2 and the target operation chown, etc.

[0030] Step S202: Determine multiple key-value operation requests according to the target object and the target operation.

[0031] In step S202, the metadata service 110 determines multiple key-value operation requests according to the target object and the target operation in the operation request message. In an exemplary implementation, the operation request message: create / dir1 / dir2 / A, that is, to create file A under directory / dir1 / dir2. Then the metadata service 110 obtains the parent directory of A and gets the key-value operation requests: get / ; get dir1; get dir2; create a new key-value pair under the / dir1 / dir2 directory, and gets the key-value operation requests: Get(key( / dir1 / dir2.A), A.value); Put(key( / dir1 / dir2.A), A.value); update the metadata of the / dir1 / dir2 directory and mark it as a non-empty directory to reflect the existence of file A, and gets the key-value operation request: Put(dir2.key, dir2.newvaluelist).

[0032] Step S203: According to the pre-configured logic function, by aggregating at least some of the key-value operation requests in the multiple key-value operation requests, a target call request is obtained.

[0033] Continuing with the above example, assume the configured logic function is:

[0034]

[0035] According to the above pre-configured logic function, recursively obtain the file A to be created: Recursivefetch( / dir1 / dir2); if file A does not exist, insert and update file A in the directory: Insertupdate(key( / dir1 / dir2.A), A.value); furthermore, update the metadata of the / dir1 / dir2 directory: Put(dir2.key, dir2.newvaluelist).

[0036] Thus, by aggregating at least some of the key-value operation requests in the multiple key-value operation requests in step S202 above, a target call request is obtained: Recursivefetch( / dir1 / dir2), Insertupdate(key( / dir1 / dir2.A), A.value), and Put(dir2.key, dir2.newvaluelist).

[0037] Step S204: According to the target call request, perform processing corresponding to each key-value operation request in the target call request on the target metadata.

[0038] In step S204, the Key-Value storage system 120 is called according to the target call request, and corresponding processing is performed on the target metadata for each key-value operation request in the target call request.

[0039] Through the above steps, by aggregating at least some of the key-value operation requests among multiple key-value operation requests according to a logical function, the calls of at least some of the key-value operation requests to the KV storage can be converted into a one-time call to the KV storage through the target call request, and the target metadata can be batch-processed at one time according to the target call request, which can reduce the number of calls to the KV storage, reduce the latency of metadata operations, and improve the efficiency of metadata operations.

[0040] In a possible implementation manner, in the above step S202, determining multiple key-value operation requests according to the target object and the target operation includes:

[0041] Determining a first key-value operation request according to the target object, where the first key-value operation request is used to request to obtain the storage location information of the target object;

[0042] Determining a second key-value operation request according to the target operation, where the second key-value operation request is used to request to perform processing corresponding to the target operation on the target metadata of the target object according to the storage location information.

[0043] In the embodiment of the present application, when an operation request message sent by the client is obtained, a first key-value operation request can be determined according to the target object in the operation request message. For example, taking the operation request message as create / dir1 / dir2 / A, the storage location information of file A is requested according to / dir1 / dir2 / A, then the first key-value operation request can be get / ; get dir1; get dir2; determining a second key-value operation request according to the target operation in the operation request message. Continuing with the above example, according to the target operation create, requesting to perform processing corresponding to the target operation create on the target metadata of the target object according to the storage location information, then the second key-value operation request can be Get(key( / dir1 / dir2.A),A.value), Put(key( / dir1 / dir2.A),A.value), Put(dir2.key,dir2.newvaluelist).

[0044] Taking the operation request message mv / src / A / dat as an example, if the storage location information of file A is requested according to / src / A / dat, the first key-value operation request can be get / ; get src; get / ; get dst; According to the target operation mv, the request is to perform processing corresponding to the target operation mv on the target metadata of the target object according to the storage location information, then the second key-value operation request can be Get(key(dst,A),A.value), Put(key(dst,A),A.value).

[0045] In a specific application, the Key-Value storage system 120 (KV storage system) can read the value Value through the key Key, but usually does not support partial reading and updating of the value. Even if only a small part of the value needs to be read, the entire value needs to be read from the disk. The update operation also needs to read the entire value first, modify the partial value and then write it, which exacerbates the read amplification and write amplification in the scenario where the value occupies a relatively large space. In the embodiments of the present application, the second key-value operation request includes the key Key and the operation field; the above-mentioned processing of the target metadata corresponding to each key-value operation request in the target call request according to the target call request includes:

[0046] For each second key-value operation request in the target call request, obtain the value value corresponding to the key Key in the second key-value operation request;

[0047] Determine the field offset according to the position information of the operation field in the value value;

[0048] Process the target metadata corresponding to the operation field of the key-value storage data according to the field offset.

[0049] In a specific application, the metadata can include directory metadata, file metadata, and block metadata, and their corresponding keys and values are shown in the following table:

[0050] Metadata Key Value Directory Metadata Parent Directory ID + Name Attributes such as ID, Permission, Type, Pid, Mtime, Ctime, etc. File Metadata Parent Directory ID + Name Attributes such as ID, Permission, Type, Pid, Mtime, Chunk, etc. Chunk Metadata File ID + Chunk ID Attributes such as Diskinfo, chunkinfo, etc.

[0051] The metadata service 110 can obtain the value by calling the get operation through the key, and can update the value by calling the put operation. Generally speaking, the get operation and the put operation operate on the entire value and do not support partial reading and updating of the value. Even if only a small part of the value needs to be read, the entire value needs to be read from the disk, so there is a problem of read and write amplification. The embodiments of the present application realize partial reading and updating of the value according to the field offset of the operation field in the entire value, so as to reduce the read and write amplification problem of the KV storage and reduce the number of read and write operations on the disk. Such as Figure 3As shown in the figure, during reading, the MDS passes the Key and the field offset to the KV Store. After the KV Store finds the Value based on the Key, it then returns the value required by the MDS according to the field offset. During update, the MDS takes the field offset of the value to be modified and the new Value as input parameters to the KV Store. After the KV Store finds the Value based on the Key, it then updates part of the value on the disk according to the field offset.

[0052] In a possible implementation manner, when there are multiple operation fields, determining the field offset according to the position information of the operation fields in the value value includes:

[0053] Determining a field offset address set and a field length set according to the position information of the multiple operation fields in the value value;

[0054] The above-mentioned processing of the target metadata corresponding to the operation field of the key-value stored data includes:

[0055] Determining the starting position information corresponding to each operation field in the key-value stored data according to the field offset address set; determining the storage position information of each operation field according to the starting position information of each operation field and the field length corresponding to each operation field in the field length set, and processing the target metadata of the storage area indicated by the storage position information of each operation field.

[0056] In an exemplary implementation manner, the target metadata corresponding to the operation field of the key-value stored data can be processed through the following statements:

[0057] Get(table,column_value_list,column_offset_list,column_size_list)whereis key=’id’;

[0058] Put(table,column_value_list,column_offset_list,column_size_list)whereis key=’id’;

[0059] Among them, table is the table name; column_value_list is the updated field value sequence, column_offset_list is the set of field offset addresses, and column_size_list is the set of field lengths. MDS calculates the field offset address offset according to the order of the attribute values in Value and the size of the attributes, and fills it into column_offset_list to call KVStore. Assume that attr1 of the file is in the first position of Value and its size is 4, then its offset is 0. The attribute value attr2 is in the second position of Value and its size is 8, then its offset should be the step length from the starting position of value to attr2, that is, the size of attr1, so its offset is 4. KV Store finds Value according to the key, and then can access the corresponding position on the disk according to the offset and size, realizing partial reading and updating.

[0060] In this way, it is possible to support reading and modifying multiple fields at once, avoid read amplification and write amplification caused by read-modify-write, and reduce frequent put operations to modify metadata.

[0061] In a possible implementation, the logical function includes at least one of the following: recursive acquisition, batch acquisition, batch deletion, insert update, and batch update.

[0062] In the embodiments of this application, according to a pre-configured logical function, by aggregating at least some of the key-value operation requests in multiple key-value operation requests, a target call request is obtained. Among them, the logical function can include: batch get (batchget), batch delete (batchdelete), recursive fetch (recursivefetch), insert update (insertupdate), and batch update (batchupdate), as shown in the following table:

[0063]

[0064]

[0065] By combining the above aggregation interfaces, the metadata service can provide services more quickly, reduce the number of calls to the KV store, and reduce the latency of metadata operations. For example, mkdir and create can be implemented through Recursivefetch and Insertupdate.

[0066] In an exemplary implementation, as Figure 4 shown, a file can be created through the following steps:

[0067] Step 401: The client Client sends a create file message create / dir1 / dir2 / A to the metadata service MDS;

[0068] Step 402: The metadata service MDS recursively fetches the parent directory of file A through the target call request Recursivefetch( / dir1 / dir2);

[0069] In this step, the target call request Recursivefetch( / dir1 / dir2) is equivalent to key-value operation requests get / ; get dir1; get dir2. In this way, multiple key-value operation requests' calls to the KV storage are transformed into a one-time call to the KV storage through the target call request, which can reduce the number of calls to the KV storage;

[0070] Step 403: Lock the parent directory of A, Lock( / dir1 / dir2), to avoid write conflicts with other threads;

[0071] Step 404: Add metadata according to the parent directory and file name A, Insertupdate(key( / dir1 / dir2.A), A.value);

[0072] In this step, the target call request Insertupdate(key( / dir1 / dir2.A), A.value) is equivalent to key-value operation requests Get(key( / dir1 / dir2.A), A.value) and Put(key( / dir1 / dir2.A), A.value), reducing the number of calls to the KV storage;

[0073] Step 405: Update the attributes of the parent directory dir2, Put(dir2.key, dir2.newvaluelist);

[0074] Step 406: Release the parent directory lock, Unlock( / dir1 / dir2);

[0075] Step 407: The metadata service MDS sends a create file operation result Create result to the client Client.

[0076] In an exemplary implementation, as Figure 5 shown, the following steps can be used to delete a directory:

[0077] Step 501: The client Client sends a delete directory message delete / dir1 / dir2 to the metadata service MDS;

[0078] Step 502: The Metadata Service MDS sends a recursive fetch request for its parent directory, Recursivefetch( / dir1 / dir2), to the KV Store; in this step, the target call request Recursivefetch( / dir1 / dir2) is equivalent to key-value operation requests get / , get dir1 / , get dir2, reducing the number of calls to the KV store;

[0079] Step 503: The KV Store returns a recursive fetch response Ack;

[0080] Step 504: Lock the parent directory Lock( / dir1) to avoid write conflicts with other threads;

[0081] Step 505: The Metadata Service MDS sends a batch delete request for this directory, its subdirectories, and files, Batchdelete(dir2), to the KV Store; in this step, the target call request Batchdelete(dir2) is equivalent to key-value operation requests Get(dir2,child); Get(dir2.child,child); Delete(dir2.child); Delete(dir2), reducing the number of calls to the KV store;

[0082] Step 506: The KV Store returns a batch delete response Ack;

[0083] Step 507: The Metadata Service MDS updates the parent directory attributes Put(dir1,dir1.newvaluelist);

[0084] Step 508: Release the parent directory lock Unlock( / dir1);

[0085] Step 509: The metadata server sends the result of the directory deletion operation to the client.

[0086] In an exemplary implementation, as Figure 6 shown, file renaming can be performed through the following steps:

[0087] Step 601: The client Client sends an mv message mv / src / A / dst / to the Metadata Service MDS;

[0088] Step 602: The Metadata Service MDS sends a recursive fetch request for the source parent directory of file A, Recursivefetch(src), to the KV Store; in this step, the target call request Recursivefetch(src) is equivalent to key-value operation requests get / , get src, reducing the number of calls to the KV store;

[0089] Step 603: The KV Store returns the recursive source parent directory response Ack;

[0090] Step 604: The Metadata Service MDS sends a request to the KV Store to recursively obtain the destination parent directory of file A, Recursivefetch(dst); this target call request Recursivefetch(dst) is equivalent to the key-value operation request get / ; get dst;

[0091] Step 605: The KV Store returns the recursive destination parent directory response Ack;

[0092] Step 606: Lock the source parent directory Lock(src) to avoid write conflicts with other threads;

[0093] Step 607: Lock the destination parent directory Lock(dst) to avoid write conflicts with other threads;

[0094] Step 608: Lock A: Lock(A) to avoid write conflicts with other threads;

[0095] Step 609: The Metadata Service MDS sends a request to the KV Store to delete the original metadata based on the source parent directory and file name A, Delete(key(src,A));

[0096] Step 610: The KV Store returns the delete response Ack;

[0097] Step 611: The Metadata Service MDS sends a request to the KV Store to add metadata based on the destination parent directory and file name A, Insertupdate(key(dst.A,A.value)); in this step, the target call request Insertupdate(key(dst.A,A.value)) is equivalent to the key-value operation requests Get(key(dst.A),A.value); Put(key(dst.A),A.value), reducing the number of calls to the KV store;

[0098] Step 612: The KV Store returns the add metadata response Ack;

[0099] Step 613: Release the source parent directory lock Unlock(src);

[0100] Step 614: Release the destination parent directory lock Unlock(src);

[0101] Step 615: Release the A lock Unlock(A);

[0102] Step 616: The metadata server sends the mv operation result mv result to the client.

[0103] In an exemplary implementation, as Figure 7 shown, the directory group can be modified through the following steps:

[0104] Step 701: The client Client sends a message chown / dir1 / dir2 gid uid to modify the group of the directory / dir1 / dir2 to the metadata service MDS;

[0105] Step 702: The metadata service MDS sends a recursive fetch directory request Recursivefetch( / dir1 / dir2) to the KV Store; in this step, the target call request Recursivefetch( / dir1 / dir2) is equivalent to the key-value operation requests get / ; get dir1; get dir2, reducing the number of calls to the KV store;

[0106] Step 703: The KV Store returns a recursive fetch response Ack;

[0107] Step 704: Lock the directory Lock( / dir1 / dir2) to avoid write conflicts with other threads;

[0108] Step 705: The metadata server MDS sends a request Batchupdate(dir2,value(gid,uid)) to the KV Store to batch modify the group of the directory, its subdirectories, and files; in this step, the target call request Batchupdate(dir2,value(gid,uid)) is equivalent to the key-value operation requests Get(dir2,child); Get(dir2.child,child); Put(dir2.child,value(gid,uid)); Put(dir2,value(gid,uid)), reducing the number of calls to the KV store;

[0109] Step 706: The KV Store returns a batch modification response Ack;

[0110] Step 707: Release the directory lock Unlock( / dir1 / dir2);

[0111] Step 708: The metadata service MDS sends the result of the directory modification operation to the client Client.

[0112] In a possible implementation, the above step S201, obtaining the operation request message sent by the client, includes:

[0113] In response to a request initiated by a client, an operation request message is generated according to the content information corresponding to the request.

[0114] In an embodiment of the present application, a user requests through a client (Client), and by performing corresponding processing on the content corresponding to the request, an operation request message is generated.

[0115] In a possible implementation manner, the above method 200 further includes:

[0116] S205: Send a response message to the client, where the response message is used to respond to the operation request message, and the response message includes target metadata and a result after processing the target metadata.

[0117] In step S205, when the metadata service (MDS) completes the processing of the target metadata for the operation request message sent by the client (Client), a response message for responding to the operation request message is sent to the client (Client). The response message may be the target metadata or the result after processing the target metadata.

[0118] An embodiment of the present application provides a metadata processing method. An operation request message sent by a client is obtained. The operation request message includes a target object and a target operation. According to the target object and the target operation, a plurality of key-value operation requests are determined. According to a pre-configured logical function, by aggregating at least some of the key-value operation requests among the plurality of key-value operation requests, a target call request is obtained. According to the target call request, corresponding processing is performed on the target metadata for each key-value operation request in the target call request. In this embodiment, by aggregating at least some of the key-value operation requests among the plurality of key-value operation requests through a logical function, the call of at least some of the key-value operation requests to the KV storage can be converted into a one-time call to the KV storage through the target call request. By performing one-time batch processing on the target metadata according to the target call request, the number of calls to the KV storage can be reduced, the latency of metadata operations can be reduced, and the efficiency of metadata operations can be improved.

[0119] Figure 8 The structure diagram of a metadata processing device provided by an embodiment of the present application is shown. The metadata processing device can implement all or part of the content in the embodiment as Figure 2 shown in the embodiment. The metadata processing device 800 includes:

[0120] An obtaining module 810, configured to obtain an operation request message sent by a client, where the operation request message includes a target object and a target operation;

[0121] A determining module 820, configured to determine a plurality of key-value operation requests according to the target object and the target operation;

[0122] An aggregation module 830, configured to obtain a target call request by aggregating at least some of the multiple key-value operation requests according to a pre-configured logical function;

[0123] A processing module 840, configured to perform processing corresponding to each key-value operation request in the target call request on the target metadata according to the target call request.

[0124] In a possible implementation, when the determination module 820 is configured to determine multiple key-value operation requests according to the target object and the target operation, it is specifically configured to;

[0125] Determine a first key-value operation request according to the target object, where the first key-value operation request is used to request to obtain the storage location information of the target object;

[0126] Determine a second key-value operation request according to the target operation, where the second key-value operation request is used to request to perform processing corresponding to the target operation on the target metadata of the target object according to the storage location information.

[0127] Wherein, the second key-value operation request includes a key Key and an operation field; when the determination module 820 is configured to perform processing corresponding to each key-value operation request in the target call request on the target metadata according to the target call request, it is specifically configured to:

[0128] For each second key-value operation request in the target call request, obtain the value value corresponding to the key Key in the second key-value operation request;

[0129] Determine a field offset according to the position information of the operation field in the value value;

[0130] Process the target metadata corresponding to the operation field of the key-value storage data according to the field offset.

[0131] In a possible implementation, when there are multiple operation fields, the determination module 820 is further configured to:

[0132] Determine a field offset address set and a field length set according to the position information of the multiple operation fields in the value value;

[0133] Determine the starting position information corresponding to each operation field in the key-value storage data according to the field offset address set;

[0134] Determine the storage location information of each operation field according to the start position information of each operation field and the field length corresponding to each operation field in the field length set, and process the target metadata in the storage area indicated by the storage location information of each operation field.

[0135] Among them, the above logical functions include at least one of the following: recursive acquisition, batch acquisition, batch deletion, insert update, and batch update.

[0136] In a possible implementation manner, when the acquisition module 810 is used to acquire the operation request message sent by the client, it is specifically used for:

[0137] In response to the request initiated by the client, generate the operation request message according to the content information corresponding to the request.

[0138] In a possible implementation manner, the metadata processing device 800 further includes:

[0139] A sending module, configured to send a response message to the client, where the response message is used to respond to the operation request message, and the response message includes the target metadata and the result of processing the target metadata.

[0140] An embodiment of the present application provides a metadata processing device, including an acquisition module, a determination module, an aggregation module, and a processing module; the acquisition module acquires an operation request message sent by a client; the determination module determines a plurality of key-value operation requests according to the target object and the target operation; the aggregation module obtains a target call request by performing aggregation processing on at least some of the key-value operation requests in the plurality of key-value operation requests according to a pre-configured logical function; the processing module performs processing corresponding to each key-value operation request in the target call request on the target metadata. In this way, by performing aggregation processing on at least some of the key-value operation requests in the plurality of key-value operation requests through a logical function, it is possible to convert the calls of at least some of the key-value operation requests to the KV storage into a one-time call to the KV storage through the target call request, and perform one-time batch processing on the target metadata according to the target call request, which can reduce the number of calls to the KV storage, reduce the latency of metadata operations, and improve the efficiency of metadata operations.

[0141] Figure 9The figure shows a schematic diagram of the hardware structure of the electronic device provided in the embodiments of the present application. Referring to this figure, at the hardware level, the electronic device 900 includes a processor 910. Optionally, it includes an internal bus 920, a network interface 930, and a memory 940. Among them, the memory 940 may include a memory 941, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory 942 (non-volatile memory), such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0142] The processor 910, the network interface 930, and the memory can be interconnected through the internal bus 920. The internal bus 920 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a bidirectional arrow is used in this figure, but it does not mean that there is only one bus or one type of bus.

[0143] The memory 940 stores programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory 940 may include a memory 941 and a non-volatile memory 942, and provide instructions and data to the processor 910.

[0144] The processor 910 reads the corresponding computer program from the non-volatile memory 942 into the memory and then runs it, forming a device for locating the target user at the logical level. The processor 910 executes the program stored in the memory and specifically executes: Figure 2 or Figures 4 to 7 The method disclosed in the illustrated embodiment and implements the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be elaborated herein.

[0145] The above as in the present application Figure 2 or Figures 4 to 7The method disclosed in the illustrated embodiment can be applied to or implemented by the processor 910. The processor 910 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 910 or the instructions in the form of software. The above-mentioned processor 910 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step, and logic block diagram disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0146] This computer device can also execute each method described in the foregoing method embodiments and implement the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be elaborated herein.

[0147] Of course, in addition to the software implementation, the electronic device of the present application does not exclude other implementation manners, such as a logic device or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or a logic device.

[0148] The embodiments of the present application also propose a computer-readable storage medium. The computer-readable medium stores one or more programs. When the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute Figure 2 or Figures 4 to 7 the method disclosed in the illustrated embodiment and implement the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be elaborated herein.

[0149] Among them, the computer-readable storage medium includes a read-only memory (ROM for short), a random access memory (RAM for short), a magnetic disk, an optical disc, etc.

[0150] Furthermore, the embodiment of the present application also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the following process is implemented: Figure 2 or Figures 4 to 7 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be elaborated herein.

[0151] The embodiment of the present application can be applied to various scenarios of electronic device cooperation or interconnection, including: cooperation and interconnection between a mobile phone and a laptop / tablet computer; cooperation and interconnection between a mobile terminal and a smart TV / display; cooperation and interconnection between a mobile phone or a tablet computer and an in-vehicle entertainment system; cooperation and interconnection between a mobile terminal and a smart conference system, etc. Thus, it can meet the diverse scenario requirements of users in smart home, smart office, smart travel, etc.

[0152] In summary, the above are only the preferred embodiments of the present application, and do not limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

[0153] The system, device, module or unit illustrated in the foregoing embodiment can be specifically implemented by a computer chip or an entity, or by a product with a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0154] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can store information accessible by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0155] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0156] Each embodiment in this specification is described in a progressive manner, and the same or similar parts among the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

Claims

1. A metadata processing method, characterized in that, including: obtaining an operation request message sent by a client, where the operation request message includes a target object and a target operation; determining a plurality of key-value operation requests according to the target object and the target operation; obtaining a target call request by aggregating at least some of the key-value operation requests in the plurality of key-value operation requests according to a pre-configured logic function; performing processing corresponding to each key-value operation request in the target call request on target metadata according to the target call request.

2. The method according to claim 1, characterized in that The determining a plurality of key-value operation requests according to the target object and the target operation includes: determining a first key-value operation request according to the target object, where the first key-value operation request is used to request to obtain storage location information of the target object; determining a second key-value operation request according to the target operation, where the second key-value operation request is used to request to perform processing corresponding to the target operation on the target metadata of the target object according to the storage location information.

3. The method according to claim 2, wherein The second key-value operation request includes a key Key and an operation field; the performing processing corresponding to each key-value operation request in the target call request on target metadata according to the target call request includes: for each second key-value operation request in the target call request, obtaining a value value corresponding to the key Key in the second key-value operation request; determining a field offset according to position information of the operation field in the value value; processing target metadata corresponding to the operation field of the key-value storage data according to the field offset.

4. The method according to claim 3, characterized in that, When there are multiple operation fields, the determining a field offset according to position information of the operation field in the value value includes: determining a field offset address set and a field length set according to position information of the multiple operation fields in the value value; The processing target metadata corresponding to the operation field of the key-value storage data according to the field offset includes: determining start position information corresponding to each operation field in the key-value storage data according to the field offset address set; determining storage position information of each operation field according to the start position information of each operation field and the field length corresponding to each operation field in the field length set, and processing target metadata in a storage area indicated by the storage position information of each operation field.

5. The method according to claim 1, wherein The logic function includes at least one of the following: recursive acquisition, batch acquisition, batch deletion, insert update, and batch update.

6. The method according to claim 1, characterized in that The obtaining an operation request message sent by a client includes: responding to a request initiated by a client, and generating the operation request message according to content information corresponding to the request.

7. The method according to claim 1, characterized in that, It further includes: sending a response message to the client, where the response message is used to respond to the operation request message, and the response message includes target metadata and a result of processing the target metadata.

8. A metadata processing device, characterized in that, including: an obtaining module, configured to obtain an operation request message sent by a client, where the operation request message includes a target object and a target operation; A determination module, configured to determine a plurality of key-value operation requests according to the target object and the target operation; An aggregation module, configured to obtain a target call request by performing aggregation processing on at least some of the plurality of key-value operation requests according to a pre-configured logical function; A processing module, configured to perform processing corresponding to each key-value operation request in the target call request on the target metadata according to the target call request.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, and the memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that, A program or instruction is stored on the computer-readable storage medium. When the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images