Data processing method, system, server and medium

Abstract

Embodiments of the present application provide a data processing method, system, server and medium. The method comprises: receiving an acquisition request for obtaining at least one object or any target data block in target data; wherein the acquisition request contains at least one object identifier; generating a metadata request for a metadata storage unit based on the correspondence between the at least one object identifier and the metadata storage unit; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence; receiving metadata fed back by each metadata storage unit in the metadata end in response to the metadata request; and sending at least one target data or target data block obtained based on the metadata. Combining object requests and combining metadata requests can effectively reduce the frequency of network transmission acquisition requests in the object storage system and reduce network link overhead.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods, systems, servers and media. Background Technology

[0002] With the development of big data technology, various storage methods have been proposed to meet users' needs for storing various types of data, and object storage is one of them.

[0003] In big data use cases, as data volume and size continuously increase, retrieving objects from object storage generates numerous small files and a large number of random reads. In big data jobs, the ever-increasing data volume and varying file sizes can lead to a surge in query requests when accessing object storage, sometimes reaching a very high queries per second (QPS). This limits the server's request processing capacity, impacting the throughput efficiency of object storage. The large number of small files and random reads severely hinders the efficiency of object read jobs. Summary of the Invention

[0004] To address or improve the problems existing in the prior art, various embodiments of this application provide data processing methods, systems, servers, and media.

[0005] Firstly, in one embodiment of this application, a data processing method is provided. Applied to a server, the method includes:

[0006] Receive an acquisition request for acquiring target data of at least one object or any target data block within the target data; wherein the acquisition request contains at least one object identifier;

[0007] Based on the correspondence between at least one of the object identifiers and metadata storage units, a metadata request for the metadata storage unit is generated; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence.

[0008] The metadata received by each of the metadata storage units in the metadata receiving end in response to the metadata request;

[0009] Send at least one of the target data or the target data block obtained based on the metadata.

[0010] Secondly, in one embodiment of this application, a data processing method is provided, applied to a client, the method comprising:

[0011] In response to an object retrieval request, determine at least one object's target data or any target data block within the target data;

[0012] A request is generated based on the object identifier corresponding to the at least one object and / or the data block identifier of the target data block in the target data of the object;

[0013] Send the acquisition request to the server; so that after receiving the acquisition request, the server generates a metadata request for the metadata storage unit based on the correspondence between the metadata corresponding to at least one of the object identifiers and the metadata storage unit; and receives the metadata fed back by each of the metadata storage units in the metadata terminal;

[0014] Receive at least one of the target data or the target data block obtained based on the metadata.

[0015] Thirdly, in one embodiment of this application, a data processing method is provided, applied to a client, the method comprising:

[0016] Obtain a metadata request for a metadata storage unit provided by a server; wherein the server receives an acquisition request for obtaining target data of at least one object or any target data block within the target data; wherein the acquisition request contains at least one object identifier; and generates a metadata request for the metadata storage unit based on the correspondence between the metadata corresponding to at least one object identifier and the metadata storage unit; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence.

[0017] Based on the metadata request, determine the metadata stored in the metadata storage unit;

[0018] The metadata is sent to the server so that the server can send at least one of the target data or the target data block obtained based on the metadata to the client.

[0019] Fourthly, in one embodiment of this application, a data processing system is provided, the system comprising:

[0020] The server-side is used to execute the method described in the first aspect;

[0021] The client is used to execute the method described in the second aspect;

[0022] The metadata side is used to execute the method described in the third aspect.

[0023] Fifthly, in one embodiment of this application, a server is provided, including a memory and a processor; wherein,

[0024] The memory is used to store programs;

[0025] The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the data processing method described in the first aspect, or the data processing method described in the second aspect, or the data processing method described in the third aspect.

[0026] In a sixth aspect, in one embodiment of this application, a non-transitory machine-readable storage medium is provided, wherein executable code is stored on the non-transitory machine-readable storage medium, and when the executable code is executed by a processor of an electronic device, the processor performs the data processing method as described in the first aspect, or implements the data processing method as described in the second aspect, or implements the data processing method as described in the third aspect.

[0027] The technical solution provided in this application addresses a common issue in object storage systems. In such systems, clients may frequently initiate numerous requests to retrieve small files or perform random reads from objects, depending on their needs. When a client initiates an object retrieval request, requests for target data or target data blocks within at least one object can be merged to generate a single retrieval request. This reduces the frequency of object requests between the client and server, thus minimizing network overhead. Upon receiving the retrieval request, the server parses it to obtain the object identifier and the corresponding metadata. It's easy to understand that metadata may reside in the same storage unit; multiple different metadata items may be stored in the same unit. Therefore, retrieval requests for metadata stored in the same unit can be merged into a single metadata request. The metadata server then sends the retrieved metadata to the server, which retrieves the target data or target data blocks of at least one object based on the metadata and sends this data to the client. This solution effectively reduces the frequency of data retrieval requests from the server to the metadata server, alleviating network overhead. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1A flowchart illustrating a data processing method provided in an embodiment of this application;

[0030] Figure 2 This application provides a flowchart illustrating a method for generating metadata requests.

[0031] Figure 3 A flowchart illustrating multi-terminal interaction provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the structure of the data processing system provided in the embodiments of this application;

[0033] Figure 5 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0034] Figure 6 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the structure of another data processing apparatus provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0038] In some processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. These operations may be executed out of order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. Moreover, the embodiments described below are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0039] In object storage applications, as data volumes continue to increase, some big data applications generate a large number of small file read requests due to direct operations and usage of data in object storage. Simultaneously, the use of indexes for large-scale data queries in online analytical processing (OLAP) also generates a large number of random read requests. The queries per second (QPS) for small file access and random small file read requests can reach very high levels. This significantly impacts the execution time and performance of big data jobs, as well as the pressure on the object storage itself. Therefore, a solution that can effectively improve the efficiency of user profile creation is needed.

[0040] Terminology Explanation:

[0041] An object is the basic unit of Cloud Object Storage (COS). It can be understood as data of any format, such as images, documents, and audio / video files. Each object consists of an object key (also called an object identifier), an object value (the object's target data), and the object's metadata.

[0042] Bucket: A bucket is a container for objects; each bucket can hold any number of objects. Objects in COS can be specified as different storage types.

[0043] Object Key: The object key is a unique identifier for an object within the bucket; it can be called an object identifier, or simply understood as a file path. For example, the object naming format is... <objectkey>.

[0044] Object value: This refers to the target data of the object to be retrieved, which can be simply understood as the file content.

[0045] Object metadata is a set of key-value pairs, which can be simply understood as file attributes. Object storage describes the target data and provides indexes. Examples include file modification time and storage type.

[0046] Small files: In the field of big data, small files typically refer to files whose size is significantly smaller than an HDFS block (64MB or 128MB).

[0047] Random read: refers to the input and output generated by random read (Range Get) of object storage.

[0048] Queries Per Second (QPS) is a measure of how much traffic a specific query server processes within a given time period.

[0049] The technical solution implemented in this application will be explained and described below with reference to specific embodiments.

[0050] like Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application. The method can be applied to a server. Specifically, the method includes the following steps:

[0051] 101: Receive an acquisition request for acquiring target data of at least one object or any target data block within the target data; wherein the acquisition request contains at least one object identifier.

[0052] 102: Based on the correspondence between at least one of the object identifiers and metadata storage units, generate a metadata request for the metadata storage unit; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence.

[0053] 103: Receive the metadata from each of the metadata storage units in the metadata receiving end in response to the metadata request.

[0054] 104: Send at least one of the target data or the target data block obtained based on the metadata.

[0055] In practical applications, a client's active or passive operation request may be a request to retrieve target data from multiple objects simultaneously, or one or more target data blocks from the target data. For example, a client might request information about a car show event, which includes various types of data such as images, text, and multiple videos. When storing this information, object storage is often used. As mentioned earlier, object storage includes object identifiers, target data, and metadata. In this embodiment, the object to be retrieved can be understood as the target data of the object. When storing the target data, it may be divided into multiple target data blocks (ranges) and stored in multiple different metadata storage units.

[0056] When a client initiates an object request, it can be a request to retrieve the target data of one or more whole objects, or a request to retrieve one or more partial target data blocks within an object. In existing technologies, the client sends separate object requests for multiple images, multiple videos, or partial data blocks based on the operation request. When the number of object requests reaches a large order of magnitude, it negatively impacts normal network operation. Therefore, in this application's solution, the client merges multiple object requests into a single retrieval request. This retrieval request can be, for example, an HTTP request, and it includes the object identifier of at least one object, such as img1, img2, etc. By merging multiple object requests into one retrieval request, the client only needs to send one HTTP request to complete multiple object request tasks, effectively reducing network link occupation between the client and server and lowering HTTP generation costs. As an optional approach, the client and server can establish a connection based on a batch interface when communicating.

[0057] As mentioned earlier, the server receives a retrieval request containing object requests for multiple objects. Therefore, after receiving the request, the server further parses it to obtain the identifiers of the multiple objects contained within the request. Based on these object identifiers, the server can determine which objects the client needs to retrieve, or the metadata corresponding to the target data blocks within those target data.

[0058] It's important to note that in object storage, each object has its own metadata. This metadata accurately describes the object's attributes, supporting functions such as indicating storage location, historical data, resource lookup, and file records. Object metadata is a set of key-value pairs within the object; it's a string sent by the server before transmitting HTML data to the browser via the HTTP protocol, also known as the HTTP Header. Furthermore, metadata can be used to locate target data within a storage unit or target data blocks within target data.

[0059] It should be noted that the metadata request for a metadata storage unit mentioned here can be understood as merging multiple retrieval requests for multiple metadata items stored in the same metadata storage unit into a single metadata request. This allows multiple required metadata items to be retrieved from the metadata storage unit using a single metadata request, rather than sending a separate retrieval request for each metadata item.

[0060] Specifically, metadata can be stored in multiple metadata storage units, and some metadata storage units may store multiple metadata items simultaneously. Previously, metadata requests were sent separately for each different metadata item. In other words, retrieving multiple metadata items required sending a separate metadata request for each item. When the amount of metadata to be retrieved is large, this means sending numerous metadata requests, placing significant pressure on the metadata indexing system. Therefore, in this embodiment, metadata requests are sent to the metadata storage unit itself, rather than generating and sending requests for the metadata items directly. Especially when multiple metadata items are stored in the same metadata storage unit, merging separate retrieval requests for each item into a single request for the same storage unit effectively reduces the number of requests sent and alleviates the pressure on the metadata indexing server.

[0061] After receiving a metadata request, the metadata server (i.e., the metadata index server) will locate the required metadata in the metadata storage unit. Then, based on the received metadata, the server will locate the target data or target data block stored in the storage unit, and then send the target data or target data block of at least one object found to the client.

[0062] In one or more embodiments of this application, such as Figure 2 This is a flowchart illustrating a method for generating metadata requests, provided in an embodiment of this application. Figure 2 As can be seen, the step of generating a metadata request for the metadata storage unit based on the correspondence between at least one object identifier and the metadata storage unit specifically includes the following steps:

[0063] 201: Obtain at least one object identifier contained in the acquisition request.

[0064] 202: Determine the metadata corresponding to the object identifier, and the metadata storage unit storing at least one metadata.

[0065] 203: Based on the correspondence between the metadata and the metadata storage unit, generate a metadata request for a storage unit that stores at least one metadata.

[0066] In practical applications, if the object request received by the server is a retrieval request, it needs to be parsed to obtain at least one object identifier. If it is not a retrieval request, the unique object identifier contained in the object request can be directly extracted. To facilitate the subsequent generation of metadata requests for the metadata storage unit, multiple received object requests can be cached. For example, if the number of cached requests reaches a certain threshold, or the cache time reaches a certain time threshold, a metadata request for the metadata storage unit will be generated based on the object identifiers carried in each of the multiple object requests.

[0067] After obtaining the object identifier, the corresponding metadata and its respective metadata storage unit are determined based on the mapping in the object storage. In practical applications, when there is a lot of metadata, it will be stored in different storage units. In other words, multiple metadata items may be stored in the same metadata storage unit simultaneously. Therefore, when retrieving metadata from a metadata storage unit, a metadata request containing multiple metadata retrieval requests can be sent to that unit. This is equivalent to merging multiple metadata retrieval requests into a single metadata request for retrieving multiple metadata items.

[0068] For example, metadata request 1: metadata storage unit corresponding to metadata 1 is A1; metadata request 2: metadata storage unit corresponding to metadata 2 is A2; metadata request 3: metadata storage unit corresponding to metadata 3 is A1; metadata request 4: metadata storage unit corresponding to metadata 4 is A1. It can be seen that metadata 1, metadata 3, and metadata 4 are stored in metadata storage unit A1. Therefore, metadata request 1, metadata request 3, and metadata request 4 can be merged into a single metadata request. When retrieving the above four metadata items, it is not necessary to send four metadata requests; only two requests are needed to obtain the required metadata.

[0069] Further, as described in step 203, generating a metadata request for a storage unit storing at least one metadata based on the correspondence between the metadata and the metadata storage unit includes:

[0070] 2031: Determine a first number of metadata corresponding to at least one object identifier included in the acquisition request, and a second number of storage units for storing the first number of metadata; wherein the first number and the second number are both positive integers.

[0071] 2032: If the first quantity of the metadata is greater than the second quantity of the storage units, then it is determined that the metadata and the metadata storage units have a many-to-one correspondence.

[0072] 2033: Merge multiple metadata retrieval requests for the same storage unit into a single metadata request.

[0073] In practical applications, after parsing the object identifier and determining the corresponding metadata and its storage unit, it's necessary to comprehensively assess whether to merge the metadata requests. Specifically, based on the object identifier, the corresponding metadata is determined, and a first count of the metadata is calculated. After determining the metadata, the storage unit where each metadata resides is also determined, and a second count of the metadata storage units is calculated. For example, if multiple requests are for the same metadata (e.g., requests for multiple data blocks of the same object), these requests can be merged into a single metadata request. After obtaining the first count of metadata and the second count of metadata storage units, the relationship between the first and second counts is further compared.

[0074] If the first quantity of metadata is greater than the second quantity of metadata storage units, it indicates that multiple metadata items are stored in the same metadata storage unit. To reduce the frequency of metadata request sending, multiple requests for retrieving multiple metadata items are merged, and this merging is performed on a per-metadata storage-unit basis. Specifically, multiple requests for retrieving multiple metadata items stored in the same metadata storage unit are merged into a single metadata request. Therefore, the server and the metadata server can retrieve multiple metadata items in that metadata storage unit using a single metadata request.

[0075] If the first quantity of metadata is not greater than the second quantity of metadata storage units, meaning there is a one-to-one correspondence between metadata and metadata storage units, then multiple metadata items are stored in the same metadata storage unit. Generally, if the first quantity is not greater than the second quantity, then the first quantity is usually equal to the second quantity, and the first quantity will not be less than the second quantity.

[0076] The above approach effectively reduces the pressure on the indexing system itself while accelerating the performance of metadata retrieval when generating metadata requests for metadata storage units.

[0077] Furthermore, if a non-fetch request is sent, that is, a regular object request containing an object identifier, the server can cache the received object requests. For example, after a certain period of time, it can merge the object identifiers contained in multiple object requests to obtain a metadata request.

[0078] In one or more embodiments of this application, receiving a request to acquire target data of at least one object or any target data block in the target data; wherein the acquisition request includes at least one object identifier, includes: receiving an acquisition request carrying an object identifier of at least one object and used to acquire the target data of the object; or, acquiring an acquisition request carrying an object identifier of at least one object and / or a data block identifier corresponding to the target data block of the object and used to acquire the target data and the target database.

[0079] In practical applications, the target data of an object may be divided into multiple target data blocks and stored in the same or different buckets. Therefore, when a client retrieves an object, it can retrieve the entire target data of the object or a portion of the target data. When generating batch requests, different merging methods are used for different object request types. Specifically, when retrieving at least one entire object's target data, the retrieval request includes the object identifier. When retrieving at least one target data block from at least one target data point, or at least one target data block from a single target data point, the retrieval request includes both the object identifier and the data block identifier. Of course, there is also a case where the client responds to both object requests for the entire target data of an object and object requests for a specific target data block; in this case, the generated retrieval request includes both the object identifier and the data block identifier.

[0080] For example, a generated retrieval request containing an object identifier might be HTTP...img1 img2 img3..., where img1 is assumed to be the object identifier. A generated retrieval request containing a data block identifier might be HTTP...img1<Range1:Range2> img2 img3……. The above method allows for the merging of multiple object requests into a single retrieval request, effectively reducing the frequency of object requests between the client and server, and alleviating network pressure.

[0081] In one or more embodiments of this application, sending at least one target data or target data block obtained based on the metadata includes: determining, based on the object identifier, the object request corresponding to at least one object in the acquisition request; and sending the target data or target data block of the target data of at least one object obtained based on the metadata to the corresponding client according to the correspondence between the object identifier and the object request.

[0082] In practical applications, object requests are typically sent to the server one by one. This solution merges object requests to reduce the frequency of requests. However, the client's needs for retrieving objects are independent. In other words, after the server locates the target data or target data block of an object, it still needs to provide accurate feedback based on the client's actual object request. Specifically, when generating a retrieval request, the client determines the object identifiers included in the request. Each object identifier represents an object request (this object request can be either not generated on the client or stored on the client after generation). When the client receives the target data returned by the server, it determines the correspondence between the object identifiers and the object requests, and returns the target data of the objects one by one according to the correspondence.

[0083] It's important to note that when a client generates a retrieval request, if multiple object requests are merged and the objects corresponding to the merged requests are all different, the mapping between the object requests and the returned objects can be determined based on the object identifiers. If some of the merged requests correspond to the same objects, then when returning the target object data, it's necessary to distinguish between the multiple merged object requests. For example, if three object requests are merged, the returned target object data must also contain three identical objects. Although the objects the client wants to obtain are the same, the uses of the obtained objects may not be entirely the same. This approach effectively reduces network pressure and metadata indexing server pressure while meeting the demand for reading a large number of objects, and effectively avoids the problem of repeatedly retrieving the same object.

[0084] In one or more embodiments of this application, sending at least one target data or target data block obtained based on the metadata includes:

[0085] Based on the aforementioned metadata, the target data of at least one object obtained through the main path, or the target data block within the target data, will be sent to the client.

[0086] Based on the metadata, the target data of at least one object obtained through the backup path, or the target data block in the target data, will be stored in the cache unit.

[0087] In practical applications, some objects may be distributed into multiple data blocks stored in different buckets. Therefore, when retrieving data blocks, there's a problem of varying return times for each block. To ensure the client can fully retrieve the target data or target data blocks of the object it needs, a backup route is used in addition to returning the target data or target data blocks of the object to the client via the main route. However, the backup route caches the returned target data or target data blocks of the object. Only after all the target data or target data blocks of the object required by the client have been retrieved are they returned to the client in the correct order. This ensures the client retrieves all the target data or target data blocks of the object, preventing data corruption or loss.

[0088] In one or more embodiments of this application, sending the target data or target data block of at least one object obtained through the main path to the client based on the metadata includes:

[0089] If the size of the target data of the at least one object or the target data block in the target data is greater than the data threshold, then multiple data units carrying object identifiers are generated.

[0090] The multiple data units are sequentially sent to the client based on the object identifier.

[0091] In practical applications, when returning data directly to the client via the main route, if the target data of the object or the target data block within the target data is too large, for example, exceeding a data threshold (e.g., 512MB), the target data of the object or the target data block within the target data will be split into multiple data units and returned to the client. During transmission, the data units will be sent to the client in the order they were split.

[0092] For example, the server retrieves metadata, accesses the object source data (i.e., the object itself) in the bucket through the metadata, and begins returning it to the user client in frame format. The object's source data may generate multiple frame-formatted return objects depending on its size. The object's refId is used to index the object request corresponding to each frame. While returning objects to the user, the server simultaneously caches the requested objects in memory and continuously returns them to the client.

[0093] To facilitate understanding, this application will now provide specific examples through concrete embodiments. Figure 3 This is a flowchart illustrating the multi-terminal interaction process provided in an embodiment of this application.

[0094] Specifically, the steps include the following:

[0095] The client executes S301: In response to the object retrieval request, it determines the target data of at least one object or any target data block within the target data; based on the object identifier corresponding to the at least one object and / or the data block identifier of the target data block within the target data of the object, it generates a retrieval request. S302: The client sends the retrieval request to the server.

[0096] The server receives a request to obtain target data or a target data block within at least one object; wherein the request includes at least one object identifier. S303: Based on the correspondence between metadata corresponding to at least one object identifier and a metadata storage unit, a metadata request for the metadata storage unit is generated. S304: The metadata request is sent to the metadata endpoint.

[0097] The metadata client obtains a metadata request for the metadata storage unit from the server. S305: Determines the metadata stored in the metadata storage unit based on the metadata request. S306: Sends the metadata to the server. After receiving the metadata, the server sends a lookup request to the storage device. S308: The storage device locates the target data of the corresponding object or the target data block within the target data based on the metadata. S309: The storage device sends the located target data of the object or the target data block within the target data to the server. S310: The server sends the target data of the object or the target data block within the target data to the client based on the merged object request.

[0098] In one or more embodiments of this application, the step of determining the target data of at least one object or any target data block in the target data in response to an object acquisition request includes: after acquiring the object acquisition request, determining whether there are multiple object acquisition requests for the target data of the same object or the target data block in the target data; if so, merging the multiple object acquisition requests for the target data of the same object or the target data block in the target data.

[0099] In practical applications, if a client initiates multiple object requests targeting the same object or a target data block within the target data of an object, these multiple object requests can be merged into a single retrieval request. This single retrieval request is then sent to the server. This method reduces the number of requests sent between the client and server, alleviating network load.

[0100] Based on the same idea, this application also provides a data processing method applied to the metadata end, the method including:

[0101] Obtain a metadata request for a metadata storage unit provided by a server; wherein the server receives an acquisition request for obtaining target data of at least one object or any target data block within the target data; wherein the acquisition request contains at least one object identifier; and generates a metadata request for the metadata storage unit based on the correspondence between the metadata corresponding to at least one object identifier and the metadata storage unit; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence.

[0102] Based on the metadata request, determine the metadata stored in the metadata storage unit;

[0103] The metadata is sent to the server so that the server can send at least one of the target data or the target data block obtained based on the metadata to the client.

[0104] This application provides a data processing system. For example... Figure 4 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. The system specifically includes: a server 41 (e.g., OSS), a metadata server 42 (e.g., MetadataServer, MDS), a storage device 43 (e.g., OSD), and a client 44.

[0105] In the object storage system, at least one client 44 establishes a communication connection with the server 41. When the client 44 has a retrieval need, it sends an object request to the server 41. In this application, instead of sending object requests one by one, multiple object requests are merged into a single retrieval request before being sent. The server provides a batch interface, which allows receiving HTTP retrieval requests from clients that simultaneously carry multiple object identifiers.

[0106] It should be noted that when client 44 merges multiple object requests, if multiple object requests contain the same object identifier, they will be merged into a single request. However, upon obtaining the object, it is still necessary to provide the required identical object to each of the multiple object requests carrying the same object identifier. Furthermore, merging object requests is not limited to requests with the same object identifier; any number of object requests in the same batch can be merged.

[0107] After receiving the retrieval request, server 41 parses it. The parsing process reveals multiple object identifiers contained within the request. Based on these object identifiers, the server determines the corresponding metadata and the metadata storage unit where each metadata item resides, in order to generate a retrieval request for the metadata. During generation, multiple retrieval requests for metadata can also be merged. It should be noted that since a single storage unit may contain multiple metadata items, merging retrieval requests for metadata is done on a per-metadata storage unit basis. In other words, retrieval requests corresponding to multiple metadata items within the same storage unit are merged. This reduces the frequency of requests sent from the server to the metadata server, lowers network pressure, reduces the pressure on the metadata indexing system, and accelerates metadata retrieval performance.

[0108] The metadata endpoint 42 retrieves the corresponding metadata from various storage units based on the received metadata request. When a single storage unit stores multiple metadata requests, multiple metadata lookups can be completed based on a single request, effectively improving metadata retrieval efficiency.

[0109] Using the system described above, requests to retrieve metadata stored in the same storage unit are merged into a single metadata request. The metadata client sends the retrieved metadata to the server, which then retrieves the target data or target data blocks of at least one object based on the metadata and sends this data to the client. This approach effectively reduces the frequency of data retrieval requests sent from the server to the metadata client, thus alleviating network overhead.

[0110] Based on the same idea, this embodiment also provides a data processing apparatus applied to a server. For example... Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. Figure 5 As can be seen, the device includes:

[0111] The acquisition module 51 is configured to receive an acquisition request for acquiring target data of at least one object or any target data block in the target data; wherein the acquisition request contains at least one object identifier.

[0112] The parsing module 52 is used to generate a metadata request for the metadata storage unit based on the correspondence between the metadata corresponding to at least one of the object identifiers and the metadata storage unit; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence.

[0113] The receiving module 53 is used to receive the metadata returned by each of the metadata storage units in the metadata terminal in response to the metadata request.

[0114] The sending module 54 is used to send at least one of the target data or the target data block obtained based on the metadata.

[0115] Optionally, the parsing module 52 is used to obtain at least one object identifier contained in the acquisition request;

[0116] Determine the metadata corresponding to the object identifier, and the storage unit storing at least one metadata;

[0117] Based on the correspondence between the metadata and the storage unit, a metadata request is generated for the storage unit that stores at least one metadata.

[0118] Optionally, the parsing module 52 is used to determine a first number of metadata corresponding to at least one object identifier included in the acquisition request, and a second number of storage units for storing the first number of metadata; wherein the first number and the second number are both positive integers;

[0119] If the first quantity of the metadata is greater than the second quantity of the storage units, then it is determined that the metadata and the metadata storage units have a many-to-one correspondence.

[0120] Multiple metadata retrieval requests targeting multiple metadata in the same metadata storage unit are merged into a single metadata request.

[0121] Optionally, the acquisition module 51 is configured to receive an object identifier carrying at least one object and to acquire the target data of the object; or, to acquire an object identifier carrying at least one object and / or a data block identifier corresponding to the target data block of the object and to acquire the target data and the target database.

[0122] Optionally, the sending module 54 is configured to determine, based on the object identifier, the object request corresponding to at least one object in the acquisition request;

[0123] Based on the correspondence between the object identifier and the object request, the target data of at least one object obtained based on the metadata, or the target data block in the target data, is sent to the corresponding client.

[0124] Optionally, the sending module 54 is used to send the target data of at least one object obtained through the main path or the target data block in the target data to the client based on the metadata;

[0125] Based on the metadata, the target data of at least one object obtained through the backup path, or the target data block in the target data, will be stored in the cache unit.

[0126] Optionally, the sending module 54 is configured to generate multiple data units carrying object identifiers if the size of the target data or the target data block in the target data of the at least one object is greater than a data threshold.

[0127] The multiple data units are sequentially sent to the client based on the object identifier.

[0128] Based on the same idea, this embodiment also provides another data processing device for use on a client side. For example... Figure 6 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application. Figure 6 As can be seen, the device includes:

[0129] The determination module 61 is used to determine, in response to an object acquisition request, at least one object's target data or any target data block within the target data.

[0130] The generation module 62 is used to generate an acquisition request based on the object identifier corresponding to the at least one object and / or the data block identifier of the target data block in the target data of the object.

[0131] The sending module 63 is used to send the acquisition request to the server; so that after receiving the acquisition request, the server generates a metadata request for the metadata storage unit based on the correspondence between the metadata corresponding to at least one object identifier and the metadata storage unit; and receives the metadata fed back by each of the metadata storage units in the metadata terminal.

[0132] The receiving module 64 is used to receive at least one of the target data or the target data block obtained based on the metadata.

[0133] The determination module 61 is used to determine, after obtaining the object acquisition request, whether there are multiple object acquisition requests for the same object's target data or the target data block in the target data;

[0134] If they exist, multiple requests for retrieving target data or target data blocks within the same object will be merged.

[0135] Based on the same idea, this embodiment also provides another data processing device for use on the metadata side. For example... Figure 7 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application. Figure 7 As can be seen, the device includes:

[0136] The acquisition module 71 is used to acquire a metadata request for a metadata storage unit provided by the server; wherein the server receives an acquisition request for acquiring target data of at least one object or any target data block in the target data; wherein the acquisition request includes at least one object identifier; and generates a metadata request for the metadata storage unit based on the correspondence between the metadata corresponding to at least one object identifier and the metadata storage unit; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence.

[0137] The determination module 72 is used to determine the metadata stored in the metadata storage unit based on the metadata request.

[0138] The sending module 73 is used to send the metadata to the server so that the server can send at least one of the target data or the target data block obtained based on the metadata to the client.

[0139] One embodiment of this application also provides a server. For example... Figure 8 This application provides a schematic diagram of the structure of a server according to an embodiment. The server includes a memory 801, a processor 802, and a communication component 803; wherein,

[0140] The memory 801 is used to store programs;

[0141] The processor 802, coupled to the memory, is used to execute the program stored in the memory. When the server is the server-side described above, the processor 802 is used to:

[0142] Receive an acquisition request for acquiring target data of at least one object or any target data block within the target data; wherein the acquisition request contains at least one object identifier;

[0143] Based on the correspondence between at least one of the object identifiers and metadata storage units, a metadata request for the metadata storage unit is generated; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence.

[0144] The metadata received by each of the metadata storage units in the metadata receiving end in response to the metadata request;

[0145] Send at least one of the target data or the target data block obtained based on the metadata.

[0146] The processor 802 is further configured to obtain at least one object identifier contained in the acquisition request;

[0147] Determine the metadata corresponding to the object identifier, and the storage unit storing at least one metadata;

[0148] Based on the correspondence between the metadata and the storage unit, a metadata request is generated for the storage unit that stores at least one metadata.

[0149] Processor 802 is further configured to determine a first number of metadata corresponding to at least one object identifier included in the acquisition request, and a second number of storage units for storing the first number of metadata; wherein the first number and the second number are both positive integers;

[0150] If the first quantity of the metadata is greater than the second quantity of the storage units, then it is determined that the metadata and the metadata storage units have a many-to-one correspondence.

[0151] Multiple metadata retrieval requests targeting multiple metadata in the same metadata storage unit are merged into a single metadata request.

[0152] Processor 802 is further configured to receive an object identifier carrying at least one object, and to obtain a request for acquiring the target data of the object; or,

[0153] Receive an object identifier carrying at least one object and / or a data block identifier corresponding to the target data block of the object, and use it to obtain the target data and the target database.

[0154] The processor 802 is further configured to determine, based on the object identifier, the object request corresponding to at least one object in the acquisition request;

[0155] Based on the correspondence between the object identifier and the object request, at least one of the target data or a target data block in the target data obtained based on the metadata is sent to the corresponding client.

[0156] The processor 802 is also configured to send target data or target data blocks of at least one object obtained through the main path to the client based on the metadata;

[0157] Based on the metadata, the target data of at least one object obtained through the backup path, or the target data block in the target data, will be stored in the cache unit.

[0158] The processor 802 is further configured to generate multiple data units carrying object identifiers if the size of the target data of the at least one object or the target data block in the target data is greater than a data threshold.

[0159] The multiple data units are sequentially sent to the client based on the object identifier.

[0160] When the server is the client mentioned above, the processor 802 is used to:

[0161] In response to an object retrieval request, determine at least one object's target data or any target data block within the target data;

[0162] A request is generated based on the object identifier corresponding to the at least one object and / or the data block identifier of the target data block in the target data of the object.

[0163] Send the acquisition request to the server; so that after receiving the acquisition request, the server generates a metadata request for the metadata storage unit based on the correspondence between the metadata corresponding to at least one object identifier and the metadata storage unit; and receives the metadata fed back by each of the metadata storage units in the metadata terminal;

[0164] Receive at least one of the target data or the target data block obtained based on the metadata.

[0165] When the server is the metadata server mentioned above, the processor 802 is used to:

[0166] Obtain a metadata request for a metadata storage unit provided by a server; wherein the server receives an acquisition request for obtaining target data of at least one object or any target data block within the target data; wherein the acquisition request contains at least one object identifier; and generates a metadata request for the metadata storage unit based on the correspondence between the metadata corresponding to at least one object identifier and the metadata storage unit; wherein the correspondence between the metadata and the metadata storage unit includes a one-to-one correspondence or a many-to-one correspondence.

[0167] Based on the metadata request, determine the metadata stored in the metadata storage unit;

[0168] The metadata is sent to the server so that the server can send at least one of the target data or the target data block obtained based on the metadata to the client.

[0169] The aforementioned memory 801 can be configured to store various other data to support operations on the server. Examples of this data include instructions for any application or method used to operate on the server. The memory can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0170] Furthermore, the processor 802 in this embodiment may specifically be a programmable switching processing chip, which is configured with a data copying engine and can copy the received data.

[0171] When the processor 802 executes the program in memory, in addition to the functions described above, it can also perform other functions, as detailed in the descriptions of the preceding embodiments. Furthermore, as... Figure 8 As shown, the server also includes other components such as the power supply component 804.

[0172] This application also provides a non-transitory machine-readable storage medium storing executable code. When the executable code is executed by a processor of an electronic device, the processor performs... Figure 1 The method described in the corresponding embodiment.

[0173] Based on the above embodiments, in an object storage system, depending on client needs, a large number of requests for small files or random reads within objects may be initiated frequently. When a client initiates an object retrieval request, object requests for target data or target data blocks within at least one object can be merged into a single retrieval request. This reduces the frequency of object requests sent between the client and server, thus reducing network overhead. After receiving the retrieval request, the server parses it to obtain the object identifier and the corresponding metadata. It's easy to understand that metadata may share the same storage unit; multiple different metadata items may be stored in the same unit. Therefore, retrieval requests for metadata stored in the same unit can be merged into a single metadata request. The metadata server sends the retrieved metadata to the server, which then retrieves the target data or target data blocks within at least one object based on the metadata and sends this data to the client. This solution effectively reduces the frequency of data retrieval requests sent from the server to the metadata server, alleviating network overhead.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / objectkey>

Claims

1. A data processing method, the method comprising: Receive a request for obtaining target data of at least one object or a target data block within the target data; wherein the request contains at least one object identifier; the request is obtained by the client by merging multiple object requests initiated by the client. If the correspondence between at least one of the object identifiers and the metadata storage unit is many-to-one, multiple metadata requests for multiple metadata stored in the same metadata storage unit will be merged into a single metadata request for the same metadata storage unit; wherein, the correspondence between the metadata and the metadata storage unit is many-to-one, indicating that multiple metadata corresponding to the at least one object identifier are stored in the same metadata storage unit. The metadata received by each of the metadata storage units in the metadata receiving end in response to the metadata request; Sending at least one target data or target data block obtained based on the metadata includes: determining, based on the object identifier, the object request corresponding to at least one object in the acquisition request; and sending at least one target data or target data block obtained based on the metadata to the client according to the correspondence between the object identifier and the object request.

2. The method according to claim 1, further comprising: Determine a first number of metadata corresponding to at least one object identifier included in the acquisition request, and a second number of storage units for storing the first number of metadata; wherein the first number and the second number are both positive integers; If the first quantity of the metadata is greater than the second quantity of the storage units, then it is determined that the metadata and the metadata storage units have a many-to-one correspondence.

3. The method according to claim 1, wherein receiving a request for acquiring target data of at least one object or any target data block within the target data; wherein, The retrieval request contains at least one object identifier, including: Receive a request carrying an object identifier of at least one object, and used to obtain the target data of the object; or, Receive an object identifier carrying at least one object and / or a data block identifier corresponding to the target data block of the object, and use it to obtain the target data and the target data block.

4. The method according to claim 1, wherein sending at least one of the target data or the target data block obtained based on the metadata comprises: Based on the metadata, at least one of the target data or the target data block obtained through the main route will be sent to the client; Based on the metadata, at least one of the target data or the target data block obtained through the backup path will be stored in the cache unit.

5. The method according to claim 4, wherein sending at least one of the target data or the target data block obtained through the main route to the client based on the metadata includes: If the size of at least one of the target data or the target data block is greater than the data threshold, then multiple data units carrying object identifiers are generated; The multiple data units are sequentially sent to the client based on the object identifier.

6. A data processing method applied to a client, the method comprising: In response to an initiated object request, determine at least one object's target data or any target data block within the target data; Based on the object identifier corresponding to the at least one object and / or the data block identifier of the target data block in the target data of the object, a retrieval request merging multiple object requests is generated; Send the acquisition request to the server; So that after receiving the acquisition request, if the correspondence between at least one of the object identifiers and the metadata storage units is many-to-one, the server will merge multiple metadata requests for multiple metadata stored in the same metadata storage unit into a single metadata request for the same metadata storage unit; wherein, the many-to-one correspondence between the metadata and the metadata storage unit indicates that multiple metadata corresponding to the at least one object identifier are stored in the same metadata storage unit; and receive metadata from each metadata storage unit in the metadata terminal; and send at least one target data or target data block obtained based on the metadata, including: determining the object request corresponding to at least one object in the acquisition request according to the object identifier; and sending at least one target data or target data block obtained based on the metadata to the client according to the correspondence between the object identifier and the object request. Receive at least one of the target data or the target data block obtained based on the metadata.

7. The method of claim 6, wherein determining the target data of at least one object or any target data block within the target data in response to an object request comprises: After obtaining the object request, determine whether there are multiple object requests for the same object's target data or the target data block within the target data; If present, multiple object requests for the same object's target data or target data blocks within the same target data will be merged.

8. A data processing method applied to a metadata endpoint, the method comprising: Obtain a metadata request for a metadata storage unit provided by the server; wherein the server receives an acquisition request for obtaining target data of at least one object or any target data block within the target data; wherein the acquisition request contains at least one object identifier; an acquisition request is obtained by the client by merging multiple object requests initiated by the client; if the correspondence between at least one object identifier and the metadata storage unit is many-to-one, multiple metadata requests for multiple metadata stored in the same metadata storage unit are merged into a single metadata request for the same metadata storage unit; wherein the many-to-one correspondence between the metadata and the metadata storage unit indicates that multiple metadata corresponding to the at least one object identifier are stored in the same metadata storage unit; Based on the metadata request, determine the metadata stored in the metadata storage unit; Sending the metadata to the server so that the server can send at least one target data or target data block obtained based on the metadata to the client includes: determining the object request corresponding to at least one object in the acquisition request based on the object identifier; and sending at least one target data or target data block obtained based on the metadata to the client based on the correspondence between the object identifier and the object request.

9. A data processing system, characterized in that, The system includes: The server-side is used to execute the method according to any one of claims 1 to 5; A client for performing the method according to any one of claims 6 to 7; The metadata side is used to execute the method described in claim 8.

10. A server, comprising a memory and a processor; wherein, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory for implementing the method of any one of claims 1 to 5, or the method of any one of claims 6 to 7, or the method of claim 8.

11. A non-transitory machine-readable storage medium storing executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 5, or the method as described in any one of claims 6 to 7, or the method as described in claim 8.

Images