Data acquisition method and device, electronic equipment and storage medium

Abstract

The invention provides a data acquisition method and device, electronic equipment and a storage medium, and the method comprises the steps: loading preset second-level object index metadata to a memory by an object storage gateway under the condition that a permission verification result of a data acquisition request is passed, obtaining the second-level object index metadata from the memory through a flow control subsystem, and storing the second-level object index metadata in the memory; the flow control subsystem sends the second-level object index metadata to the load balancing subsystem according to a preset target data transmission rate, and the load balancing subsystem obtains object index metadata corresponding to the second-level object index metadata from an index metadata system according to the second-level object index metadata; and sending the object index metadata to the client. According to the method, the object index metadata corresponding to the second-level object index metadata is obtained from the index metadata system through the second-level object index metadata, and the client can obtain all the object index metadata only by sending the data obtaining request once, so that the data obtaining efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of object storage technology, and in particular to a data acquisition method, device, electronic device and storage medium. Background Art

[0002] As a stable and scalable data storage environment, object storage has become the preferred solution for processing massive unstructured data such as images, audio and video. Its capacity and number of objects can be expanded to PB level and billions of objects, showing strong storage capabilities. However, with the rapid growth of data volume, object storage faces many challenges in processing large-scale data, especially the performance bottleneck and efficiency issues of data acquisition.

[0003] In the prior art, object storage adopts the S3 (Simple Storage Service) object storage standard, which avoids the pressure on the server from obtaining all objects at once through restrictions such as object matching prefix, starting object key tag, and the maximum number of entries returned in a single data acquisition request.

[0004] However, although this method can avoid the pressure of a single request on the server, it increases the number of call operations on the client, which leads to the problem of low data acquisition efficiency. Summary of the invention

[0005] In view of the above problems, a data acquisition method, device, electronic device and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0006] In a first aspect, an embodiment of the present application provides a data acquisition method, which is applied to an object storage system, wherein the object storage system includes an object storage gateway, an underlying storage system, and a memory, wherein the object storage gateway includes a flow control subsystem and a load balancing subsystem, and the underlying storage system includes an index metadata system, and the index metadata system includes a plurality of object index metadata, and the method includes:

[0007] When the object storage gateway receives a data acquisition request sent by a client, determining whether the client has the authority to acquire data according to the data acquisition request, and obtaining an authority verification result;

[0008] If the permission verification result is passed, loading the preset secondary object index metadata into the memory;

[0009] Acquire the secondary object index metadata from the memory through the flow control subsystem;

[0010] When the flow control subsystem obtains the secondary object index metadata, the secondary object index metadata is sent to the load balancing subsystem according to a preset target data transmission rate;

[0011] When the load balancing subsystem receives the secondary object index metadata, acquiring the object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata;

[0012] The object index metadata is sent to the client.

[0013] Optionally, the underlying storage system includes a secondary index metadata system, and the object index metadata includes an object identification number, an object name, and an object metadata set;

[0014] Before the step of determining whether the client has the authority to obtain data according to the data acquisition request and obtaining the authority verification result, the method includes:

[0015] Pre-blocking the plurality of object index metadata to obtain a plurality of secondary object index metadata, wherein the secondary object index metadata includes a secondary index metadata identification number, a starting object key tag, and the number of objects;

[0016] The plurality of secondary object index metadata are stored in the secondary index metadata system.

[0017] Optionally, the underlying storage system includes an incremental index metadata system;

[0018] After the step of loading the preset secondary object index metadata into the memory, the method includes:

[0019] Upon completion of loading the secondary object index metadata into the memory, acquiring newly added object index metadata and / or deleted object index metadata;

[0020] The newly added object index metadata and / or the deleted object index metadata are stored in the incremental index metadata system.

[0021] Optionally, before the step of sending the secondary object index metadata to the load balancing subsystem according to a preset target data transmission rate, the method includes:

[0022] Acquire the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the usage rate of the memory, the usage rate of the central processing unit and the data transmission rate through the flow control subsystem;

[0023] Respectively comparing the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the usage rate of the memory, and the usage rate of the central processing unit with respective corresponding preset thresholds;

[0024] If at least one of the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate, and the central processing unit usage rate is greater than or equal to the corresponding preset threshold, the data transmission rate is updated to obtain the target data transmission rate.

[0025] Optionally, the object storage gateway includes an aggregate computing subsystem;

[0026] The sending the object index metadata to the client comprises:

[0027] Sending the object index metadata to the aggregate computing subsystem through the load balancing subsystem;

[0028] When the aggregate computing subsystem receives the object index metadata, the object index metadata is sent to the client in descending order of the secondary index metadata identification numbers.

[0029] Optionally, after the step of sending the object index metadata to the client, the method includes:

[0030] Acquire the newly added object index metadata and / or the deleted object index metadata from the incremental index metadata system through the index metadata system;

[0031] The object index metadata in the index metadata system is updated according to the newly added object index metadata and / or the deleted object index metadata to obtain the updated index metadata system;

[0032] The newly added object index metadata and / or the deleted object index metadata stored in the incremental index metadata system are deleted.

[0033] In a second aspect, an embodiment of the present application provides a data acquisition system, the data acquisition system comprising an object storage gateway, an underlying storage system and a memory, the object storage gateway comprising a flow control subsystem and a load balancing subsystem, the underlying storage system comprising an index metadata system, wherein the object storage gateway establishes communication connections with a client and the memory respectively, the flow control subsystem establishes communication connections with the memory and the load balancing subsystem respectively, and the load balancing subsystem establishes communication connections with the client;

[0034] The object storage gateway is used to, when receiving a data acquisition request sent by a client, determine whether the client has the authority to acquire data according to the data acquisition request, obtain a permission verification result, and if the permission verification result is passed, load the preset secondary object index metadata into the memory;

[0035] The flow control subsystem is used to obtain the secondary object index metadata from the memory, and when the secondary object index metadata is obtained, send the secondary object index metadata to the load balancing subsystem according to a preset target data transmission rate;

[0036] The load balancing subsystem is used to, upon receiving the secondary object index metadata, obtain the object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata, and send the object index metadata to the client.

[0037] In a third aspect, an embodiment of the present application provides a data acquisition device, the device comprising:

[0038] A data acquisition request verification module is used to determine whether the client has the authority to acquire data according to the data acquisition request when the object storage gateway receives the data acquisition request sent by the client, and obtain a permission verification result;

[0039] A data loading module, used for loading preset secondary object index metadata into the memory if the permission verification result is passed;

[0040] A secondary object index metadata acquisition module, used to acquire the secondary object index metadata from the memory through the flow control subsystem;

[0041] A secondary object index metadata data transmission module, configured to send the secondary object index metadata to the load balancing subsystem according to a preset target data transmission rate when the flow control subsystem obtains the secondary object index metadata;

[0042] a target object index metadata acquisition module, configured to acquire, when the load balancing subsystem receives the secondary object index metadata, the object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata;

[0043] The target object index metadata feedback module is used to send the object index metadata to the client.

[0044] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement any of the data acquisition methods described above.

[0045] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the data acquisition methods described above.

[0046] In an embodiment of the present application, when the object storage gateway receives a data acquisition request sent by a client, it determines whether the client has the authority to acquire data according to the data acquisition request, obtains the authority verification result, and when the authority verification result is passed, loads the preset secondary object index metadata into the memory, acquires the secondary object index metadata from the memory through the flow control subsystem, and when the flow control subsystem acquires the secondary object index metadata, sends the secondary object index metadata to the load balancing subsystem according to the preset target data transmission rate, and when the load balancing subsystem receives the secondary object index metadata, acquires the object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata, and sends the object index metadata to the client. The present application acquires the object index metadata corresponding to the secondary object index metadata from the index metadata system through the secondary object index metadata, and the client only needs to send a data acquisition request once to acquire all the object index metadata, which reduces the number of data acquisition operations of the client. Compared with the related art that requires multiple executions of the data acquisition operation, the efficiency of data acquisition is greatly improved, and the problem of low data acquisition efficiency caused by increasing the number of client call operations is solved.

[0047] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0049] Figure 1 is a flowchart of a data acquisition method provided by an embodiment of the present application;

[0050] Figure 2 It is a flow chart of a data acquisition method provided by the present application;

[0051] Figure 3 It is a schematic diagram of the overall framework of a data acquisition system provided in an embodiment of the present application;

[0052] Figure 4 is a device block diagram of a data acquisition device provided in an embodiment of the present application;

[0053] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0055] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0056] To facilitate understanding, the terms involved in the embodiments of the present application are first briefly introduced.

[0057] 1. Object Storage

[0058] Object storage is an object-centric storage method that stores data as objects rather than files. It has the advantages of high reliability, high scalability, and high performance.

[0059] 2. Bucket (storage bucket)

[0060] Buckets are "containers" for storing objects in object storage, and there is no upper limit on the capacity of the "container". Objects are stored in buckets in a flat structure, and users can choose to store objects in a single bucket or multiple buckets.

[0061] 3. Object

[0062] An object is the basic unit of object storage and can be understood as data of any format or type, such as pictures, documents, audio and video files, etc.

[0063] 4. List

[0064] List refers to traversing the buckets storing objects and obtaining the traversed objects and object metadata.

[0065] 5. Object metadata

[0066] Object metadata refers to descriptive information fragments associated with a storage object. This information does not constitute the actual data content of the object, but is key data used to describe data attributes, manage data, or help data retrieval.

[0067] In conjunction with the accompanying drawings, a data acquisition method, device, storage medium and vehicle provided in an embodiment of the present application are described in detail through specific embodiments and their application scenarios.

[0068] Figure 1 is a flow chart of the steps of a data acquisition method provided in an embodiment of the present application, such as Figure 1 As shown, the method is applied to an object storage system, and the method includes:

[0069] Step 101: When the object storage gateway receives a data acquisition request sent by a client, it determines whether the client has the authority to acquire data according to the data acquisition request, and obtains an authority verification result.

[0070] It should be noted that in the embodiment of the present application, the object storage system includes an object storage gateway, an underlying storage system and a memory, wherein the object storage gateway includes a flow control subsystem and a load balancing subsystem, the underlying storage system includes an index metadata system and a secondary index metadata system, and the index metadata system includes a number of object index metadata.

[0071] like Figure 2 As shown, the client sends a data acquisition request to the object storage gateway, wherein the data acquisition request is used to acquire all object index metadata in the index metadata system. When the object storage gateway receives the data acquisition request sent by the client, the object storage gateway determines whether the client has the authority to acquire all object index metadata in the index metadata system based on the data acquisition request. If the client has the authority to acquire all object index metadata in the index metadata system, the authority verification result of the data acquisition request is determined to be passed. If the client does not have the authority to acquire all object index metadata in the index metadata system, the authority verification result of the data acquisition request is determined to be failed.

[0072] Step 102: if the permission verification result is passed, load the preset secondary object index metadata into the memory.

[0073] Step 103: Obtain secondary object index metadata from the memory through the flow control subsystem.

[0074] Step 104: When the flow control subsystem obtains the secondary object index metadata, the secondary object index metadata is sent to the load balancing subsystem according to a preset target data transmission rate.

[0075] Step 105: When the load balancing subsystem receives the secondary object index metadata, the load balancing subsystem obtains object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata.

[0076] Step 106: Send the object index metadata to the client.

[0077] It should be noted that in the embodiments of the present application, Figure 2 As shown, when the permission verification result of the data acquisition request is passed, the object storage gateway obtains the secondary object index metadata from the secondary index metadata system and loads the secondary object index metadata into the memory.

[0078] When the object storage gateway completes loading the secondary object index metadata into the memory, the flow control subsystem can obtain the secondary object index metadata from the memory. When the flow control subsystem obtains the secondary object index metadata, the flow control subsystem sends the secondary object index metadata to the load balancing subsystem according to the preset target data transmission rate.

[0079] When the load balancing subsystem receives the secondary object index metadata sent by the flow control subsystem, the load balancing subsystem obtains the object index metadata corresponding to the secondary object index metadata from the index metadata system based on the secondary object index metadata, and sends the obtained object index metadata to the client.

[0080] This application obtains object index metadata corresponding to the secondary object index metadata from the index metadata system through the secondary object index metadata. The client only needs to send a data acquisition request once to obtain all object index metadata, which reduces the number of client data acquisition operations. Compared with the related technology that requires multiple data acquisition operations, it greatly improves the efficiency of data acquisition and solves the problem of low data acquisition efficiency caused by increasing the number of client call operations. In addition, after the object storage gateway receives the data acquisition request sent by the client, the application verifies the permission of the data acquisition request to ensure that only authorized clients can obtain data, thereby improving the security of data and the effectiveness of access control.

[0081] Furthermore, in the embodiment of the present application, the following steps may be included before step 101:

[0082] Step 11, pre-blocking a number of object index metadata to obtain a number of secondary object index metadata, wherein the secondary object index metadata includes a secondary index metadata identification number, a starting object key tag and the number of objects.

[0083] Step 12: Store a number of secondary object index metadata into a secondary index metadata system.

[0084] It should be noted that in the embodiments of the present application, Figure 2 As shown, the index metadata system includes several object index metadata, where each object index metadata includes an object ID (IDentity, identity number), an object name, and an object metadata set. The object metadata set may include an object key (Key), an object entity tag (ETag), an object size (Size), an object creation time (LastModified), an object storage class (StorageClass), an object owner (Owner), an object restore status (RestoreStatus), etc.

[0085] Several object index metadata in the index metadata system are divided into blocks to obtain several secondary object index metadata, wherein each secondary object index metadata includes a secondary index metadata ID, a starting object key tag and the number of objects, and the number of objects defaults to 1000.

[0086] For example: if the index metadata system includes 10,000 object index metadata, the 10,000 object index metadata can be divided into 10 blocks, each block includes 1,000 object index metadata, where one block is a secondary object index metadata, and each secondary object index metadata includes a secondary index metadata ID, a starting object key tag, and the number of objects. The starting object key tag refers to sorting all object index metadata corresponding to the secondary index metadata ID according to the object ID, and marking the object index metadata ranked first. It should be noted that the number of blocks is determined by the number of object index metadata in the index metadata system. If the number of object index metadata in the index metadata system increases, the number of blocks needs to be adjusted.

[0087] The obtained secondary object index metadata is stored in the secondary index metadata system in the format of {Id, start_after, number}. Figure 2As shown, the secondary index metadata system stores N secondary object index metadata, including: {0, obj01, 1000}, {1, obj02, 1000}...{N, obj0N, 1000}.

[0088] The present application obtains secondary object index metadata by dividing large-scale object index metadata into multiple small blocks, which not only reduces the amount of data that needs to be processed in a single data acquisition operation, significantly shortens the response time of each data acquisition operation, and improves the efficiency of data acquisition, but also avoids excessive occupation of system resources by a single data acquisition operation, reduces system load, and improves system stability and reliability.

[0089] Furthermore, in the embodiment of the present application, the following steps may be included after step 102:

[0090] Step 21: After loading the secondary object index metadata into the memory, obtain the newly added object index metadata and / or deleted object index metadata.

[0091] Step 22: Store the newly added object index metadata and / or deleted object index metadata into the incremental index metadata system.

[0092] It should be noted that in the embodiment of the present application, as objects are continuously added and / or deleted, the object index metadata in the index metadata system is also continuously added and / or deleted. When the secondary object index metadata is loaded into the memory, the newly added and / or deleted object index metadata is no longer stored in the index metadata system, but is stored in the incremental index metadata system.

[0093] Specifically, Figure 2 As shown, the underlying storage system includes an incremental index metadata system, which, after completing loading the secondary object index metadata into the memory, obtains the newly added object index metadata and / or deleted object index metadata, and stores the newly added object index metadata and / or deleted object index metadata in the incremental index metadata system.

[0094] The present application stores the newly added and / or deleted object index metadata in an incremental index metadata system, which is separated from the index metadata system. This not only helps to avoid frequent modifications to the entire index metadata system when the object index metadata is constantly updated (objects are constantly added and deleted), but also reduces interference with the existing secondary object index metadata in the memory, making the data structure in the memory more stable and facilitating fast query and processing operations in the memory.

[0095] Furthermore, in the embodiment of the present application, the following steps may be included before step 104:

[0096] Step 31, obtaining the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate, the CPU usage rate and the data transmission rate through the flow control subsystem.

[0097] Step 32 , respectively comparing the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate, and the CPU usage rate with respective corresponding preset thresholds.

[0098] Step 33, if at least one of the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate, and the CPU usage rate is greater than or equal to the corresponding preset threshold, the data transmission rate is updated to obtain the target data transmission rate.

[0099] It should be noted that in the embodiment of the present application, the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory utilization rate, the central processing unit utilization rate and the data transmission rate are obtained through the flow control subsystem, wherein the data transmission rate refers to the data transmission rate in the current object storage system.

[0100] When the flow control subsystem obtains these data, it compares the reading capacity data of the index metadata system with its corresponding preset threshold, compares the network transmission capacity data of the object storage system with its corresponding preset threshold, compares the memory usage rate with its corresponding preset threshold, and compares the central processing unit usage rate with its corresponding preset threshold.

[0101] If at least one of the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate, and the central processing unit usage rate is greater than or equal to the corresponding preset threshold, the data transmission rate is adjusted by the flow control subsystem to obtain the adjusted data transmission rate, and the adjusted data transmission rate is the target data transmission rate.

[0102] Furthermore, the maximum value of the data transmission rate is calculated based on the capacity of the object index metadata that can be loaded into the memory. The object index metadata contains information such as the object ID, object name, and object metadata collection. In order to avoid the object index metadata being too large and affecting the performance of obtaining data, the object index metadata is limited to 100 bytes, so each secondary object index metadata will read the object index metadata and occupy a maximum of 100KB of space. The flow control subsystem controls the flow into the load balancing subsystem in the control system to read no more than 1,000 secondary index metadata IDs, that is, the maximum system load is 100MB.

[0103] In the present application, the flow control subsystem can adjust the data transmission rate based on the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory utilization rate, and the central processing unit utilization rate. This can prevent the secondary object index metadata from being sent too quickly, causing the storage engine or client to be unable to process it, thereby ensuring the stable operation of the object storage system.

[0104] Furthermore, in the embodiment of the present application, step 106 may also include the following steps:

[0105] Sub-step 1061, sending the object index metadata to the aggregation computing subsystem through the load balancing subsystem.

[0106] Sub-step 1062, when the aggregate computing subsystem receives the object index metadata, the object index metadata is sent to the client in descending order of the secondary index metadata identification number.

[0107] It should be noted that in the embodiments of the present application, Figure 2 As shown, the load balancing subsystem sends the acquired object index metadata to the aggregation computing subsystem.

[0108] The object index metadata obtained based on the secondary object index metadata should be in lexicographic order. Since the query time of object index metadata varies for different secondary object index metadata, directly returning the object index metadata to the client does not meet the requirement that all object index metadata must be in lexicographic order.

[0109] Therefore, in order to ensure that the enumeration of all object index metadata meets the requirements of dictionary order, when the aggregation computing subsystem receives the object index metadata sent by the load balancing subsystem, the object index metadata must be returned to the client in descending order of the secondary index metadata ID. Specifically, the object index metadata with a large secondary index metadata ID is returned to the client first, and the object index metadata with a small secondary index metadata ID needs to be cached in memory until all the object index metadata with a high ranking of the secondary index metadata ID is returned to the client, and then the object index metadata in the memory is returned to the client to ensure the orderliness of the overall enumeration results.

[0110] In addition, when the aggregation computing subsystem cannot receive more data, the aggregation computing subsystem will send a signal to the flow control subsystem, indicating that no more data will be received for the time being. After all the object index metadata that has been transmitted is returned to the client, the data acquisition request is terminated, and a truncation mark and an end object Key (key value) are returned to the client. The truncation mark indicates that further data acquisition is required to complete all data acquisition operations, and the end object Key is used to execute the data acquisition request with the end object Key as the initial object when sending the data acquisition request again.

[0111] In this application, the aggregation computing subsystem sends the object index metadata to the client in descending order of the secondary index metadata identity number, ensuring the orderliness of the overall enumeration results. This orderliness helps the client to correctly parse and process data and improve the accuracy and predictability of data processing. In addition, when the aggregation computing subsystem cannot receive more data, this application will send a signal to the flow control subsystem, and after all the object index metadata that is already in transmission is returned to the client, it will return the truncation mark and the end object key. This mechanism ensures that when encountering system resource limitations, etc., the integrity of data acquisition can still be maintained to a certain extent. The client can perform subsequent data acquisition operations based on the truncation mark and the end object key, thereby gradually obtaining all the required data.

[0112] Furthermore, in the embodiment of the present application, the following steps may be included after step 106:

[0113] Step 41: Acquire newly added object index metadata and / or deleted object index metadata from the incremental index metadata system through the index metadata system.

[0114] Step 42: updating the object index metadata in the index metadata system according to the newly added object index metadata and / or deleted object index metadata to obtain an updated index metadata system.

[0115] Step 43: Delete the newly added object index metadata and / or deleted object index metadata stored in the incremental index metadata system.

[0116] It should be noted that in an embodiment of the present application, after the object index metadata is returned to the client in an orderly manner, the index metadata system can obtain the newly added object index metadata, or the deleted object index metadata, or the newly added object index metadata and the deleted object index metadata from the incremental index metadata system.

[0117] After obtaining these data, enable the background process to synchronize the data in the incremental index metadata system to the index metadata system, and the updated index metadata system can be obtained. Specifically, if the newly added object index metadata is obtained, the newly added object index metadata is stored in the index metadata system. If the deleted object index metadata is obtained, the corresponding object index metadata is deleted in the index metadata system. If both newly added object index metadata and deleted object index metadata are obtained, the newly added object index metadata is stored in the index metadata system, and the corresponding object index metadata is deleted in the index metadata system.

[0118] After obtaining the updated index metadata system, the newly added object index metadata and / or deleted object index metadata stored in the incremental index metadata system may be deleted.

[0119] Furthermore, the secondary index metadata system may be reconstructed according to the object index metadata stored in the updated index metadata system.

[0120] Objects in the bucket will be constantly created and deleted, and the object index metadata will also change constantly. This application introduces an incremental index metadata system, uses the incremental index metadata system to synchronously update the index metadata system, and then uses the updated index metadata system to rebuild the secondary index metadata system, which not only ensures the strong consistency and security of the data, but also improves the accuracy of object index metadata acquisition.

[0121] Corresponding to the method provided in the above-mentioned data acquisition method embodiment of the present application, the present application also provides a data acquisition system. In this embodiment, the data acquisition system includes an object storage gateway, an underlying storage system and a memory. The object storage gateway includes a flow control subsystem and a load balancing subsystem. The underlying storage system includes an index metadata system, wherein the object storage gateway establishes communication connections with the client and the memory respectively, the flow control subsystem establishes communication connections with the memory and the load balancing subsystem respectively, and the load balancing subsystem establishes a communication connection with the client.

[0122] The object storage gateway is used to determine whether the client has the authority to obtain data based on the data acquisition request when receiving the data acquisition request sent by the client, obtain the authority verification result, and load the preset secondary object index metadata into the memory if the authority verification result is passed.

[0123] The flow control subsystem is used to obtain the secondary object index metadata from the memory, and when the secondary object index metadata is obtained, the secondary object index metadata is sent to the load balancing subsystem according to a preset target data transmission rate.

[0124] The load balancing subsystem is used to, when receiving the secondary object index metadata, obtain object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata, and send the object index metadata to the client.

[0125] It should be noted that in the embodiments of the present application, Figure 3 As shown, the data acquisition system includes an object storage gateway, an underlying storage system and a memory. The object storage gateway includes a flow control subsystem, a load balancing subsystem and an aggregation computing subsystem. The underlying storage system includes an index metadata system, a secondary index metadata system and an incremental index metadata system. Among them, the client establishes a communication connection with the object storage gateway, and the object storage gateway establishes a communication connection with the underlying storage system.

[0126] like Figure 2 As shown, the client sends a data acquisition request to the object storage gateway, wherein the data acquisition request is used to acquire all object index metadata in the index metadata system. When the object storage gateway receives the data acquisition request sent by the client, the object storage gateway determines whether the client has the authority to acquire all object index metadata in the index metadata system based on the data acquisition request. If the client has the authority to acquire all object index metadata in the index metadata system, the authority verification result of the data acquisition request is determined to be passed. If the client does not have the authority to acquire all object index metadata in the index metadata system, the authority verification result of the data acquisition request is determined to be failed.

[0127] If the permission verification result of the data acquisition request is passed, the object storage gateway obtains the secondary object index metadata from the secondary index metadata system and loads the secondary object index metadata into the memory.

[0128] When the object storage gateway completes loading the secondary object index metadata into the memory, the flow control subsystem can obtain the secondary object index metadata from the memory. When the flow control subsystem obtains the secondary object index metadata, the flow control subsystem sends the secondary object index metadata to the load balancing subsystem according to the preset target data transmission rate.

[0129] When the load balancing subsystem receives the secondary object index metadata sent by the flow control subsystem, the load balancing subsystem obtains the object index metadata corresponding to the secondary object index metadata from the index metadata system based on the secondary object index metadata, and sends the obtained object index metadata to the client.

[0130] This application obtains object index metadata corresponding to the secondary object index metadata from the index metadata system through the secondary object index metadata. The client only needs to send a data acquisition request once to obtain all object index metadata, which reduces the number of client data acquisition operations. Compared with the related technology that requires multiple data acquisition operations, it greatly improves the efficiency of data acquisition and solves the problem of low data acquisition efficiency caused by increasing the number of client call operations. In addition, after the object storage gateway receives the data acquisition request sent by the client, the application verifies the permission of the data acquisition request to ensure that only authorized clients can obtain data, thereby improving the security of data and the effectiveness of access control.

[0131] Corresponding to the method provided in the above data acquisition method embodiment of the present application, see Figure 4 , the present application also provides a device block diagram of a data acquisition device. In this embodiment, the device includes:

[0132] The data acquisition request verification module 401 is used to determine whether the client has the authority to acquire data according to the data acquisition request when the object storage gateway receives the data acquisition request sent by the client, and obtain the authority verification result;

[0133] The data loading module 402 is used to load the preset secondary object index metadata into the memory when the permission verification result is passed;

[0134] The secondary object index metadata acquisition module 403 is used to acquire the secondary object index metadata from the memory through the flow control subsystem;

[0135] The secondary object index metadata transmission module 404 is used to send the secondary object index metadata to the load balancing subsystem according to a preset target data transmission rate when the flow control subsystem obtains the secondary object index metadata;

[0136] The target object index metadata acquisition module 405 is used to acquire the object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata when the load balancing subsystem receives the secondary object index metadata;

[0137] The target object index metadata feedback module 406 is used to send the object index metadata to the client.

[0138] Optionally, the underlying storage system includes a secondary index metadata system, and the object index metadata includes an object identification number, an object name, and an object metadata set;

[0139] The device includes:

[0140] A data segmentation module is used to segment a plurality of object index metadata in advance to obtain a plurality of secondary object index metadata, wherein the secondary object index metadata includes a secondary index metadata identification number, a starting object key tag and the number of objects;

[0141] The data block storage module is used to store a plurality of secondary object index metadata into the secondary index metadata system.

[0142] Optionally, the underlying storage system includes an incremental index metadata system;

[0143] The device includes:

[0144] A new / deleted data acquisition module, used to acquire the newly added object index metadata and / or deleted object index metadata after loading the secondary object index metadata into the memory;

[0145] The new / deleted data storage module is used to store the newly added object index metadata and / deleted object index metadata in the incremental index metadata system.

[0146] Optionally, the device comprises:

[0147] The data transmission rate related data acquisition module is used to obtain the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate, the CPU usage rate and the data transmission rate through the flow control subsystem;

[0148] A data transmission rate related data comparison module is used to compare the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate and the CPU usage rate with their respective corresponding preset thresholds;

[0149] The data transmission rate adjustment module is used to update the data transmission rate to obtain the target data transmission rate if at least one of the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate and the central processing unit usage rate is greater than or equal to the corresponding preset threshold.

[0150] Optionally, the object storage gateway includes an aggregate computing subsystem;

[0151] The target object index metadata feedback module 406 includes:

[0152] The object index metadata sending submodule is used to send the object index metadata to the aggregation computing subsystem through the load balancing subsystem;

[0153] The object index metadata ordered return submodule is used to send the object index metadata to the client in descending order of the secondary index metadata identity identification number when the aggregation computing subsystem receives the object index metadata.

[0154] Optionally, the device comprises:

[0155] An incremental index metadata acquisition module, used to acquire newly added object index metadata and / or deleted object index metadata from the incremental index metadata system through the index metadata system;

[0156] An index metadata system update module, used to update the object index metadata in the index metadata system according to the newly added object index metadata and / or deleted object index metadata to obtain an updated index metadata system;

[0157] The incremental index metadata deletion module is used to delete the newly added object index metadata and / or deleted object index metadata stored in the incremental index metadata system.

[0158] Figure 5 This is a structural diagram of an electronic device M00 provided in an embodiment of the present application. In the figure, the electronic device M00 includes a processor M01 and a memory M02. The memory M02 stores programs or instructions that can be run on the processor M01. When the program or instruction is executed by the processor M01, the various steps of the above-mentioned data acquisition method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0159] In an embodiment of the present application, the memory M02 may be used to store software programs and various data. The memory M02 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory M02 may include a volatile memory or a non-volatile memory, or the memory M02 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory M02 in the embodiment of the present application includes but is not limited to these and any other suitable types of memories.

[0160] The processor M01 may include one or more processing units; optionally, the processor M01 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor M01.

[0161] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned data acquisition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0162] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0163] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data acquisition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0164] It should be understood that the chip involved in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0165] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned data acquisition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0166] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0167] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0168] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data acquisition method, characterized in that: Applied to an object storage system, the object storage system includes an object storage gateway, an underlying storage system and a memory, wherein the object storage gateway includes a flow control subsystem and a load balancing subsystem, the underlying storage system includes an index metadata system, the index metadata system includes a plurality of object index metadata, and the method includes: When the object storage gateway receives a data acquisition request sent by a client, determining whether the client has the authority to acquire data according to the data acquisition request, and obtaining an authority verification result; If the permission verification result is passed, loading the preset secondary object index metadata into the memory; Acquire the secondary object index metadata from the memory through the flow control subsystem; When the flow control subsystem obtains the secondary object index metadata, the secondary object index metadata is sent to the load balancing subsystem according to a preset target data transmission rate; When the load balancing subsystem receives the secondary object index metadata, acquiring the object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata; The object index metadata is sent to the client.

2. The method according to claim 1, characterized in that The underlying storage system includes a secondary index metadata system, and the object index metadata includes an object identification number, an object name, and an object metadata set; Before the step of determining whether the client has the authority to obtain data according to the data acquisition request and obtaining the authority verification result, the method includes: Pre-blocking the plurality of object index metadata to obtain a plurality of secondary object index metadata, wherein the secondary object index metadata includes a secondary index metadata identification number, a starting object key tag, and the number of objects; The plurality of secondary object index metadata are stored in the secondary index metadata system.

3. The method according to claim 1, characterized in that The underlying storage system includes an incremental index metadata system; After the step of loading the preset secondary object index metadata into the memory, the method includes: Upon completion of loading the secondary object index metadata into the memory, acquiring newly added object index metadata and / or deleted object index metadata; The newly added object index metadata and / or the deleted object index metadata are stored in the incremental index metadata system.

4. The method according to claim 1, characterized in that Before the step of sending the secondary object index metadata to the load balancing subsystem according to the preset target data transmission rate, the method includes: Acquire the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the usage rate of the memory, the usage rate of the central processing unit and the data transmission rate through the flow control subsystem; Respectively comparing the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the usage rate of the memory, and the usage rate of the central processing unit with respective corresponding preset thresholds; If at least one of the reading capacity data of the index metadata system, the network transmission capacity data of the object storage system, the memory usage rate, and the central processing unit usage rate is greater than or equal to the corresponding preset threshold, the data transmission rate is updated to obtain the target data transmission rate.

5. The method according to claim 2, characterized in that: The object storage gateway includes an aggregate computing subsystem; The sending the object index metadata to the client comprises: Sending the object index metadata to the aggregate computing subsystem through the load balancing subsystem; When the aggregate computing subsystem receives the object index metadata, the object index metadata is sent to the client in descending order of the secondary index metadata identification numbers.

6. The method according to claim 3, characterized in that After the step of sending the object index metadata to the client, the method includes: Acquire the newly added object index metadata and / or the deleted object index metadata from the incremental index metadata system through the index metadata system; The object index metadata in the index metadata system is updated according to the newly added object index metadata and / or the deleted object index metadata to obtain the updated index metadata system; The newly added object index metadata and / or the deleted object index metadata stored in the incremental index metadata system are deleted.

7. A data acquisition system, characterized in that: The data acquisition system includes an object storage gateway, an underlying storage system and a memory, the object storage gateway includes a flow control subsystem and a load balancing subsystem, the underlying storage system includes an index metadata system, wherein the object storage gateway establishes communication connections with the client and the memory respectively, the flow control subsystem establishes communication connections with the memory and the load balancing subsystem respectively, and the load balancing subsystem establishes communication connection with the client; The object storage gateway is used to, when receiving a data acquisition request sent by a client, determine whether the client has the authority to acquire data according to the data acquisition request, obtain a permission verification result, and if the permission verification result is passed, load the preset secondary object index metadata into the memory; The flow control subsystem is used to obtain the secondary object index metadata from the memory, and when the secondary object index metadata is obtained, send the secondary object index metadata to the load balancing subsystem according to a preset target data transmission rate; The load balancing subsystem is used to, upon receiving the secondary object index metadata, obtain the object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata, and send the object index metadata to the client.

8. A data acquisition device, characterized in that: Applied to an object storage system, the object storage system includes an object storage gateway, an underlying storage system and a memory, wherein the object storage gateway includes a flow control subsystem and a load balancing subsystem, the underlying storage system includes an index metadata system, the index metadata system includes a plurality of object index metadata, and the device includes: A data acquisition request verification module is used to determine whether the client has the authority to acquire data according to the data acquisition request when the object storage gateway receives the data acquisition request sent by the client, and obtain a permission verification result; A data loading module, used for loading preset secondary object index metadata into the memory if the permission verification result is passed; A secondary object index metadata acquisition module, used to acquire the secondary object index metadata from the memory through the flow control subsystem; A secondary object index metadata data transmission module, configured to send the secondary object index metadata to the load balancing subsystem according to a preset target data transmission rate when the flow control subsystem obtains the secondary object index metadata; a target object index metadata acquisition module, configured to acquire, when the load balancing subsystem receives the secondary object index metadata, the object index metadata corresponding to the secondary object index metadata from the index metadata system according to the secondary object index metadata; The target object index metadata feedback module is used to send the object index metadata to the client.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data acquisition method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data acquisition method as claimed in any one of claims 1 to 7.

Images