Object storage method, device, system, electronic device and storage medium

Abstract

The present disclosure relates to a method, device, system, electronic device and storage medium for object storage, comprising: after receiving an operation request for a target object in an object storage, accessing a metadata cache system, wherein the metadata cache system includes cache metadata of the object; in the case where cache metadata corresponding to the target object exists in the metadata cache system, determining the target metadata according to the operation request; and executing the operation indicated by the operation request according to the target metadata, reducing the operation on the object storage by accessing the metadata cache system, and improving the interaction efficiency between the big data component and the object storage.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data storage, and in particular to an object storage method, device, system, electronic device and storage medium. Background Art

[0002] With the development of big data and artificial intelligence (AI) technology, various industries have become increasingly aware of the value of big data technology to their own products and services. Big data applications and big data platforms have become the core technologies of various information technology companies and Internet companies. Big data applications are inseparable from the big data scheduling system, which includes big data components. The big data components mainly provide computing and storage support for business tasks. Traditionally, we use local disks to store data. With the development of object storage, its low cost, high stability, and unlimited expansion capabilities are becoming more and more suitable for big data products.

[0003] In the prior art, in the process of storing big data products in object storage, due to the inherent limitations of object storage, there is a large performance loss in the process of storing big data products in object storage. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device, system, electronic device and storage medium for object storage to improve the interaction efficiency between big data components and object storage.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for object storage, including:

[0006] After receiving an operation request for a target object in the object storage, accessing a metadata cache system, wherein the metadata cache system includes cache metadata of the object;

[0007] In a case where cache metadata corresponding to the target object exists in the metadata cache system, determining the target metadata according to the operation request;

[0008] The operation indicated by the operation request is performed according to the target metadata.

[0009] Optionally, when cache metadata corresponding to the target object exists in the metadata cache system, determining the target metadata according to the operation request includes:

[0010] In a case where cache metadata corresponding to the target object exists in the metadata cache system, modifying the cache metadata according to the operation request to obtain target metadata;

[0011] The performing the operation indicated by the operation request according to the target metadata includes:

[0012] The target object in the object store is modified according to the target metadata.

[0013] Optionally, when cache metadata corresponding to the target object exists in the metadata cache system, determining the target metadata according to the operation request includes:

[0014] In a case where cache metadata corresponding to the target object exists in the metadata cache system, the target metadata is acquired from the metadata cache system according to the operation request.

[0015] Optionally, the method further includes:

[0016] When it is determined that the cache metadata corresponding to the target object does not exist in the metadata cache system, the target object in the object storage is modified according to the operation request.

[0017] Optionally, the method further includes:

[0018] Determine whether cache metadata corresponding to the target object exists in the metadata cache system according to the request information in the operation request and the cache information in the metadata cache system.

[0019] Optionally, determining whether cache metadata corresponding to the target object exists in the metadata cache system according to request information in the operation request and cache information in the metadata cache system includes:

[0020] When the request information in the operation request is consistent with the cache information in the metadata cache system, determining that cache metadata corresponding to the target object exists in the metadata cache system;

[0021] When the request information in the operation request is inconsistent with the cache information in the metadata cache system, it is determined that the cache metadata corresponding to the target object does not exist in the metadata cache system.

[0022] Optionally, the request information includes request attribute information and request operation information, and the cache information includes cache attribute information and cache operation information;

[0023] Determining whether there is cache metadata corresponding to the target object in the metadata cache system according to the request information in the operation request and the cache information in the metadata cache system includes:

[0024] When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, determining whether the request operation information in the operation request is consistent with the cache operation information in the metadata cache system;

[0025] When the request attribute information in the operation request is inconsistent with the cache attribute information in the metadata cache system, the target object in the object storage is modified according to the operation request.

[0026] Optionally, when the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, determining whether the request operation information in the operation request is consistent with the cache operation information in the metadata cache system includes:

[0027] When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is consistent with the cache operation information in the metadata cache system, determining the target metadata according to the operation request;

[0028] When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is inconsistent with the cache operation information in the metadata cache system, modify the target object in the object storage according to the operation request.

[0029] Optionally, after modifying the target object in the object storage according to the target metadata, the method further includes:

[0030] The result of performing the operation of modifying the target object in the object storage is uploaded to the metadata cache system.

[0031] In a second aspect, an embodiment of the present disclosure provides an object processing system, including: a client, a big data component, a metadata cache system, and an object storage;

[0032] After receiving the operation request for the target object sent by the big data component, the client accesses the metadata cache system; after confirming that the cached metadata corresponding to the target object exists in the metadata cache system, the client determines the target metadata according to the operation request, and performs the operation indicated by the operation request according to the target metadata.

[0033] In a third aspect, an embodiment of the present disclosure provides an object storage device, including:

[0034] A metadata cache system access module, configured to access a metadata cache system after receiving an operation request for a target object in the object storage, wherein the metadata cache system includes cache metadata of the object;

[0035] A modification module, configured to determine target metadata according to the operation request if cache metadata corresponding to the target object exists in the metadata cache system;

[0036] An operation module is used to execute the operation indicated by the operation request according to the target metadata.

[0037] Optionally, modify the module specifically for:

[0038] In a case where cache metadata corresponding to the target object exists in the metadata cache system, modifying the cache metadata according to the operation request to obtain target metadata;

[0039] The operation module is specifically used for:

[0040] The operation indicated by the operation request is performed in the object storage according to the target metadata.

[0041] Optionally, the modification module is further specifically used to:

[0042] In a case where cache metadata corresponding to the target object exists in the metadata cache system, acquiring target metadata from the metadata cache system according to the operation request;

[0043] The operation module is also specifically used for:

[0044] The operation indicated by the operation request is performed in the metadata cache system according to the target metadata.

[0045] Optionally, also include:

[0046] The target object operation module is used to modify the target object in the object storage according to the operation request when it is determined that the cache metadata corresponding to the target object does not exist in the metadata cache system.

[0047] Optionally, also include:

[0048] The judgment module is used to determine whether there is cache metadata corresponding to the target object in the metadata cache system according to the request information in the operation request and the cache information in the metadata cache system.

[0049] Optionally, also include:

[0050] A first determining unit, configured to determine that cache metadata corresponding to the target object exists in the metadata cache system when request information in the operation request is consistent with cache information in the metadata cache system;

[0051] The second determining unit is configured to determine that cache metadata corresponding to the target object does not exist in the metadata cache system when request information in the operation request is inconsistent with cache information in the metadata cache system.

[0052] Optionally, also include:

[0053] A first judgment unit, configured to judge whether the request operation information in the operation request is consistent with the cache operation information in the metadata cache system when the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system;

[0054] The first access unit is used to access the target object in the object storage according to the operation request when the request attribute information in the operation request is inconsistent with the cache attribute information in the metadata cache system.

[0055] Optionally, also include:

[0056] An operation unit, configured to determine target metadata according to the operation request when request attribute information in the operation request is consistent with cache attribute information in the metadata cache system and request operation information in the operation request is consistent with cache operation information in the metadata cache system;

[0057] The second access unit is used to access the target object in the object storage according to the operation request when the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is inconsistent with the cache operation information in the metadata cache system.

[0058] Optionally, also include:

[0059] The upload module is used to upload the result of the operation of modifying the target object in the object storage to the metadata cache system.

[0060] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0061] one or more processors;

[0062] a storage device for storing one or more programs,

[0063] When the one or more programs are executed by the one or more processors, the one or more processors implement any method as described in the first aspect.

[0064] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any method described in the first aspect.

[0065] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0066] The object storage method, device, electronic device and storage medium provided by the embodiments of the present disclosure access the metadata cache system after receiving an operation request for a target object in the object storage, and when it is determined that the cache metadata corresponding to the target object exists in the metadata cache system, the cache metadata is modified according to the operation request to obtain the target metadata, and the target object in the object storage is modified according to the target metadata. By accessing the metadata cache system during the interaction between the big data component and the object storage, determining the target metadata according to the operation request for the target object in the object storage, and executing the operation indicated by the operation request according to the target metadata, it is achieved that during the interaction between the big data component and the object storage, the operation on the object storage is reduced by accessing the metadata cache system, and the interaction efficiency between the big data component and the object storage is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0069] Figure 1 is a schematic diagram of an application scenario of object storage provided by an embodiment of the present disclosure;

[0070] Figure 2 It is a flowchart of an object storage method provided by an embodiment of the present disclosure;

[0071] Figure 3 is a flowchart of another object storage method provided by an embodiment of the present disclosure;

[0072] Figure 4 is a flowchart of another object storage method provided by an embodiment of the present disclosure;

[0073] Figure 5 is a flowchart of another object storage method provided by an embodiment of the present disclosure;

[0074] Figure 6 is a schematic diagram of the structure of an object storage device provided by an embodiment of the present disclosure;

[0075] Figure 7It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0076] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0077] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0078] Exemplarily, the present disclosure provides an object storage method, apparatus, electronic device and computer storage medium, which, during the interaction between a big data component and object storage, accesses a metadata cache system and makes modifications in the metadata cache system.

[0079] The object storage method disclosed in the present invention is executed by an electronic device or an application, web page, public account, etc. in the electronic device. The electronic device can be a tablet computer, a mobile phone, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart TV, a smart screen, a high-definition TV, a 4K TV, a smart speaker, a smart projector, and other devices. The present invention does not impose any restrictions on the specific types of electronic devices. Specifically, Figure 1 is a schematic diagram of an application scenario of object storage provided by an embodiment of the present disclosure, such as Figure 1 As shown, when the client receives an operation request for a target object in an object storage from a big data component, the client accesses the metadata cache system. When there is cached metadata corresponding to the target object in the metadata cache system, the client modifies the cached metadata corresponding to the target object in the metadata cache system to obtain the target metadata according to the operation request for the target object in the object storage from the received big data component. The client modifies the target object in the object storage at the back end according to the target metadata obtained after the modification in the metadata cache system, thereby solving the performance loss caused by the direct interaction between the big data component and the object storage, ensuring data consistency and improving the interaction efficiency between the big data component and the object storage.

[0080] The present disclosure does not limit the type of operating system of the electronic device, for example, Android system, Linux system, Windows system, iOS system, etc.

[0081] Based on the foregoing description, the present disclosure will take an electronic device as an example in an embodiment and combine it with an application scenario to elaborate on the object storage method provided by the present disclosure.

[0082] like Figure 2 As shown, the object storage method includes:

[0083] S110: After receiving an operation request for a target object in the object storage, access a metadata cache system.

[0084] The metadata cache system includes cache metadata of objects.

[0085] Specifically, the metadata cache system is an additional system added to the object storage method provided by the present application, which mainly stores the cache metadata of the objects in the object storage. In the prior art, the properties of the storage objects are stored in the metadata service, and the data of the storage objects are stored in the object storage. Since the object storage is a closed product, the object storage itself will have performance loss during the user's operation on the storage objects of the object storage. Specifically, the process of operating the storage objects of the object storage will put the intermediate results into the cache, and then execute the target operation. Since the prior art requires at least two operation processes to operate the storage objects of the object storage, the performance loss of the big data component directly interacting with the object storage is relatively large. Based on this, the object storage method provided by the embodiment of the present disclosure, after receiving an operation request for the target object in the object storage, first accesses the metadata cache system to avoid direct access to the object storage.

[0086] Operation requests include opening, creating, renaming, deleting, batch querying, creating empty directories, requesting file information, etc. In the process of direct interaction between the big data component and the object storage, an exemplary operation request for renaming the target object in the object storage is:

[0087] ListObjectsRequest(1)

[0088] HeadObjectResult(1)

[0089] CopyObjectRequest(A)

[0090] DeleteObject(A)

[0091] That is, first request the source file information, then request the destination file information. After obtaining the destination file information, if the destination file information is a file, call the ks3 copy function, and then delete the file. If the destination file information is a directory, call the sub-file rename method in a loop to delete the files in batches. Therefore, when the big data component interacts directly with the object storage, the steps that need to be experienced are the List-Copy-Delect process. Since the interaction between the big data component and the object storage is through network interaction, there is a certain performance loss in network interaction, and the ListObject operation for the object storage needs to traverse all files in the bucket, so the performance loss is large, and the big data component accesses the object storage slowly, which affects the interaction performance between the big data component and the object storage.

[0092] In addition, during the rename process, during the interaction between the big data component and the object storage, the big data component first divides the files corresponding to the rename into multiple tasks, first modifies the file name corresponding to each task and puts it into the intermediate file, and after the file name corresponding to each task is modified, the file names of each task corresponding to the intermediate file are finally renamed, which will cause a large performance loss for the object storage itself.

[0093] In order to reduce the performance loss of the large array component directly interacting with the object storage, the object storage method provided by the embodiment of the present disclosure accesses the metadata cache system after receiving an operation request for the target object in the object storage. Exemplarily, when the client receives a rename operation request for the target object A in the object storage, the client first accesses the metadata cache system.

[0094] S120: If there is cache metadata corresponding to the target object in the metadata cache system, determine the target metadata according to the operation request.

[0095] The metadata cache system is used to store information about objects in the object storage, such as file name, file version, file storage path, file permissions, file size, and the last modification time of the file. When the client accesses the metadata cache system, and there is cache metadata corresponding to the target object in the metadata cache system, after the client receives an operation request for the target object in the object storage, according to the operation request, the cache metadata corresponding to the target object is modified in the metadata cache system to the target metadata. Exemplarily, the client receives a renaming of the target object A in the object storage to B, and the client finds the cache metadata corresponding to the target object A in the metadata cache system in the metadata cache system. Exemplarily, the cache metadata stored in the metadata cache system for the target object A is file-A. When there is cache metadata file-A of the target object A in the metadata cache system, the client modifies the cache metadata file-A corresponding to the target object A in the metadata cache system to file-B. That is, in the process of renaming the target object in the object storage, when requesting the source file information, that is, in the List step, the metadata cache service is queried, and the cache metadata corresponding to the target object in the metadata cache system is modified to implement rename, so the object storage file is not moved, and the deletion operation can be performed in the background.

[0096] The big data component interacts with the object storage through the client. The client will first access the metadata cache system and determine whether it needs to access the object storage based on the result of accessing the metadata cache system. The entire metadata cache system is a cluster mode and uses the raft protocol to ensure data availability.

[0097] Specifically, according to the operation request, the target metadata is determined to include:

[0098] According to the operation request, the cache metadata is modified to obtain the target metadata; or, according to the operation request, the target metadata is obtained from the cache metadata system.

[0099] Among them, according to the operation request, a possible implementation method of modifying the cache metadata to obtain the target metadata is as follows: the client receives the renaming of the target object A in the object storage to B, the client finds the cache metadata corresponding to the target object A in the metadata cache system in the metadata cache system, the cache metadata of the target object A stored in the metadata cache system is file-A, when the cache metadata file-A of the target object A exists in the metadata cache system, the client modifies the cache metadata file-A corresponding to the target object A in the metadata cache system to file-B. That is, in the process of renaming the target object in the object storage, the cache metadata corresponding to the target object in the metadata cache system is modified to obtain the target metadata, reducing the interaction process between the big data component and the object storage directly.

[0100] According to the operation request, one possible implementation method of obtaining the target metadata from the metadata cache system is:

[0101] The client receives and opens the target object A in the object storage, and finds the corresponding cache metadata of the target object A in the metadata cache system in the metadata cache system, and uses the cache metadata of the target object A stored in the metadata cache system as the target metadata.

[0102] S130: Execute the operation indicated by the operation request according to the target metadata.

[0103] After the cache metadata is modified according to the operation request to obtain the target metadata, the target object in the object storage is modified according to the target metadata in the metadata cache system, that is, the cache metadata file-A of the target object A is modified in the metadata cache system to obtain the target metadata file-B, and the client's backend synchronizes the operation with the target object in the object storage, that is, the client's backend modifies the name of the target object A to B in the object storage according to the modified target metadata, and deletes it at the backend.

[0104] Exemplarily, Table 1 is a correspondence between metadata in a metadata cache system and objects in object storage, as shown in Table 1:

[0105] Table 1 Metadata and object correspondence

[0106]

[0107] It should be noted that the above embodiment exemplifies the process of renaming the target object in the object storage. The object storage method provided by the embodiment of the present disclosure is also applicable to other operation requests, such as deletion, batch query, request for file directory, etc., and the embodiment of the present disclosure is not limited to this.

[0108] In addition, in the process of modifying the target object in the object storage according to the target metadata, since the object storage is a closed system, the process of modifying the target object includes modifying the metadata stored in the metadata service of the storage object and the data of the storage object.

[0109] Compared with the prior art, the object storage method provided in the embodiment of the present application obtains the target metadata in the metadata cache system, and then performs the modification of the target metadata of the target object in the object storage. Therefore, the process of modifying the target metadata of the target object in the object storage only needs to be executed once, reducing the performance loss of the object storage itself.

[0110] Specifically, according to the target metadata, the operation indicated by the operation request is performed including:

[0111] According to the target metadata, the operation indicated by the operation request is performed in the object storage; or, according to the target metadata, the operation indicated by the operation request is performed in the metadata cache system.

[0112] Among them, according to the target metadata, an implementable method of performing the operation indicated by the operation request in the object storage includes: after modifying the cache metadata file-A of the target object A to obtain the target metadata file-B in the metadata cache system, the client's backend synchronizes the target object in the object storage, that is, the client's backend modifies the name of the target object A to B in the object storage according to the modified target metadata.

[0113] According to the target metadata, a possible implementation method of performing the operation indicated by the operation request in the metadata cache system includes: the client searches the metadata cache system for the cache metadata corresponding to the target object A in the metadata cache system, and uses the obtained cache metadata of the target object A stored in the metadata cache system as the target metadata. At this time, the target metadata of the target object is obtained in the metadata cache system, and the operation corresponding to the operation request is performed in the metadata cache system.

[0114] The object storage method provided by the embodiment of the present disclosure accesses the metadata cache system after receiving an operation request for a target object in the object storage, and when it is determined that the cache metadata corresponding to the target object exists in the metadata cache system, modifies the cache metadata according to the operation request to obtain the target metadata, and modifies the target object in the object storage according to the target metadata. By accessing the metadata cache system during the interaction between the big data component and the object storage, determining the target metadata according to the operation request for the target object in the object storage, and executing the operation indicated by the operation request according to the target metadata, it is achieved that during the interaction between the big data component and the object storage, the operation on the object storage is reduced by accessing the metadata cache system, and the interaction efficiency between the big data component and the object storage is improved.

[0115] Figure 3 is a flow chart of another object storage method provided by an embodiment of the present disclosure. This embodiment is based on the above embodiment. Figure 3 As shown, before step S120, the following steps are also included:

[0116] S111. Determine whether there is cache metadata corresponding to the target object in the metadata cache system according to the request information in the operation request and the cache information in the metadata cache system.

[0117] When the client receives an operation request for the target object in the object storage, the client accesses the metadata cache system. The metadata cache system sends the ID corresponding to the target object to the storage node of the target object according to the operation request received from the large array component, and responds to the client with the ID mapping table corresponding to the target object and the storage node location information of the target object. After receiving the response information from the metadata cache system, the client sends a request command packet to the storage node corresponding to the target object and establishes a connection with the storage node of the target object.

[0118] Specifically, if the operation request to rename the target object in the object storage is:

[0119] ListObjectsRequest(1)

[0120] HeadObjectResult(1)

[0121] CopyObjectRequest(A)

[0122] DeleteObject(A)

[0123] At this time, the target object is determined to be A according to the operation request, and the metadata cache system is accessed. The metadata cache system determines that the ID corresponding to the target object A is ID1, and the storage node of the target object A is node 1. The metadata cache system stores the ID1 corresponding to the target object A on node 1 according to the operation request, and the metadata cache system returns the ID1 mapping table corresponding to the target object A and the node 1 location information of the target object node to the client. After receiving the response information of the metadata cache system, the client sends a request command packet to the node 1 corresponding to the target object A and establishes a connection with the node 1 of the target object. At this time, the mapping table corresponding to the target object is stored in node 1, that is, the cache information of the target object stored in the metadata cache system. At this time, the client determines whether there is cache metadata corresponding to the target object in the metadata cache system based on the request information in the operation request and the cache information in the metadata cache system.

[0124] Specifically, when the request information in the operation request is consistent with the cache information in the metadata cache system, it is determined that cache metadata corresponding to the target object exists in the metadata cache system.

[0125] When the request information in the operation request is inconsistent with the cache information in the metadata cache system, it is determined that the cache metadata corresponding to the target object does not exist in the metadata cache system.

[0126] Exemplarily, when the request information in the request operation is to rename the target object A, at this time, the request information includes the name and operation information of the target object, the name of the target object is A, and the operation information of the target object is renaming. When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is consistent with the cache operation information in the metadata cache system, it is determined that the cache metadata corresponding to the target object exists in the metadata cache system. If the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is inconsistent with the cache operation information in the metadata cache system, or the request attribute information in the operation request is inconsistent with the cache attribute information in the metadata cache system, or the request attribute information in the operation request is inconsistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is inconsistent with the cache operation information in the metadata cache system, it is determined that the cache metadata corresponding to the target object does not exist in the metadata cache system.

[0127] The object storage method provided by the embodiment of the present disclosure determines whether cache metadata corresponding to the target object exists in the metadata cache system based on the request information in the operation request and the cache information in the metadata cache system, enables the client to establish a connection with the storage node corresponding to the target object on the metadata cache system, and then determines whether cache metadata corresponding to the target object exists in the metadata cache system based on the request information in the operation request and the cache information on the storage node corresponding to the target object in the metadata cache system.

[0128] Figure 4 is a flow chart of another object storage method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is based on the above embodiment. Figure 4 As shown, one possible implementation of step S111 includes:

[0129] S1111. Determine whether the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system. When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, execute step S1112. When the request operation information in the operation request is inconsistent with the cache operation information in the metadata cache system, execute step S1113.

[0130] S1112: Determine whether the requested operation information in the operation request is consistent with the cache operation information in the metadata cache system. When the requested operation information in the operation request is consistent with the cache operation information in the metadata cache system, execute step S1114. When the requested operation information in the operation request is inconsistent with the cache operation information in the metadata cache system, execute step S1113.

[0131] S1113. Modify the target object in the object storage according to the operation request.

[0132] S1114. Determine target metadata according to the operation request.

[0133] Specifically, the request information includes request attribute information and request operation information, and the cache information includes cache attribute information and cache operation information. The attribute information includes file name, file version, file storage path, file permissions, etc., and the operation information includes rename(), delete(), and listStatus().

[0134] When the client receives an operation request for the target object in the object storage, the client accesses the metadata cache system to determine whether there is cache attribute information corresponding to the target object in the metadata cache system. Assuming that the metadata cache system stores the cache attribute information corresponding to the target object, the client executes step S1112 to determine whether there is cache operation information corresponding to the target object in the data service. Assuming that the metadata cache system stores the cache operation information corresponding to the target object, the metadata cache system can implement the operation on the target object in the operation request. At this time, the client determines the target metadata based on the operation request.

[0135] For example, assuming that the metadata cache system stores cache metadata with attribute M and corresponding value m1, and in the operation request, the request attribute corresponding to the target object is also M, and the corresponding value m2. After determining that attribute M has been stored in the metadata cache system, determine whether the request operation information in the operation request is consistent with the cache operation information in the metadata cache system. If the request operation information in the operation request is consistent with the cache operation information in the metadata cache system, then use the value m2 corresponding to the request attribute M in the operation request to update the value m1 corresponding to the cache attribute M in the metadata cache system, so that the value corresponding to the cache attribute M is m2, and the updated metadata is set as the target metadata.

[0136] For example, suppose the metadata cache system stores cache metadata with attribute N and corresponding value n1. In the operation request, the request attribute corresponding to the target object is also M and the corresponding value m1. When it is determined that the metadata cache system does not store cache metadata with attribute M, the object storage is directly accessed according to the operation request to modify the target object in the object storage, that is, the value corresponding to the target object in the object storage is modified to m1.

[0137] Figure 5 is a flow chart of another object storage method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is based on the above embodiment. Figure 5As shown, the method also includes:

[0138] S140: Upload the result of executing the operation to modify the target object in the object storage to the metadata cache system.

[0139] It should be noted that, in this embodiment, after modifying the target object in the object storage according to the target metadata, the time of modifying the object in the object storage is obtained, and the current time is uploaded to the metadata cache system, that is, the log of the operation on the object in the object storage is recorded in the metadata cache system, that is, the operation time of executing the operation is recorded in the log, so that the time of executing the modification of the target object metadata in the object storage can be known by checking the log later, that is, the time of updating the cached metadata in the metadata cache system can be known, so as to analyze and manage the status of the target object, thereby further improving the user experience.

[0140] Based on the above embodiments, this embodiment further illustrates and optimizes the technical solution. Specifically, the attributes of the target metadata corresponding to the target object in the operation request and the corresponding value are written into the cache list of the metadata cache system to modify the cache metadata corresponding to the target stored in the metadata cache system.

[0141] The present disclosure also provides an object storage system. Figure 1 The object storage system includes: a client, a big data component, a metadata cache system and an object storage; after receiving an operation request for a target object sent by the big data component, the client accesses the metadata cache system; after confirming that the metadata cache system has cached metadata corresponding to the target object, the client determines the target metadata according to the operation request, and performs the operation indicated by the operation request according to the target metadata.

[0142] When the client receives an operation request for a target object in the object storage from a big data component, the client accesses the metadata cache system. If there is cached metadata corresponding to the target object in the metadata cache system, the client determines the target metadata based on the operation request for the target object in the object storage from the received big data component, and modifies the target object in the object storage at the back end based on the determined target metadata, thereby solving the performance loss caused by the direct interaction between the big data component and the object storage, ensuring data consistency, and improving the interaction efficiency between the big data component and the object storage.

[0143] Figure 6 is a schematic diagram of a structure of an object storage device provided by an embodiment of the present disclosure, such as Figure 6 As shown, the object storage device includes:

[0144] The metadata cache system access module 610 accesses the metadata cache system after receiving an operation request for a target object in the object storage, wherein the metadata cache system includes cache metadata of the object;

[0145] A modification module 620 is used to determine the target metadata according to the operation request when there is cache metadata corresponding to the target object in the metadata cache system;

[0146] The operation module 630 is used to execute the operation indicated by the operation request according to the target metadata.

[0147] In the object storage device provided by the embodiment of the present disclosure, after receiving an operation request for a target object in the object storage, the metadata cache system access module accesses the metadata cache system, and when determining that the cache metadata corresponding to the target object exists in the metadata cache system, the modification module determines the target metadata according to the operation request, and the operation module modifies and executes the operation indicated by the operation request according to the target metadata. By accessing the metadata cache system during the interaction between the big data component and the object storage, determining the target metadata according to the operation request for the target object in the object storage, and executing the operation indicated by the operation request according to the target metadata, it is achieved that during the interaction between the big data component and the object storage, the operation on the object storage is reduced by accessing the metadata cache system, and the interaction efficiency between the big data component and the object storage is improved.

[0148] Optionally, modify the module specifically for:

[0149] In the case where cache metadata corresponding to the target object exists in the metadata cache system, modifying the cache metadata according to the operation request to obtain the target metadata;

[0150] The operation module is specifically used for:

[0151] Performs the operation indicated by the operation request in the object store according to the target metadata.

[0152] Optionally, the modification module is further specifically used to:

[0153] When there is cache metadata corresponding to the target object in the metadata cache system, the target metadata is obtained from the metadata cache system according to the operation request;

[0154] The operation module is also specifically used for:

[0155] The operation indicated by the operation request is performed in the metadata cache system according to the target metadata.

[0156] Optionally, also include:

[0157] The target object operation module is used to modify the target object in the object storage according to the operation request when it is determined that the cache metadata corresponding to the target object does not exist in the metadata cache system.

[0158] Optionally, also include:

[0159] The judgment module is used to determine whether there is cache metadata corresponding to the target object in the metadata cache system according to the request information in the operation request and the cache information in the metadata cache system.

[0160] Optionally, also include:

[0161] A first determining unit, configured to determine that cache metadata corresponding to the target object exists in the metadata cache system when request information in the operation request is consistent with cache information in the metadata cache system;

[0162] The second determining unit is configured to determine that cache metadata corresponding to the target object does not exist in the metadata cache system when request information in the operation request is inconsistent with cache information in the metadata cache system.

[0163] Optionally, also include:

[0164] A first judgment unit, configured to judge whether the request operation information in the operation request is consistent with the cache operation information in the metadata cache system when the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system;

[0165] The first access unit is used to access the target object in the object storage according to the operation request when the request attribute information in the operation request is inconsistent with the cache attribute information in the metadata cache system.

[0166] Optionally, also include:

[0167] An operation unit, configured to determine target metadata according to the operation request when request attribute information in the operation request is consistent with cache attribute information in the metadata cache system and request operation information in the operation request is consistent with cache operation information in the metadata cache system;

[0168] The second access unit is used to access the target object in the object storage according to the operation request when the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is inconsistent with the cache operation information in the metadata cache system.

[0169] Optionally, also include:

[0170] The upload module is used to upload the result of the operation of modifying the target object in the object storage to the metadata cache system.

[0171] It is worth noting that in the embodiment of the above-mentioned device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0172] The device provided by the embodiment of the present invention can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0173] Figure 7 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Figure 7 As shown, the electronic device includes a processor 810, a memory 820, an input device 830 and an output device 840; the number of the processor 810 in the electronic device can be one or more. Figure 7 A processor 810 is taken as an example; the processor 810, the memory 820, the input device 830 and the output device 840 in the electronic device can be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0174] The memory 820 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present invention. The processor 810 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 820, that is, implements the method provided in the embodiment of the present invention.

[0175] The memory 820 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 820 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 820 may further include a memory remotely arranged relative to the processor 810, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0176] The input device 830 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 840 may include a display device such as a display screen.

[0177] The embodiment of the present disclosure also provides a storage medium containing computer executable instructions, and the computer executable instructions are used to implement the method provided by the embodiment of the present disclosure when executed by a computer processor.

[0178] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, but can also execute related operations in the method provided by any embodiment of the present invention.

[0179] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0180] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for object storage, characterized in that: include: After receiving an operation request for a target object in the object storage, accessing a metadata cache system, wherein the metadata cache system includes cache metadata of the object; the cache metadata is description information corresponding to the object in the object storage; In a case where cache metadata corresponding to the target object exists in the metadata cache system, determining the target metadata according to the operation request; According to the target metadata, executing the operation indicated by the operation request; In the case where the cache metadata corresponding to the target object exists in the metadata cache system, determining the target metadata according to the operation request; and performing the operation indicated by the operation request according to the target metadata, including: In the case where cache metadata corresponding to the target object exists in the metadata cache system, modifying the cache metadata to obtain target metadata according to the operation request; and performing the operation indicated by the operation request in the object storage according to the target metadata; or, In the case where cache metadata corresponding to the target object exists in the metadata cache system, the target metadata is obtained from the metadata cache system according to the operation request; and the operation indicated by the operation request is performed in the metadata cache system according to the target metadata.

2. The method according to claim 1, characterized in that The method further comprises: In a case where the cache metadata corresponding to the target object does not exist in the metadata cache system, the target object in the object storage is modified according to the operation request.

3. The method according to claim 1, characterized in that The method further comprises: Determine whether cache metadata corresponding to the target object exists in the metadata cache system according to the request information in the operation request and the cache information in the metadata cache system.

4. The method according to claim 3, characterized in that: The determining, according to the request information in the operation request and the cache information in the metadata cache system, whether there is cache metadata corresponding to the target object in the metadata cache system comprises: When the request information in the operation request is consistent with the cache information in the metadata cache system, determining that cache metadata corresponding to the target object exists in the metadata cache system; When the request information in the operation request is inconsistent with the cache information in the metadata cache system, it is determined that the cache metadata corresponding to the target object does not exist in the metadata cache system.

5. The method according to claim 3 or 4, characterized in that: The request information includes request attribute information and request operation information, and the cache information includes cache attribute information and cache operation information; Determining whether there is cache metadata corresponding to the target object in the metadata cache system according to the request information in the operation request and the cache information in the metadata cache system includes: When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, determining whether the request operation information in the operation request is consistent with the cache operation information in the metadata cache system; When the request attribute information in the operation request is inconsistent with the cache attribute information in the metadata cache system, the target object in the object storage is modified according to the operation request.

6. The method according to claim 5, characterized in that When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, determining whether the request operation information in the operation request is consistent with the cache operation information in the metadata cache system includes: When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is consistent with the cache operation information in the metadata cache system, determining the target metadata according to the operation request; When the request attribute information in the operation request is consistent with the cache attribute information in the metadata cache system, and the request operation information in the operation request is inconsistent with the cache operation information in the metadata cache system, modify the target object in the object storage according to the operation request.

7. The method according to claim 1, characterized in that After executing the operation indicated by the operation request according to the target metadata, the method further includes: The result of performing the operation of modifying the target object in the object storage is uploaded to the metadata cache system.

8. An object storage system, characterized in that: include: Clients, big data components, metadata cache systems, and object storage; After receiving the operation request for the target object sent by the big data component, the client accesses the metadata cache system; After confirming that the cache metadata corresponding to the target object exists in the metadata cache system, the client determines the target metadata according to the operation request, and performs the operation indicated by the operation request according to the target metadata; The cache metadata is the description information corresponding to the object in the object storage; If the cache metadata corresponding to the target object exists in the metadata cache system, the client modifies the cache metadata to obtain the target metadata according to the operation request; According to the target metadata, the operation indicated by the operation request is performed in the object storage. or, When the metadata cache system has cache metadata corresponding to the target object, the client obtains the target metadata from the metadata cache system according to the operation request; The operation indicated by the operation request is performed in the metadata cache system according to the target metadata.

9. An object storage device, characterized in that: include: A metadata cache system access module, configured to access a metadata cache system after receiving an operation request for a target object in an object storage, wherein the metadata cache system includes cache metadata of an object; the cache metadata is description information corresponding to the object in the object storage; A modification module, configured to determine target metadata according to the operation request if cache metadata corresponding to the target object exists in the metadata cache system; An operation module, configured to execute the operation indicated by the operation request according to the target metadata; a modification module, specifically used to modify the cache metadata to obtain target metadata according to the operation request when the cache metadata corresponding to the target object exists in the metadata cache system; an operation module, specifically used to perform the operation indicated by the operation request in the object storage according to the target metadata; or, A modification module is specifically used to obtain target metadata from the metadata cache system according to the operation request when there is cache metadata corresponding to the target object in the metadata cache system; an operation module is specifically used to perform the operation indicated by the operation request in the metadata cache system according to the target metadata.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images