Object processing method and device, equipment and storage medium
Through the object processing methods of multi-level cache and data pool, the problem of insufficient cache space in the existing technology in large files and massive data scenarios is solved, efficient processing of objects of different data specifications is achieved, and the reliability of object storage is improved. performance.
Patent Information
- Application Number
- CN202411074501.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-12
AI Technical Summary
The existing object processing schemes are not able to be used in large-scale files and massive data scenarios due to cost limitations, resulting in insufficient cache space and limited application scenarios, so their reliability is not high.
The object processing method of multi-level cache and data pool is adopted. The initial responsible cache is determined according to the data specifications of the object to be uploaded, and the cache queue reaches the threshold is aggregated, and it is gradually raised to the previous level cache or data pool until the cache queue is not Reach the threshold and realize the processing of objects of different data specifications.
This method is suitable for various scenarios such as small files and large files, improving the reliability and processing efficiency of object storage, reducing the number of CPU access to memory, reducing database pressure, and improving program performance.
Smart Images

Figure CN118939199A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of cloud computing, and in particular to an object processing method, apparatus, device and storage medium. Background Art
[0002] With the popularization of cloud computing, the growing demand for data storage and management, and the deepening of digital transformation in various industries, unstructured data has shown explosive growth. Object storage is a storage solution that can provide large-scale scalability and cost-effectiveness.
[0003] Object storage is widely used in content distribution and data backup scenarios, and is an important tool in cloud computing environments. Improving the performance optimization of object storage has obvious advantages for processing large amounts of unstructured data. In related technologies, a cache solution is used to optimize file storage. However, this solution is only applicable to small file scenarios. In scenarios with large files and massive data, the cache cannot be used on a large scale due to cost constraints, resulting in insufficient cache space in scenarios with large files and massive data. In other words, the current object processing solution is only applicable to individual scenarios, so the reliability is not high. Summary of the invention
[0004] The present application provides an object processing method, apparatus, device and storage medium to solve the problem that current object processing solutions have limited applicable scenarios.
[0005] On the one hand, the present application provides an object processing method, which is applied to an object storage system, the system includes a multi-level cache and a data pool, a higher level cache corresponds to a larger data specification range, and the upper level of the highest level cache is the data pool; the method includes: responding to an upload request from a client, and obtaining the data specification of the object to be uploaded; the upload request is used to request storage of the object to be uploaded; the object to be uploaded is used as the current target object, and a first cache is determined as the current target cache, and the data specification of the object to be uploaded is within the data specification range corresponding to the first cache; the current target object is written into the current target cache, and it is detected whether the cache queue of the current target cache reaches a preset threshold; if the threshold is reached, the data to be aggregated is obtained from the current target cache for aggregation, the target object is updated to the aggregated object, the target cache is updated to the upper level cache, and the step of writing the current target object into the current target cache is returned; until the cache queue of the current target cache does not reach the threshold, the current target object is written into the current target cache, or, if the upper level of the current target cache is the data pool, the current target object is written into the data pool.
[0006] In some examples, the method further includes: updating index information of the current target object in the index pool, wherein the index information of the target object represents a storage location of the target object in the object storage system.
[0007] In some examples, before obtaining the data specifications of the object to be uploaded, it also includes: generating a feature code to be checked based on the data specifications and content of the object to be uploaded; querying whether there is first index information corresponding to the feature code to be checked in the index pool; obtaining the data specifications of the object to be uploaded, including: if the first index information does not exist, obtaining the data specifications of the object to be uploaded.
[0008] In some examples, before taking the object to be uploaded as the current target object, it also includes: determining the type of the object to be uploaded, where the object type includes large files and small files; if the object to be uploaded is a small file, the object to be uploaded is not sliced; if the object to be uploaded is a large file, the object to be uploaded is sliced, and each slice result is used as an updated object to be uploaded.
[0009] In some examples, data specifications and contents are obtained according to the object to be uploaded, and a feature code to be checked is generated, including: if the object to be uploaded is a small file, a feature code is generated according to the size and content of the object to be uploaded as the feature code to be checked; if the object to be uploaded is a large file, a feature code is generated according to the size and content of the slicing result as the feature code to be checked.
[0010] In some examples, the method also includes: responding to a read request from the client, querying whether there is second index information corresponding to the object to be read in the index pool; the read request is used to request to read the object to be read; if the second index information does not exist, reading the object to be read from the data pool, and returning the read object to be read to the client; if the second index information exists, determining the storage location of the object to be read according to the second index information; obtaining the object to be read from the storage location, and returning the object to be read to the client.
[0011] In some examples, obtaining data to be read from a storage location includes: if the storage location of the object to be read does not belong to a data pool, searching in a cache queue of a cache where the storage location of the object to be deleted is located; if the object to be read is retrieved, returning the object to be read to the client; if the object to be read is not retrieved, loading the object to be read from the cache where the storage location of the object to be read is located, and returning the loaded object to be read to the client; if the storage location of the object to be read belongs to a data pool, obtaining an aggregate object under the storage location of the object to be read from the data pool, obtaining index information of the aggregate object from an index pool for parsing, obtaining a data specification of the object to be read and an offset of the object to be read in the aggregate object, reading the object to be read from the aggregate object based on the data specification and the offset of the object to be read, and returning the read object to the client.
[0012] In some examples, after querying whether there is first index information corresponding to the feature code to be queried in the index pool, the method further includes: if the first index information exists, increasing the reference count of the object to be uploaded by one.
[0013] In some examples, the method also includes: responding to a deletion request from a client, querying whether there is third index information corresponding to the object to be deleted in the index pool; the deletion request is used to request deletion of the object to be deleted; if the third index information does not exist, obtaining a reference count of the object to be deleted from the index pool, and retrieving whether the reference count of the object to be deleted is one, if so, deleting the object to be deleted, otherwise reducing the reference count of the object to be deleted by one; if the third index information exists, determining the storage location of the object to be deleted according to the third index information, and deleting the data to be deleted according to the storage location of the object to be deleted.
[0014] In some examples, data to be deleted is deleted according to the storage location of the object to be deleted, including: if the storage location of the object to be deleted does not belong to the data pool, retrieving the reference count of the object to be deleted in the cache queue of the cache where the storage location of the object to be deleted is located; if the reference count of the object to be deleted retrieved is one, the object to be deleted is deleted; if the reference count of the object to be deleted retrieved is not one, the reference count of the object to be deleted is reduced by one; if the storage location of the object to be deleted belongs to the data pool, the fourth index information of the second aggregate object under the storage location of the object to be deleted is read from the data pool to obtain the reference count of the second aggregate object; if the reference count of the second aggregate object is not one, the reference count of the second aggregate object is reduced by one; if the reference count of the second aggregate object is one, the fourth index information of the second aggregate object is deleted, and the hole rate of the second aggregate object is calculated; if the hole rate of the second aggregate object is not less than a preset hole rate threshold, the second aggregate object is marked as an object to be recycled, so that the second aggregate object can be recycled through system space recycling processing.
[0015] In some examples, the system space recovery processing includes: detecting whether there is an object to be recovered; if there is an object to be recovered and the object to be recovered is not an aggregate object, determining whether the object to be recovered is expired, and deleting the object to be recovered if it is expired; if there is an object to be recovered and the object to be recovered is an aggregate object, creating a blank aggregate object, extracting small files according to the index information of the object to be recovered, and writing the extracted small files into the blank aggregate object to obtain a new aggregate object, and updating the index information of the new aggregate object, and using the new aggregate object to cover the object to be recovered.
[0016] In some examples, obtaining the data to be aggregated from the current target cache for aggregation includes: obtaining the longest unused data from each object in the current target cache as the data to be aggregated, and aggregating the data to be aggregated.
[0017] On the other hand, the present application provides an object processing device, which is applied to an object storage system. The system includes a multi-level cache and a data pool. The higher the level of the cache, the larger the data specification range corresponding to the cache. The upper level of the highest level cache is the data pool. The device includes: an acquisition module, which is used to respond to an upload request from a client and obtain the data specification of the object to be uploaded; the upload request is used to request the storage of the object to be uploaded; a processing module, which is used to take the object to be uploaded as the current target object, determine the first cache as the current target cache, and the data specification of the object to be uploaded is within the data specification range corresponding to the first cache; the processing module is also used to write the current target object into the current target cache, and detect whether the cache queue of the current target cache reaches a preset threshold; if the threshold is reached, the data to be aggregated is obtained from the current target cache for aggregation, the target object is updated to the aggregated object, the target cache is updated to the upper level cache, and the step of writing the current target object into the current target cache is returned; until the cache queue of the current target cache does not reach the threshold, the current target object is written into the current target cache, or, if the upper level of the current target cache is the data pool, the current target object is written into the data pool.
[0018] In some examples, the processing module is further used to: update index information of the current target object in the index pool, wherein the index information of the target object represents a storage location of the target object in the object storage system.
[0019] In some examples, the device also includes: a merging module, used to generate a feature code to be checked based on the data specifications and content of the object to be uploaded; query whether there is first index information corresponding to the feature code to be checked in the index pool; and an acquisition module, specifically used to: if the first index information does not exist, obtain the data specifications of the object to be uploaded.
[0020] In some examples, the merging module is also used to: determine the type of the object to be uploaded, where the object type includes large files and small files; if the object to be uploaded is a small file, the object to be uploaded is not sliced; if the object to be uploaded is a large file, the object to be uploaded is sliced, and each slice result is used as an updated object to be uploaded.
[0021] In some examples, the merging module is specifically used to: if the object to be uploaded is a small file, generate a feature code based on the size and content of the object to be uploaded as the feature code to be checked; if the object to be uploaded is a large file, generate a feature code based on the size and content of the slicing result as the feature code to be checked.
[0022] In some examples, the processing module is also used to: respond to a read request from the client and query whether there is second index information corresponding to the object to be read in the index pool; the read request is used to request to read the object to be read; if the second index information does not exist, read the object to be read from the data pool and return the read object to be read to the client; if the second index information exists, determine the storage location of the object to be read according to the second index information; obtain the data to be read from the storage location and return the object to be read to the client.
[0023] In some examples, the processing module is specifically used to: if the storage location of the object to be read does not belong to the data pool, then search in the cache queue of the cache where the storage location of the object to be deleted is located; if the object to be read is retrieved, then the object to be read is returned to the client; if the object to be read is not retrieved, then load the object to be read from the cache where the storage location of the object to be read is located, and return the loaded object to be read to the client; if the storage location of the object to be read belongs to the data pool, then obtain the aggregate object under the storage location of the object to be read from the data pool, obtain the index information of the aggregate object from the index pool for parsing, obtain the data specification of the object to be read and the offset of the object to be read in the aggregate object, read the object to be read from the aggregate object based on the data specification and offset of the object to be read, and return the read object to the client.
[0024] In some examples, the merge module is further used to: if the first index information exists, increase the reference count of the object to be uploaded by one.
[0025] In some examples, the processing module is also used to: respond to a deletion request from the client, and query whether there is third index information corresponding to the object to be deleted in the index pool; the deletion request is used to request deletion of the object to be deleted; if the third index information does not exist, obtain the reference count of the object to be deleted from the index pool, and retrieve whether the reference count of the object to be deleted is one, if so, delete the object to be deleted, otherwise reduce the reference count of the object to be deleted by one; if the third index information exists, determine the storage location of the object to be deleted according to the third index information, and delete the data to be deleted according to the storage location of the object to be deleted.
[0026] In some examples, the processing module is specifically used to: if the storage location of the object to be deleted does not belong to the data pool, then retrieve the reference count of the object to be deleted in the cache queue of the cache where the storage location of the object to be deleted is located; if the reference count of the object to be deleted is retrieved to be one, then delete the object to be deleted; if the reference count of the object to be deleted is not one, then reduce the reference count of the object to be deleted by one; if the storage location of the object to be deleted belongs to the data pool, then read the fourth index information of the second aggregate object under the storage location of the object to be deleted from the data pool to obtain the reference count of the second aggregate object; if the reference count of the second aggregate object is not one, then reduce the reference count of the second aggregate object by one; if the reference count of the second aggregate object is one, then delete the fourth index information of the second aggregate object and calculate the hole rate of the second aggregate object; if the hole rate of the second aggregate object is not less than a preset hole rate threshold, then mark the second aggregate object as an object to be recycled, so that the second aggregate object can be recycled by the system space recycling processing module.
[0027] In some examples, the system space recovery processing module is used to: detect whether there is an object to be recovered; if there is an object to be recovered and the object to be recovered is not an aggregate object, determine whether the object to be recovered is expired, and if expired, delete the object to be recovered; if there is an object to be recovered and the object to be recovered is an aggregate object, create a blank aggregate object, extract small files according to the index information of the object to be recovered, and write the extracted small files into the blank aggregate object to obtain a new aggregate object, update the index information of the new aggregate object, and use the new aggregate object to cover the object to be recovered.
[0028] In some examples, the processing module is specifically used to: obtain the longest unused data from each object in the current target cache as the data to be aggregated, and aggregate the data to be aggregated.
[0029] On the other hand, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the above method.
[0030] On the other hand, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above method.
[0031] The object processing method, device, equipment and storage medium provided by the present application first respond to the client's upload request, obtain the data specifications of the object to be uploaded, determine the initial cache responsible for storing the object to be uploaded according to the data specifications, use the cache as the current target cache, write the object to be uploaded as the current target object to the target cache, and detect whether the cache queue of the target cache has reached the preset threshold. If it has reached it, the object to be aggregated is obtained in the current cache queue for aggregation, and submitted to the previous level cache. The operation for the previous level cache is similar until the queue of the cache currently being written does not reach the threshold or reaches the data pool, and the processing of the object to be uploaded is completed. In this solution, multiple levels of cache are set up, which is suitable for storing objects of different sizes, and when any cache is full during the storage process, the objects in the cache at that level are aggregated into larger objects and transferred to the cache at the previous level, thereby realizing the processing of objects with different data specifications. It is suitable for a variety of scenarios such as small files and large files, and has high reliability. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0033] Figure 1 A schematic diagram of a process flow of an object processing method provided in an embodiment of the present application is exemplarily shown in FIG.
[0034] Figure 2 An architectural diagram of an object storage system is exemplarily shown in FIG.
[0035] Figure 3 A flowchart of an object processing method is exemplarily shown in FIG.
[0036] Figure 4 A flowchart of an object processing method is exemplarily shown in FIG.
[0037] Figure 5 A schematic diagram of a pre-processing process is shown in FIG.
[0038] Figure 6 A flowchart of an object processing method is exemplarily shown in FIG.
[0039] Figure 7 A system space recovery flow chart is exemplarily shown in FIG.
[0040] Figure 8 A flowchart of system space recovery is shown in FIG.
[0041] Fig. 9 A schematic structural diagram of an object processing device is exemplarily shown in FIG.
[0042] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0043] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0046] The terms "including" and "having" in this application are used to express an open-ended inclusion, and mean that in addition to the listed elements / components / etc., there may be additional elements / components / etc.; the terms "first" and "second" etc. are only used as marks or distinctions, and are not intended to limit the order or quantity of their objects. In addition, the different elements and areas in the drawings are only shown schematically, and are therefore not limited to the sizes or distances shown in the drawings. The technical solution is described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0047] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices. The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that can perform the functions associated with the element.
[0048] With the popularization of cloud computing, the growing demand for data storage and management, and the deepening of digital transformation in various industries, unstructured data has shown explosive growth. Object storage is a storage solution that can provide large-scale scalability and cost-effectiveness.
[0049] Object storage is widely used in content distribution and data backup scenarios, and is an important tool in cloud computing environments. Improving the performance of object storage has obvious advantages in processing large amounts of unstructured data. Object Storage Service (OSS) is a massive, secure, low-cost, and highly reliable cloud storage service that is suitable for storing any type of file. Its capacity and processing power can be elastically expanded, and it provides a variety of storage types to choose from, fully optimizing storage costs.
[0050] Object storage is a network storage architecture that stores data through a flat address space, eliminating the hierarchical directory structure in traditional file systems. Each object consists of three main parts: a globally unique ID (identifier), metadata, and the data itself. The globally unique ID ensures fast retrieval of the object, while the metadata contains descriptive information about the object, such as creation date, type, permissions, etc. The object storage system responds to requests through an application programming interface (API), such as HTTP / HTTPS, and supports operations such as uploading objects, retrieving objects, updating metadata, and deleting objects.
[0051] In the related art, a high-speed cache solution is used to optimize file storage. However, this solution is only applicable to small file scenarios. In the case of large files and massive data, the high-speed cache cannot be used on a large scale due to cost constraints, resulting in insufficient cache space in large files and massive data scenarios. Therefore, the current object processing solution is only applicable to individual scenarios and is not highly reliable.
[0052] The technical content provided by the present application is intended to solve the above-mentioned technical problems of related technologies. In the embodiment of the present application, first respond to the upload request of the client, obtain the data specifications of the object to be uploaded, determine the initial cache responsible for storing the object to be uploaded according to the data specifications, use the cache as the current target cache, write the object to be uploaded as the current target object to the target cache, and detect whether the cache queue of the target cache reaches the preset threshold. If it reaches, obtain the object to be aggregated in the current cache queue for aggregation, and submit it to the upper-level cache. The operation for the upper-level cache is similar until the queue of the cache currently written does not reach the threshold or reaches the data pool, and the processing of the object to be uploaded is completed. In this scheme, multiple levels of cache are set, which is suitable for storing objects of different sizes, and when any cache is full during the storage process, the objects in the cache at that level are aggregated into larger objects and transferred to the cache at the upper level, thereby realizing the processing of objects with different data specifications, which is suitable for a variety of scenarios such as small files and large files.
[0053] Some aspects of the embodiments of the present application relate to the above considerations. The following is an example introduction of the solution in combination with some embodiments.
[0054] Embodiment 1
[0055] Figure 1 , a flow chart of an object processing method is exemplarily shown in the figure. This embodiment relates to object storage processing and is applied to an object storage system. The system includes a multi-level cache and a data pool. A higher level cache corresponds to a larger range of data specifications. The upper level of the highest level cache is a data pool. Figure 1 As shown, the method includes:
[0056] S101: Respond to an upload request from a client and obtain data specifications of an object to be uploaded; the upload request is used to request storage of the object to be uploaded.
[0057] S102: The object to be uploaded is taken as the current target object, the first cache is determined as the current target cache, and the data specification of the object to be uploaded is within the data specification range corresponding to the first cache.
[0058] S103: Write the current target object into the current target cache, and detect whether the cache queue of the current target cache reaches a preset threshold; if the threshold is reached, obtain the data to be aggregated from the current target cache for aggregation, update the target object to the aggregated object, update the target cache to the upper-level cache, and return to execute the step of writing the current target object into the current target cache; until the cache queue of the current target cache does not reach the threshold, write the current target object into the current target cache, or, if the upper level of the current target cache is a data pool, write the current target object into the data pool.
[0059] The threshold data specification ranges of each level of cache are different. For example, taking the two-level cache as an example, the data specification ranges corresponding to the cache and the data pool can be: less than or equal to 4k, greater than 4k and less than 4M, and greater than or equal to 4M. In actual applications, the data specification ranges of each level of cache can be set according to requirements. In addition, the type of cache can also be set according to actual conditions.
[0060] Figure 2 An example of an architecture diagram of an object storage system is shown in Figure 2 , still taking the two-level cache as an example, you can set up a memory cache cluster to store objects less than or equal to 4k, set up a cache pool to store objects less than 4M, and set up a data pool to store objects greater than or equal to 4M. Among them, the cache pool can use a solid state disk (SSD) as the storage medium, and the data pool uses a mechanical hard disk (HDD) as the storage medium. The memory cache cluster is used to cache objects less than or equal to 4K. Compared with the solution of using SSD as the first-level cache, the performance is improved, and data security can be guaranteed through clustered deployment. The cache pool uses SSD as the storage medium to cache objects less than 4M, and can use multiple copies or EC policies (Erasure Coding Policy) to ensure data security. The data pool uses HDD as the storage medium, and can also use multiple copies or EC policies to ensure data security.
[0061] In this embodiment, large files and small files can be determined according to the data size of the object. As an example, an object whose data size exceeds a certain size threshold is called a large file, and an object whose data size does not exceed the size threshold is called a small file. For example, it can be set that a file whose data size is less than 4k is a small file, and a file whose data size is greater than 4M is a large file.
[0062] In practical applications, the executor of the caching method can be an object processing device, which can be implemented in many ways. For example, it can be implemented through a computer program, such as application software, etc.; or, it can be implemented as a medium that stores relevant computer programs, such as a USB flash drive, a cloud disk, a hard disk, etc.; or, it can be implemented through a physical device that integrates or installs relevant computer programs, such as a server, etc.
[0063] Specifically, in response to the client's upload request, after obtaining the data specifications of the file to be uploaded, the file to be uploaded can be transferred to the corresponding level of cache or data pool for processing according to the data specification range of each level of cache. Figure 3 A flowchart of an object processing method is shown in FIG. Figure 3, assuming that the data size of the object to be uploaded is greater than 4M, such as 5M, the object to be uploaded will be directly written to the data pool; assuming that the data size of the object to be uploaded is less than 4k, such as 3k, the object to be uploaded will be written to the memory cache cluster, and the cache queue of the current memory cache cluster will be checked to see if it reaches the preset threshold. If it reaches the threshold, the objects to be aggregated will be selected from the memory cache cluster for aggregation, and then transferred to the upper level cache, that is, the cache pool, for processing. Similarly, for the cache pool, if the cache queue exceeds the threshold, it will be aggregated again and the aggregated object will be transferred to the data pool for writing, and then the process will end. During the process, if it is checked that the cache queue of the cache does not exceed the threshold, aggregation is not required. By adopting a multi-level cache method, the number of times the CPU accesses the memory can be greatly reduced, the pressure on the database can be reduced, thereby improving program performance, and at the same time providing a flexible cache strategy.
[0064] In some examples, after the object storage is completed, the index information in the index pool can be updated. As an example, the method also includes: updating the index information of the current target object in the index pool. Among them, the index information of the target object represents the storage location of the target object in the object storage system. Specifically, the index information is used to represent the storage location of the object, and the method of maintaining the index information is not limited. As an example, the index information of the object can be updated in the way of metadata maintenance. In one example, the metadata of each object is maintained in the index pool, and the metadata includes index information. This example can accurately record the basic information and storage location of the object by updating the index information, which is convenient for data management.
[0065] In actual applications, when selecting objects to be aggregated from the cache, a selection strategy can be set according to actual conditions. As an example, obtaining the data to be aggregated from the current target cache for aggregation in S103 may specifically include:
[0066] From each object in the current target cache, the data that has not been used for the longest time is obtained as the data to be aggregated, and the data to be aggregated is aggregated.
[0067] Among them, the method of determining the data that has not been used for the longest time is not limited. In one example, the data in the cache that has not been used for the longest time can be determined based on the Least Recently Used (LRU) algorithm. Specifically, the aggregated object obtained by aggregating the data to be aggregated is transmitted to the upper-level cache, and the cache of the data to be aggregated is cleaned up in the cache of this level. The above method realizes the second classification of hot and cold data while further completing the aggregation, ensuring that the most frequently used data is in the cache, thereby improving data utilization efficiency. The LRU algorithm can be used to separate hot and cold data, convert a large number of random IO (Input / Output) into sequential IO, improve the cluster IOPS (Input / Output Operations Per Second) performance, and provide caching capabilities to improve response speed.
[0068] The object processing method, device, equipment and storage medium provided in this embodiment first respond to the upload request of the client, obtain the data specifications of the object to be uploaded, determine the initial cache responsible for storing the object to be uploaded according to the data specifications, use the cache as the current target cache, write the object to be uploaded as the current target object to the target cache, and detect whether the cache queue of the target cache has reached the preset threshold. If it has reached it, the objects to be aggregated are obtained in the current cache queue for aggregation and submitted to the previous level cache. The operation for the previous level cache is similar until the queue of the cache currently being written does not reach the threshold or reaches the data pool, and the processing of the object to be uploaded is completed. In this solution, multiple levels of cache are set up, which is suitable for storing objects of different sizes, and when any cache is full during the storage process, the objects in the cache at that level are aggregated into larger objects and transferred to the cache at the previous level, thereby realizing the processing of objects with different data specifications. It is suitable for a variety of scenarios such as small files and large files, and has high reliability.
[0069] In one example, Figure 4 A flowchart of an object processing method is exemplarily shown in FIG. This example involves preprocessing an object to be uploaded by a client before obtaining the data specifications of the file to be uploaded. Specifically, based on any example, before S101, the method may further include:
[0070] S201, generating a feature code to be checked according to the data specification and content of the object to be uploaded; and querying whether there is first index information corresponding to the feature code to be checked in the index pool.
[0071] S202, obtaining data specifications of the object to be uploaded, including: if the first index information does not exist, obtaining data specifications of the object to be uploaded.
[0072] The feature code refers to a unique code for identifying a file or data block. In this embodiment, a small file refers to a file with a data size less than 4k. In practical applications, a small file refers to a file smaller than the first-level cache threshold set by the user according to his or her needs.
[0073] In some examples, before generating the feature code, the size of the object to be uploaded is standardized to ensure the storage balance of the cache and the data pool and improve the processing efficiency. As an example, the method also includes: determining the type of the object to be uploaded, where the type of the object includes large files and small files; if the object to be uploaded is a small file, the object to be uploaded is not sliced; if the object to be uploaded is a large file, the object to be uploaded is sliced, and each slice result is used as an updated object to be uploaded. Specifically, the processing method of large files is not limited to slicing. This example uses slicing of large files to adapt to the scenario of optimizing large and small files at the same time. As an example, the feature code to be checked is generated in S201, specifically including: if the object to be uploaded is a small file, a feature code is generated according to the size and content of the object to be uploaded as the feature code to be checked; if the object to be uploaded is a large file, a feature code is generated according to the size and content of the slicing result as the feature code to be checked. The format of the feature code is not limited, and the user can choose to set the format of the feature code according to their own needs.
[0074] As an example, after querying whether there is index information of the feature code to be checked in the index pool in S201, it may also include: if the first index information exists, the reference count of the object to be uploaded is increased by one. Specifically, the reference count is used to monitor the number of times the object is referenced, and when there is a new reference to the object, the count is increased by one. By increasing the reference count of the object by one to indicate the increase in the frequency of the object to be uploaded, it is ensured that the object will not be released by mistake when needed, thereby improving the accuracy and efficiency of memory management.
[0075] Combined with the scenario example, Figure 5 A schematic diagram of a preprocessing process is shown in FIG. Figure 5, the feature code is generated according to the data specifications and content of the object to be uploaded. The user can set the format of the feature code according to his own settings. The user can choose the SHA256 algorithm, that is, generate the feature code in the "data specification, SHA256" format. For example, first determine the type of the object to be uploaded, where the object type includes large files and small files; if it is a small file, the feature code can be directly generated according to the data specifications and content of the small file in the "data specification, SHA256" format. If it is a large file, it needs to be sliced and processed, and the feature code is generated according to the data specifications and content of the slice in the "data specification, SHA256" format; then use the feature code to obtain index information from the index pool. If there is index information, it means that the same content object already exists in the system, then the reference count in the metadata of the object is increased by one, and the object is merged. If it does not exist, the small file or large file is directly returned. Through this file processing method, the current problem of only processing small file scenarios and the file merging and storage method are solved, which greatly improves the storage performance.
[0076] In this example, based on the data specifications and content of the uploaded object, a feature code to be checked is generated, and the index pool is queried to see whether there is index information corresponding to the feature code to be checked. If there is no index information, the data specifications of the object to be propagated are directly obtained. The feature code set in this solution can ensure that the file has not been tampered with during the storage process, and can merge and store small or large file slices according to the feature code, reducing the storage space occupied while improving the response speed.
[0077] Figure 6 The flowchart of an object processing method is exemplarily shown. This embodiment relates to a process of reading an object. Based on any example, the method further includes:
[0078] S301, responding to a read request from a client, querying whether second index information corresponding to an object to be read exists in an index pool; the read request is used to request to read the object to be read.
[0079] S302, if the second index information does not exist, read the object to be read from the data pool and return the read object to be read to the client; if the second index information exists, determine the storage location of the object to be read according to the second index information; obtain the object to be read from the storage location, and return the object to be read to the client.
[0080] Among them, the size of the object requested to be read is not limited in the embodiment, and can be less than or equal to 4k, or less than 4M, and greater than or equal to 4M. As an example, only the information of the requested object can be read to read the corresponding index information in the index pool. Specifically, if there is no index information of the object to be read in the index pool, the object to be read is directly searched from the data pool. If there is index information of the object to be read in the index pool, the location where the object to be read is stored is determined according to the index information, and the object to be read is obtained from the location to realize object reading.
[0081] In some examples, in S302, the data to be read is obtained from the storage location, specifically including: if the storage location of the object to be read does not belong to the data pool, then searching in the cache queue of the cache where the storage location of the object to be deleted is located, if the object to be read is retrieved, then the object to be read is returned to the client, if the object to be read is not retrieved, then the object to be read is loaded from the cache where the storage location of the object to be read is located, and the loaded read object is returned to the client; if the storage location of the object to be read belongs to the data pool, then the aggregate object under the storage location of the object to be read is obtained from the data pool, the index information of the aggregate object is obtained from the index pool for parsing, the data specification of the object to be read and the offset of the object to be read in the aggregate object are obtained, based on the data specification and offset of the object to be read, the object to be read is read from the aggregate object, and the read object is returned to the client. Specifically, according to the index information of the object to be read, if it is determined that it is not stored in the data pool, it is preferentially searched from the cache queue of the cache where it is located, thereby improving the efficiency of object reading.
[0082] The specific object storage cache queue also has the advantage of strong scalability, which can easily expand the flat architecture of object storage horizontally without being restricted by file storage or block storage; it can effectively shorten the data reading path and reduce the frequency of I / O operations, thereby improving the data reading response speed.
[0083] Combined with the scenario example: when requesting to read an object, first retrieve the index information of the requested object from the index pool. If the index information exists, determine the location of the object based on its location, such as the cache pool and memory cache cluster; first search in the queues of each cache, and if the object is found, return it to the client. If the index information is not found, load it from each cache and return it to the client; if it is a written aggregate object, read the aggregate object data from the data pool, read the metadata of the aggregate object from the index pool, parse the size of the requested object and the offset in the aggregate object from the metadata, and read the requested object from the aggregate object based on the above information. If there is no mapping information, read the object from the data pool and return it to the client.
[0084] In this example, in response to the client's read request, the search pool is queried to see whether there is index information corresponding to the read object. The read request is used to request to read the object to be read. If there is no index information, the object to be read is read from the data pool and the read object to be read is returned to the client. If there is index information, the storage location of the object to be read is determined according to the index information, the data to be read is obtained from the storage location, and the object to be read is returned to the client. In this solution, the object to be read is read through the index information, which improves the query speed, reduces IO operations, and improves data retrieval efficiency.
[0085] In one example, a scene involving object deletion is involved. Based on any example, the method further includes:
[0086] In response to the client's deletion request, query whether there is third index information corresponding to the object to be deleted in the index pool; the deletion request is used to request deletion of the object to be deleted;
[0087] If the third index information does not exist, the reference count of the object to be deleted is obtained from the referenced pool, and whether the reference count of the object to be deleted is one is retrieved. If so, the object to be deleted is deleted, otherwise the reference count of the object to be deleted is reduced by one;
[0088] If the third index information exists, the storage location of the object to be deleted is determined according to the third index information, and the data to be deleted is deleted according to the storage location of the object to be deleted.
[0089] As an example, deleting the data to be deleted according to the storage location of the object to be deleted specifically includes:
[0090] If the storage location of the object to be deleted does not belong to the data pool, the reference count of the object to be deleted in the cache queue of the cache where the storage location of the object to be deleted is located is retrieved; if the reference count of the object to be deleted is retrieved to be one, the object to be deleted is deleted; if the reference count of the object to be deleted is not one, the reference count of the object to be deleted is reduced by one;
[0091] If the storage location of the object to be deleted belongs to the data pool, the fourth index information of the second aggregate object under the storage location of the object to be deleted is read from the data pool to obtain a reference count of the second aggregate object; if the reference count of the second aggregate object is not one, the reference count of the second aggregate object is reduced by one; if the reference count of the second aggregate object is one, the fourth index information of the second aggregate object is deleted, and the hole rate of the second aggregate object is calculated; if the hole rate of the second aggregate object is not less than a preset hole rate threshold, the second aggregate object is marked as an object to be recycled, so that the second aggregate object is recycled through system space recycling processing.
[0092] Among them, when the reference count of an object is not one, when the object is to be deleted, the reference count is reduced by one; when the reference count is one, the object can be directly deleted. This is because when the reference count is 0, it means that the object is not referenced by any client, so it can be directly destroyed, that is, deleted from the storage location. When the reference count is not one, the deletion response is achieved by reducing the reference count by one. The effect of maintaining the reference count is to achieve real-time management of storage management, that is, as long as the reference count of an object is 0, it can be released.
[0093] In this example, combined with the scenario example, when requesting to delete an object, first retrieve the index information of the object from the index pool. If there is index information, the location of the object is determined based on it, such as the cache pool, memory cache cluster, and data pool. For the storage location of the memory cache cluster and cache pool, first query the reference count of the object from the cache queue to see if it is one. If it is one, delete the object directly. If it is not one, reduce the reference count by one. For objects stored in the data pool, read the index information of the aggregate object from the data pool to query whether the reference count of the aggregate object is one. If it is not one, reduce the reference count by one. If it is one, delete the index information of the requested object, and calculate the void rate of the aggregate object. If the void rate is greater than or equal to a certain value, such as 50%, the aggregate object is marked as a recyclable object and recycled through the space recycling system. Among them, the void rate of the aggregate object includes but is not limited to 50%, and users can set it according to their own needs.
[0094] In practical applications, the storage system space recovery method mainly involves the allocation and recovery strategies of memory management, including call-in strategies, placement strategies, replacement strategies, and allocation structures. In addition, the recovery of memory space also involves specific allocation algorithms, such as the neighbor adaptation algorithm (Next Fit), and different memory management methods, such as fixed partition allocation and dynamic partition allocation. In dynamic partition allocation, the memory space can be dynamically divided according to the needs of the process. This flexibility makes the memory utilization higher. At a more specific operational level, for example, in a Linux system, the steps to recover storage space may include executing specific commands to view, delete, and verify the deletion of the World Wide Name (WWN), as well as checking the system link status, to ensure the effective recovery of storage space.
[0095] Figure 7 An exemplary system space recovery flow chart is shown. The system space recovery process includes:
[0096] S401, detecting whether there is an object to be recycled.
[0097] S402: If there is an object to be recycled and the object to be recycled is not an aggregate object, determine whether the object to be recycled is expired, and if it is expired, delete the object to be recycled.
[0098] S403: If there is an object to be recycled and it is an aggregate object, create a blank aggregate object, extract small files according to the index information of the object to be recycled, write the extracted small files into the blank aggregate object, obtain a new aggregate object, update the index information of the new aggregate object, and use the new aggregate object to cover the object to be recycled.
[0099] Among them, in the embodiment, a gc module can be set to realize space recovery, but space recovery is not limited to this. Space recovery can also be realized through the Linux system. Creating a new blank aggregate object is to facilitate the storage of small files extracted later. This is because the aggregate object may be an object obtained through multi-level cache aggregation. During the recovery process, it is not necessary to completely recover the aggregate object. It may be necessary to recover the small files aggregated therein. Therefore, it is necessary to create a blank aggregate object for storage.
[0100] In this example, the gc module is not part of the standard library in Python. It is a tool for memory management that can be used to control the behavior of the garbage collector. In Python, the garbage collector automatically manages memory, but in some cases, you may need more fine-grained control over memory usage or garbage collection behavior. The gc module provides several functions and classes that can be used to view and control the behavior of the garbage collector. For example, the gc.set_threshold function can set the threshold for the garbage collector to be triggered. When the size of the garbage reaches these thresholds, the garbage collector will be triggered to perform garbage collection.
[0101] Combined with the example of the actual scenario, Figure 8 An exemplary flowchart of a system space recovery is shown in FIG. Figure 8 : When executing system space recovery, first continuously read index information from the gc linked list, determine whether the read object is an aggregate object based on the index information, and if it is an aggregate object, create a blank aggregate object, extract small files based on the index information of the object to be recovered, write the extracted small files sequentially into the newly created blank aggregate object, and update the index information of the new aggregate object, use the new aggregate object to cover the aggregate object to be recovered, and complete the space recovery of the aggregate object; if the read data is not an aggregate object, determine whether the object is expired according to the gc process, and delete the object if it is expired. Combined with the previous embodiment, this process realizes asynchronous recovery of disk space through marking and replication algorithms, improves response speed and solves the problem of disk holes.
[0102] In this example, it is detected whether there is an object to be recycled. If there is an object to be recycled and it is not an aggregate object, it is determined whether the object to be recycled is expired. If it is expired, the object to be recycled is deleted. If the object to be recycled is an aggregate object, a blank aggregate object is created, and small files are extracted according to the index information of the object to be recycled. The extracted small files are written into the created blank aggregate object to obtain a new aggregate object and update its index information. The new aggregate object is used to overwrite the object to be recycled. This process of this solution realizes asynchronous recovery of disk space through marking and copying algorithms, which improves the response speed and solves the problem of disk holes.
[0103] The object processing method provided in this embodiment first responds to the client's upload request, obtains the data specifications of the object to be uploaded, determines the initial cache responsible for storing the object to be uploaded according to the data specifications, uses the cache as the current target cache, writes the object to be uploaded as the current target object to the target cache, and detects whether the cache queue of the target cache has reached a preset threshold. If it has, the object to be aggregated is obtained in the current cache queue for aggregation and submitted to the previous level cache. The operation for the previous level cache is similar until the queue of the cache currently being written does not reach the threshold or reaches the data pool, and the processing of the object to be uploaded is completed. In this solution, multiple levels of cache are set up, which is suitable for storing objects of different sizes, and when any cache is full during the storage process, the objects in the cache at that level are aggregated into larger objects and transferred to the cache at the previous level, thereby realizing the processing of objects with different data specifications. It is suitable for a variety of scenarios such as small files and large files, and has high reliability.
[0104] Embodiment 2
[0105] Fig. 9 , which exemplarily shows a structural diagram of an object processing device, and this embodiment relates to object storage processing, and is applied to an object storage system. The system includes a multi-level cache and a data pool. A higher level of cache corresponds to a larger range of data specifications. The upper level of the highest level cache is a data pool; Fig. 9 As shown, the device comprises:
[0106] The acquisition module 11 is used to respond to the upload request of the client and obtain the data specification of the object to be uploaded; the upload request is used to request the storage of the object to be uploaded.
[0107] The processing module 12 is used to take the object to be uploaded as the current target object, determine the first cache as the current target cache, and the data specification of the object to be uploaded is within the data specification range corresponding to the first cache.
[0108] The processing module 12 is also used to write the current target object into the current target cache, and detect whether the cache queue of the current target cache reaches a preset threshold; if the threshold is reached, the data to be aggregated is obtained from the current target cache for aggregation, the target object is updated to the aggregated object, the target cache is updated to the upper-level cache, and the step of writing the current target object into the current target cache is returned to execute; until the cache queue of the current target cache does not reach the threshold, the current target object is written into the current target cache, or, if the upper level of the current target cache is a data pool, the current target object is written into the data pool.
[0109] In some examples, after the object storage is completed, the index information in the index pool can be updated. As an example, the processing module 12 is also used to: update the index information of the current target object in the index pool. Among them, the index information of the target object represents the storage location of the target object in the object storage system. Specifically, the index information is used to represent the storage location of the object, and the method of maintaining the index information is not limited. As an example, the index information of the object can be updated in the way of metadata maintenance. In one example, the metadata of each object is maintained in the index pool, and the metadata includes index information. This example can accurately record the basic information and storage location of the object by updating the index information, which is convenient for data management.
[0110] As an example, when obtaining the data to be aggregated from the current target cache for aggregation, the processing module is specifically used for:
[0111] From each object in the current target cache, the longest unused data is obtained as the data to be aggregated, and the data to be aggregated is aggregated. In one example, the longest unused data in the cache can be determined based on a least recently used (Least Recently Used, LRU) algorithm.
[0112] In one example, the device may also include: a merging module 13, used to generate a feature code to be checked based on the data specifications and content of the object to be uploaded; query whether there is first index information corresponding to the feature code to be checked in the index pool; accordingly, when obtaining the data specifications of the object to be uploaded, the acquisition module 11 is specifically used to: if the first index information does not exist, then obtain the data specifications of the object to be uploaded.
[0113] As an example, the merging module 13 is also used to: determine the type of the object to be uploaded, where the type of the object includes large files and small files; if the object to be uploaded is a small file, the object to be uploaded is not sliced; if the object to be uploaded is a large file, the object to be uploaded is sliced, and each slicing result is used as an updated object to be uploaded.
[0114] As an example, when generating the feature code to be checked, the merging module 13 is specifically used for: if the object to be uploaded is a small file, a feature code is generated according to the size and content of the object to be uploaded as the feature code to be checked; if the object to be uploaded is a large file, a feature code is generated according to the size and content of the slicing result as the feature code to be checked.
[0115] As an example, after querying whether there is index information of the feature code to be checked in the index pool, the merging module 13 is further used to: if the first index information exists, increase the reference count of the object to be uploaded by one.
[0116] Based on any one of the examples, the processing module 12 is further configured to:
[0117] In response to the read request of the client, query whether there is second index information corresponding to the object to be read in the index pool; the read request is used to request to read the object to be read.
[0118] If the second index information does not exist, the object to be read is read from the data pool and the read object to be read is returned to the client; if the second index information exists, the storage location of the object to be read is determined according to the second index information; the object to be read is obtained from the storage location and returned to the client.
[0119] In some examples, when obtaining the data to be read from the storage location, the processing module 12 is specifically used for: if the storage location of the object to be read does not belong to the data pool, then searching in the cache queue of the cache where the storage location of the object to be deleted is located; if the object to be read is retrieved, then the object to be read is returned to the client; if the object to be read is not retrieved, then the object to be read is loaded from the cache where the storage location of the object to be read is located, and the loaded read object is returned to the client; if the storage location of the object to be read belongs to the data pool, then obtaining the aggregate object under the storage location of the object to be read from the data pool, obtaining the index information of the aggregate object from the index pool for parsing, obtaining the data specification of the object to be read and the offset of the object to be read in the aggregate object, based on the data specification and offset of the object to be read, reading the object to be read from the aggregate object, and returning the read object to the client.
[0120] In one example, the processing module 12 is further configured to:
[0121] In response to the client's deletion request, query whether there is third index information corresponding to the object to be deleted in the index pool; the deletion request is used to request deletion of the object to be deleted;
[0122] If the third index information does not exist, the reference count of the object to be deleted is obtained from the referenced pool, and whether the reference count of the object to be deleted is one is retrieved. If so, the object to be deleted is deleted, otherwise the reference count of the object to be deleted is reduced by one;
[0123] If the third index information exists, the storage location of the object to be deleted is determined according to the third index information, and the data to be deleted is deleted according to the storage location of the object to be deleted.
[0124] As an example, when deleting the data to be deleted according to the storage location of the object to be deleted, the processing module 12 is specifically used to:
[0125] If the storage location of the object to be deleted does not belong to the data pool, the reference count of the object to be deleted in the cache queue of the cache where the storage location of the object to be deleted is located is retrieved; if the reference count of the object to be deleted is retrieved to be one, the object to be deleted is deleted; if the reference count of the object to be deleted is not one, the reference count of the object to be deleted is reduced by one;
[0126] If the storage location of the object to be deleted belongs to the data pool, the fourth index information of the second aggregate object under the storage location of the object to be deleted is read from the data pool to obtain a reference count of the second aggregate object; if the reference count of the second aggregate object is not one, the reference count of the second aggregate object is reduced by one; if the reference count of the second aggregate object is one, the fourth index information of the second aggregate object is deleted, and the hole rate of the second aggregate object is calculated; if the hole rate of the second aggregate object is not less than a preset hole rate threshold, the second aggregate object is marked as an object to be recycled, so that the second aggregate object is recycled by a system space recycling processing module.
[0127] In one example, the system space recovery processing module is used to: detect whether there is an object to be recovered; if there is an object to be recovered and the object to be recovered is not an aggregate object, determine whether the object to be recovered is expired, and delete the object to be recovered if it is expired; if there is an object to be recovered and the object to be recovered is an aggregate object, create a blank aggregate object, extract small files according to the index information of the object to be recovered, write the extracted small files into the blank aggregate object, obtain a new aggregate object, update the index information of the new aggregate object, and use the new aggregate object to cover the object to be recovered. As an example, the system space recovery processing module can be a gc module. Specifically, the relevant content of this embodiment can be found in the aforementioned embodiment one.
[0128] In the object processing device provided by the present embodiment, the acquisition module responds to the upload request of the client and obtains the data specification of the object to be uploaded. The processing module determines the cache initially responsible for storing the object to be uploaded according to the data specification, uses the cache as the current target cache, writes the object to be uploaded as the current target object into the target cache, and detects whether the cache queue of the target cache reaches the preset threshold. If it reaches the preset threshold, the object to be aggregated is obtained in the current cache queue for aggregation and submitted to the previous level cache. The operation for the previous level cache is similar until the queue of the cache currently being written does not reach the threshold or reaches the data pool, and the processing of the object to be uploaded is completed. In this scheme, multiple levels of cache are provided, which are suitable for storing objects of different sizes. When any cache is full during the storage process, the objects in the cache at that level are aggregated into larger objects and transferred to the cache at the previous level, thereby processing objects of different data specifications. It is suitable for a variety of scenarios such as small files and large files, and has high reliability.
[0129] Embodiment 3
[0130] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device comprising:
[0131] Processor (processor) 291, the electronic device also includes a memory (memory) 292; it can also include a communication interface (Communication Interface) 293 and a bus 294. Among them, the processor 291, the memory 292, the communication interface 293, can communicate with each other through the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can call the logic instructions in the memory 292 to execute the method of the above example. In addition, the logic instructions in the above-mentioned memory 292 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. The memory 292, as a computer-readable storage medium, can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present application. The processor 291 executes functional applications and data processing by running the software programs, instructions and modules stored in the memory 292, that is, implementing the method in the above method example.
[0132] The memory 292 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 292 may include a high-speed random access memory and may also include a non-volatile memory.
[0133] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method in any embodiment.
[0134] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in any embodiment when the computer program is executed by a processor.
[0135] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0136] In an exemplary embodiment, a computer-readable storage medium storing computer-executable instructions is also provided, and the computer-executable instructions are used to implement the above method when executed by a processor. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.
[0137] In an exemplary embodiment, a computer program product is also provided, comprising a computer program, and the computer program implements the above method when executed by a processor.
[0138] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0139] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0140] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0141] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.
[0142] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0144] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0145] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for processing an object, It is characterized in that Applied to an object storage system, the system includes a multi-level cache and a data pool, a higher level cache corresponds to a larger range of data specifications, and the upper level of the highest level cache is the data pool; the method includes: Responding to an upload request from a client to obtain data specifications of an object to be uploaded; the upload request is used to request storage of the object to be uploaded; The object to be uploaded is used as the current target object, and the first cache is determined as the current target cache, and the data specification of the object to be uploaded is within the data specification range corresponding to the first cache; The current target object is written into the current target cache, and it is detected whether the cache queue of the current target cache reaches a preset threshold; if the threshold is reached, the data to be aggregated is obtained from the current target cache for aggregation, the target object is updated to the aggregated object, the target cache is updated to the upper-level cache, and the step of writing the current target object into the current target cache is returned to execute; until the cache queue of the current target cache does not reach the threshold, the current target object is written into the current target cache, or, if the upper level of the current target cache is the data pool, the current target object is written into the data pool.
2. The method according to claim 1, It is characterized in that The method further comprises: The index information of the current target object in the index pool is updated, wherein the index information of the target object represents the storage location of the target object in the object storage system.
3. The method according to claim 2, It is characterized in that Before obtaining the data specification of the object to be uploaded, the method further includes: Generate a feature code to be checked according to the data specification and content of the object to be uploaded; query whether there is first index information corresponding to the feature code to be checked in the index pool; The obtaining of the data specification of the object to be uploaded includes: if the first index information does not exist, obtaining the data specification of the object to be uploaded.
4. The method according to claim 3, It is characterized in that Before taking the object to be uploaded as the current target object, the method further includes: Determine the type of the object to be uploaded, where the type of the object includes large files and small files; If the object to be uploaded is a small file, the object to be uploaded is not sliced; if the object to be uploaded is a large file, the object to be uploaded is sliced, and each slicing result is used as an updated object to be uploaded.
5. The method according to claim 4, It is characterized in that The step of obtaining data specifications and contents according to the object to be uploaded and generating a feature code to be checked includes: If the object to be uploaded is a small file, a feature code is generated according to the size and content of the object to be uploaded as the feature code to be checked; If the object to be uploaded is a large file, a feature code is generated according to the size and content of the slicing result as the feature code to be checked.
6. The method according to claim 2, It is characterized in that The method further comprises: In response to a read request from the client, query whether there is second index information corresponding to the object to be read in the index pool; the read request is used to request to read the object to be read; If the second index information does not exist, reading the object to be read from the data pool, and returning the read object to be read to the client; If the second index information exists, the storage location of the object to be read is determined according to the second index information; the object to be read is acquired from the storage location, and the object to be read is returned to the client.
7. The method according to claim 6, It is characterized in that The acquiring the to-be-read data from the storage location comprises: If the storage location of the object to be read does not belong to the data pool, searching in the cache queue of the cache where the storage location of the object to be deleted is located, if the object to be read is retrieved, returning the object to be read to the client, if the object to be read is not retrieved, loading the object to be read from the cache where the storage location of the object to be read is located, and returning the loaded object to be read to the client; If the storage location of the object to be read belongs to the data pool, the aggregate object under the storage location of the object to be read is obtained from the data pool, and the index information of the aggregate object is obtained from the index pool for parsing to obtain the data specification of the object to be read and the offset of the object to be read in the aggregate object; based on the data specification of the object to be read and the offset, the object to be read is read from the aggregate object, and the read object is returned to the client.
8. The method according to claim 3, It is characterized in that After querying whether there is first index information corresponding to the feature code to be checked in the index pool, the method further includes: If the first index information exists, the reference count of the object to be uploaded is increased by one.
9. The method according to claim 8, It is characterized in that The method further comprises: In response to a deletion request from the client, query whether there is third index information corresponding to the object to be deleted in the index pool; the deletion request is used to request deletion of the object to be deleted; If the third index information does not exist, obtaining the reference count of the object to be deleted from the index pool, and searching whether the reference count of the object to be deleted is one, if so, deleting the object to be deleted, otherwise reducing the reference count of the object to be deleted by one; If the third index information exists, the storage location of the object to be deleted is determined according to the third index information, and the data to be deleted is deleted according to the storage location of the object to be deleted.
10. The method according to claim 9, It is characterized in that The deleting the data to be deleted according to the storage location of the object to be deleted includes: If the storage location of the object to be deleted does not belong to the data pool, then retrieve the reference count of the object to be deleted in the cache queue of the cache where the storage location of the object to be deleted is located; if the reference count of the object to be deleted retrieved is one, then delete the object to be deleted; if the reference count of the object to be deleted retrieved is not one, then reduce the reference count of the object to be deleted by one; If the storage location of the object to be deleted belongs to the data pool, the fourth index information of the second aggregate object under the storage location of the object to be deleted is read from the data pool to obtain a reference count of the second aggregate object; if the reference count of the second aggregate object is not one, the reference count of the second aggregate object is reduced by one; if the reference count of the second aggregate object is one, the fourth index information of the second aggregate object is deleted, and the hole rate of the second aggregate object is calculated; if the hole rate of the second aggregate object is not less than a preset hole rate threshold, the second aggregate object is marked as an object to be recycled, so that the second aggregate object is recycled through system space recycling processing.
11. The method according to claim 10, It is characterized in that The system space recovery process includes: Check whether there is an object to be recycled; If there is an object to be recycled and the object to be recycled is not an aggregate object, determine whether the object to be recycled is expired, and if it is expired, delete the object to be recycled; If there is an object to be recycled and the object to be recycled is an aggregate object, a blank aggregate object is created, small files are extracted according to the index information of the object to be recycled, and the extracted small files are written into the blank aggregate object to obtain a new aggregate object, and the index information of the new aggregate object is updated, and the new aggregate object is used to cover the object to be recycled.
12. The method according to any one of claims 1 to 11, It is characterized in that The obtaining the to-be-aggregated data from the current target cache for aggregating includes: From each object in the current target cache, the data that has not been used for the longest time is obtained as the data to be aggregated, and the data to be aggregated is aggregated.
13. An object processing device, It is characterized in that Applied to an object storage system, the system includes a multi-level cache and a data pool, a higher level cache corresponds to a larger range of data specifications, the upper level of the highest level cache is the data pool, and the device includes: An acquisition module, used to respond to an upload request from a client and acquire data specifications of an object to be uploaded; the upload request is used to request storage of the object to be uploaded; A processing module, configured to take the object to be uploaded as a current target object, determine the first cache as the current target cache, and the data specification of the object to be uploaded is within a data specification range corresponding to the first cache; The processing module is further used to write the current target object into the current target cache, and detect whether the cache queue of the current target cache reaches a preset threshold; if the threshold is reached, the data to be aggregated is obtained from the current target cache for aggregation, the target object is updated to the aggregated object, the target cache is updated to the upper-level cache, and the step of writing the current target object into the current target cache is returned to execute; until the cache queue of the current target cache does not reach the threshold, the current target object is written into the current target cache, or, if the upper level of the current target cache is the data pool, the current target object is written into the data pool.
14. An electronic device, It is characterized in that include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 12.
15. A computer-readable storage medium, It is characterized in that, The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 12 when executed by a processor.
Citation Information
Cited By
Object storage method and device, storage medium and electronic equipment
CN119668522A
