Data object processing method and system

Abstract

The invention provides a data object processing method and system, and the method comprises the steps: determining a current type of a target data object in a preset storage medium, determining whether the current type is a cold data type or a hot data type, and if the current type is different from a previous type, the current type is the hot data type, and the previous type is the cold data type, determining that the target data object is the cold data type; and if yes, migrating the partial copy of the target data object stored in the first storage medium to a second storage medium. Wherein the first storage medium is a storage medium corresponding to the cold data type, the second storage medium is a storage medium corresponding to the hot data type, the preset storage medium comprises the first storage medium and the second storage medium, and the cost and the performance of the first storage medium are lower than those of the second storage medium. A balance of processing (e.g., storage) of data objects between high performance and low cost can be achieved.

Description

Technical Field

[0001] This specification relates to the technical field of data processing, and in particular, to a method and system for processing data objects. Background Art

[0002] The processing of data objects may include storage, reading, writing, etc. Taking the storage of data objects as an example, in practical applications, different storage media are usually used to implement it.

[0003] For example, storage media such as solid state drives (SSDs), Optane, etc. can be used to store data objects. Relatively speaking, this type of storage media has the characteristics of low latency and high input / output operations per second (IOPS). However, at the same time, it also has the characteristics of low storage density and high unit cost.

[0004] Another example is that storage media such as hard disk drives (HDDs) can also be used to store data objects. Relatively speaking, this type of storage media has the characteristic of high storage density. However, at the same time, it also has the characteristics of high latency for each input / output (IO) request and low IOPS that can be provided for concurrent requests.

[0005] Therefore, how to provide a method for processing data objects that balances high performance and low cost has become an urgent problem to be solved.

[0006] It should be noted that the content of the above related technologies is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the Invention

[0007] This specification provides a method and system for processing data objects to avoid at least one of the above technical problems.

[0008] In a first aspect, this specification provides a method for processing data objects, including:

[0009] Determine the current type of a target data object in a preset storage medium, where the current type is a cold data type or a hot data type; and

[0010] If the current type is different from the previous type, and the current type is the hot data type while the previous type is the cold data type, then migrate a partial copy of the target data object stored in a first storage medium to a second storage medium;

[0011] Among them, the first storage medium is a storage medium corresponding to the cold data type, the second storage medium is a storage medium corresponding to the hot data type, the preset storage medium includes the first storage medium and the second storage medium, and the cost and performance of the first storage medium are respectively lower than those of the second storage medium.

[0012] In a second aspect, this specification provides a data object processing system, including:

[0013] At least one memory storing at least one instruction set for processing data objects;

[0014] At least one processor communicatively connected to the at least one memory, wherein when the at least one processor runs, it reads the at least one instruction set and executes the method described in the first aspect according to the indication of the at least one instruction set.

[0015] In a third aspect, this specification provides a computer-readable non-transitory storage medium, wherein at least one instruction set is stored in the computer-readable non-transitory storage medium, and the at least one instruction set is executed by at least one processor to implement the method described in the first aspect.

[0016] As can be seen from the above technical solutions, the data object processing method and system provided in this specification compare the previous type of the target data object with the current type. When the two are inconsistent and specifically when it is converted from the cold data type to the hot data type, a part of the copies of the target data object is migrated from the previous low-cost first storage medium corresponding to the cold data type to the high-performance second storage medium corresponding to the hot data type. This can improve the flexibility and diversity of data object storage. In addition, since two storage media, namely the first storage medium and the second storage medium, are used to store data objects, the storage capacity can be increased. Moreover, since the performance of the second storage medium is relatively high, it has a relatively low latency; since the cost of the first storage medium is relatively low, it has a relatively low storage cost. Therefore, users can enjoy both low latency and large-capacity, low-cost storage.

[0017] Other functions of the data object processing method and system provided in this specification will be partially listed in the following description. The creative aspects of the data object processing method and system provided in this specification can be fully explained through practice or the use of the methods, devices, and combinations described in the detailed examples below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions in the embodiments of this specification, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 Schematic diagram of the application scenario of the method for processing data objects provided by the embodiments of this specification;

[0020] Figure 2 Schematic diagram of the structure of the data object processing system provided by an embodiment of this specification;

[0021] Figure 3 Schematic diagram of the flow of the method for processing data objects provided by the embodiments of this specification;

[0022] Figure 4 Schematic diagram of the structure of the data object processing system provided by another embodiment of this specification;

[0023] Figure 5 Schematic diagram of the principle of data migration provided by the embodiments of this specification;

[0024] Figure 6 Schematic diagram of the principle that the data object provided by the embodiments of this specification is a text block. Detailed implementation manners

[0025] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this specification. On the contrary, they are only examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0026] It should be understood that the terms "include" and "have" and any variations thereof in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device including a series of components does not necessarily have to be limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0027] The term "and / or" in the embodiments of this specification describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0028] In the embodiments of this specification, the term "a plurality of" means two or more, and other quantifiers are similar.

[0029] Terms such as "first", "second", "third", "target", etc. in this specification are used to distinguish similar or homogeneous objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be interchanged under appropriate circumstances, for example, it is possible to implement according to an order other than those given in the illustrations or descriptions of the embodiments of this specification.

[0030] The term "unit / module" used in this specification refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware or / and software code that can perform functions related to that element.

[0031] For the convenience of the reader's understanding of this specification, the application scenarios of this specification are now introduced.

[0032] The technical solutions provided in this specification are applicable to scenarios where data objects need to be processed (such as storage, reading, writing, etc.). For example, the technical solutions provided in this specification can be applied to scenarios with the need to process data objects such as distributed file systems, stream processing engines (Flink), etc.

[0033] Taking the scenario of the above-mentioned distributed file system and the storage of data objects as an example:

[0034] A distributed file system may include multiple storage nodes, and a storage node can store data objects of a corresponding capacity. Through the technical solutions provided in this specification, it is possible to implement storing data objects with corresponding storage requirements using storage nodes with different characteristics. Moreover, based on different actual access requirements, it is possible to migrate data objects among different storage nodes. Thus, a balance between high performance and low cost in data storage is achieved.

[0035] Figure 1 It is a schematic diagram of the application scenario of the method for processing data objects (hereinafter simply referred to as the processing method) in the embodiments of this specification. Among them, the processing method of this specification can be applied to, for example, Figure 1 Scenario 100 shown. As Figure 1 shown, Scenario 100 may include target user 101, client 102, server 103, and network 104.

[0036] Target user 101 may be a user who triggers the processing of data objects such as storage. For example, target user 101 may perform a target operation on client 102 to trigger the storage of data objects.

[0037] The client 102 can be an electronic device that provides interactive functions to the target user 101. For example, the client 102 can provide an interactive interface to the target user 101, and the target user 101 can perform interactive operations on the interactive page. In some embodiments, in response to detecting a target operation triggered by the target user 101, the client 102 executes the processing method described in this specification. At this time, the client 102 may store data or instructions for executing the processing method described in this specification and may execute or be used to execute the data or instructions. In some embodiments, the client 102 may include a hardware device with data information processing capabilities and necessary programs required to drive the hardware device to work, in order to execute the processing method described in this specification.

[0038] In some other embodiments, the client 102 can automatically execute the processing method described in this specification. Exemplarily, the client 102 can execute the processing method described in this specification based on a preset time interval, or in real-time, or at a corresponding time point, etc.

[0039] In some embodiments, the client 102 can include a mobile device, a tablet computer, a laptop computer, an in-vehicle device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device can include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device can include a smart TV, a desktop computer, etc., or any combination. In some embodiments, the smart mobile device can include a smart phone, a personal digital assistant, a gaming device, a navigation device, etc., or any combination thereof. In some embodiments, the in-vehicle device in a motor vehicle can include an in-vehicle computer, an in-vehicle TV, etc.

[0040] In some embodiments, one or more applications (Application, APP) can be installed on the client 102. The APP can provide the target user 101 with the ability and interface to interact with the outside world through the network 104. The APP includes but is not limited to: web browser type APP programs, search type APP programs, chat type APP programs, shopping type APP programs, video type APP programs, financial management type APP programs, instant messaging tools, email clients, social platform software, and so on. In some embodiments, a target APP can be installed on the client 102.

[0041] As Figure 1 shown, the client 102 can be communicatively connected to the server 103. Among them, the server 103 can be communicatively connected to one client 102 or multiple clients 102. In some embodiments, the client 102 can interact with the server 103 through the network 104 to receive or send messages, etc.

[0042] Server 103 can be a server that provides various services. For example, server 103 can be a cloud server or a local server. Server 103 can be communicatively connected to a client 102 and receive data sent by the client 102, or can be communicatively connected to multiple clients 102 and receive data sent by each of the clients 102 respectively.

[0043] In some embodiments, the processing method described in this specification can be executed on server 103. At this time, server 103 can store data or instructions for executing the processing method described in this specification, and can execute or be used to execute the data or instructions. Server 103 can include a hardware device with data information processing capabilities and necessary programs for driving the operation of the hardware device.

[0044] Network 104 is a medium for providing a communication connection between client 102 and server 103. Network 104 can facilitate the exchange of information or data. As Figure 1 shown, client 102 and server 103 can be respectively connected to network 104, and transmit information or data to each other through network 104.

[0045] In some embodiments, network 104 can be any type of wired or wireless network, or a combination thereof. For example, network 104 can include a cable network, a wired network, an optical fiber network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network TM, a short-range wireless network (ZigBee TM), a near field communication (NFC) network, or a similar network.

[0046] In some embodiments, network 104 can include one or more network access points. For example, network 104 can include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of client 102 and server 103 can be connected to network 104 to exchange data or information.

[0047] It is worth noting that Figure 1The numbers of client 102, server 103, and network 104 in [it] are merely illustrative. According to implementation requirements, there can be any number of clients 102, servers 103, and networks 104. Moreover, the processing method provided in this specification can be executed entirely on client 102, entirely on server 103, or partially on client 102 and partially on server 103.

[0048] That is to say, Figure 1 and the above description for Figure 1 is only used to exemplarily elaborate on the application scenarios to which the processing method in this specification may apply, and should not be construed as a limitation on the application scenarios.

[0049] Figure 2 FIG. shows a hardware structure diagram of a processing system 200 provided according to an embodiment of this specification. The processing system 200 can execute the processing method described in this specification. The processing method is introduced in other parts of this specification. When the processing method is executed on client 102, the processing system 200 can be client 102. When the processing method is executed on server 103, the processing system 200 can be server 103. When the processing method is partially executed on client 102 and partially on server 103, the processing system 200 can be a system including client 102 and server 103.

[0050] As Figure 2 shown, the processing system 200 can include at least one storage medium 203 and at least one processor 202. In some embodiments, the processing system 200 can further include a communication port 204 and an internal communication bus 201. The processing system 200 can further include I / O components 205.

[0051] The internal communication bus 201 can connect different system components. For example, the internal communication bus 201 can connect the storage medium 203, the processor 202, the communication port 204, and the I / O components 205.

[0052] The I / O components 205 support input / output between the processing system 200 and other components.

[0053] The communication port 204 is used for data communication between the processing system 200 and the outside world. For example, the communication port 204 can be used for data communication between the processing system 200 and the network 104. The communication port 204 can be a wired communication port or a wireless communication port.

[0054] The storage medium 203 may include a data storage device. The data storage device may be a non-transitory storage medium or a transitory storage medium. For example, the data storage device may include one or more of a magnetic disk 2031, a read-only storage medium (ROM) 2032, or a random access storage medium (RAM) 2033. The storage medium 203 further includes at least one instruction set stored in the data storage device. The instruction set includes computer program code, and the computer program code may include programs, routines, objects, components, data structures, processes, modules, etc. for executing the processing method provided in this specification.

[0055] At least one processor 202 may be communicatively connected to at least one storage medium 203. The at least one processor 202 is configured to execute the above at least one instruction set. When the processing system 200 runs, the at least one processor 202 reads the at least one instruction set and, according to the instructions of the at least one instruction set, executes the processing method provided in this specification. The processor 202 may execute all steps included in the processing method. The processor 202 may be in the form of one or more processors. In some embodiments, the processor 202 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physics processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field-programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of executing one or more functions, etc., or any combination thereof.

[0056] For illustrative purposes only, only one processor 202 is shown in the processing system 200 in the drawings. However, it should be noted that the processing system 200 in this specification may also include multiple processors. Therefore, the operations and / or method steps disclosed in this specification may be executed by one processor or jointly executed by multiple processors. For example, if it is described in this specification that the processor 202 of the processing system 200 executes step A and step B, it should be understood that step A and step B may also be jointly or separately executed by two different processors 202 (e.g., the first processor executes step A, the second processor executes step B, or the first and second processors jointly execute step A and B).

[0057] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of the processing method for data objects provided in the embodiments of this specification.

[0058] As Figure 3 shown, the method includes the following S301 and S302:

[0059] S301: Determine the current type of the target data object in the preset storage medium, where the current type is either the cold data type or the hot data type.

[0060] Among them, the preset storage medium includes a first storage medium and a second storage medium. The first storage medium is the storage medium corresponding to the cold data type, and the second storage medium is the storage medium corresponding to the hot data type. The cost and performance of the first storage medium are respectively lower than those of the second storage medium.

[0061] The preset storage medium stores data objects, and the data objects can be relatively coarser-grained files or relatively finer-grained file blocks, which are not limited in this embodiment. The target data object can be the full amount of data objects in the preset storage medium or a part of the data objects in the preset storage medium.

[0062] In this specification, the preset storage medium can include two types: one is a storage medium with relatively low cost and performance (i.e., the first storage medium); the other is a storage medium with relatively high cost and performance (i.e., the second storage medium). Moreover, the first storage medium is used to store the data objects corresponding to the cold data type (which can be called cold data objects); the second storage medium is used to store the data objects corresponding to the hot data type (which can be called hot data objects).

[0063] Combined with Figure 4 it can be seen that the processing system includes a storage medium layer. The storage medium layer includes the preset storage medium, and the preset storage medium includes a first storage medium and a second storage medium.

[0064] The first storage medium can include a storage medium with relatively low cost. As Figure 4 shown, the second storage medium can include storage media such as hard disk drives and tapes with relatively high IO request latency. The second storage medium can also be implemented using object storage.

[0065] The second storage medium can include a storage medium with relatively high performance. As Figure 4 shown, the first storage medium can include storage media such as solid-state drives, NVMe solid-state drives, and Optane with relatively low IO request latency.

[0066] An IO request can be understood as a request for the read / write operation of object data from the preset storage medium.

[0067] The preset storage medium can be a persistent storage medium. A persistent storage medium can be understood as a storage medium in which the stored data can still be saved after the system is shut down or restarted and can continue to be used when the system is started up next time.

[0068] In addition, since the data objects stored in the first storage medium are of the cold data type. Therefore, when the first storage medium is a persistent storage medium, the first storage medium can also be referred to as a cold data persistent storage medium.

[0069] Correspondingly, since the data objects stored in the second storage medium are of the hot data type. Therefore, when the second storage medium is a persistent storage medium, the second storage medium can also be referred to as a hot data persistent storage medium.

[0070] It should be noted that different data objects may correspond to different types, and the same data object may also correspond to different types at different times. The type of a data object is used to characterize the cold / hot degree of the data object. For example, based on the cold / hot degree of the data object, the data object can be divided into cold data type and hot data type.

[0071] The cold / hot degree can be understood as the activity degree, or the degree of user demand, etc. Relatively speaking, the hot data object corresponding to the hot data type can be understood as the data object that is relatively more likely to be used frequently. The cold data object corresponding to the cold data type can be understood as the data object that is relatively less likely to be used.

[0072] Exemplarily, the hot data object corresponding to the hot data type can be the data object that is frequently accessed or used. The cold data object corresponding to the cold data type can be the data object that is rarely accessed or used or archived for a long time. That is to say, compared with the cold data object, the hot data object has a relatively higher access or use frequency. And compared with the hot data object, the cold data object has a relatively lower access and use frequency.

[0073] It should be noted that this specification does not limit the determination of the access frequencies corresponding to the cold data type and the hot data type, and it can be determined based on requirements, historical records, experiments, etc. For example, the processing system can determine the data objects with relatively high access frequencies as the hot data type.

[0074] In some embodiments, as Figure 4 shown, the processing system may include an interaction layer. The interaction layer can provide user access services externally, which are Software Development Kit (SDK) or command tools.

[0075] Correspondingly, the user can interact with the processing system based on the SDK or command tool, so that the processing system can configure the discrimination strategy for a certain data object to be of the cold data type or the hot data type based on the interaction. For example, configure the access frequencies for distinguishing the cold data type and the hot data type.

[0076] In S301, the processing system may determine the types (i.e., the current types) of at least some of the data objects (i.e., the target data objects) stored in the first storage medium and the second storage medium at the current moment. For example, the processing system determines whether the target data object is of the cold data type or the hot data type at the current moment.

[0077] Combined with the above analysis, it can be seen that there are differences in performance and cost between the first storage medium and the second storage medium. In this embodiment, for data objects of different types, the processing system may use different storage media for storage.

[0078] For example, for the cold data type, the processing system may store it in a storage medium with a relatively low cost, such as the first storage medium. For the data object, the processing system may store it in a storage medium with relatively high performance, such as the second storage medium.

[0079] Relatively speaking, the processing system stores the cold data type in a storage medium with a relatively low cost and the hot data type in a storage medium with relatively high performance. It can improve the flexibility and diversity of data object storage. In addition, since the storage medium includes two different types of storage media (i.e., the first storage medium and the second storage medium), the relative capacity is relatively large; since the performance of the second storage medium is relatively high, it has a relatively low latency; the cost of the first storage medium is relatively low, so it has a relatively low storage cost. Therefore, users can enjoy both low latency and large-capacity, low-cost storage.

[0080] Combined with the above analysis, it can be seen that the target data object may be part of the data objects in the preset storage medium or all of the data objects. In some embodiments, the processing system determines the target data object may include the following steps 11 and 12:

[0081] Step 11: Scan the data objects in the preset storage medium based on a preset time interval to obtain the data objects accessed in the preset storage medium within the current time interval.

[0082] The preset time interval can be understood as the time window for scanning the data objects in the preset storage medium. The length of the preset time interval is not limited in this embodiment and can be determined by the processing system based on requirements, historical records, experiments, etc.

[0083] For example, combined with the above example and Figure 4 , the processing system may interact with the user based on the interaction layer to determine the preset time interval based on the interaction information with the user.

[0084] The current time interval can be understood as the time interval or time window to which the current moment belongs.

[0085] It is understandable that the data objects in the preset storage medium are accessed randomly. For example, some data objects may be accessed within a certain time period, while some other data objects may not be accessed within that time period.

[0086] In this step, the processing system can scan the data objects in the preset storage medium at corresponding time intervals to obtain the data objects that have been accessed within the corresponding time period.

[0087] Continuing to combine the above example and Figure 4 , the processing system may further include a service layer. The service layer includes a metadata service, and the metadata service may include a scanning service.

[0088] Correspondingly, the processing system can scan the data objects in the preset storage medium based on the scanning service to obtain the data objects that have been accessed within the corresponding time period.

[0089] In some embodiments, if a certain data object in the preset storage medium is accessed, the processing system can record the timestamp when the data object is accessed. When the processing system scans each data object in the preset storage medium based on the scanning service, it can obtain the timestamp of the most recent access corresponding to each data object.

[0090] The processing system can compare the timestamp of the most recent access corresponding to each data object with the corresponding time period. If the timestamp of the most recent access of a certain data object is within the corresponding time period, the processing system can determine that data object as a data object that has been accessed, so as to finally determine it as a target data object.

[0091] Step 12: Determine the data objects that have been accessed in the preset storage medium as target data objects.

[0092] Exemplarily, if the data objects that have been accessed are all data objects, such as all data objects in the preset storage medium have been accessed within the corresponding time period, the processing system can determine all data objects in the preset storage medium as target data objects.

[0093] On the contrary, if the data objects that have been accessed are partial data objects, such as within the corresponding time period, some data objects in the preset storage medium have been accessed and some data objects have not been accessed, the processing system can determine the accessed data objects as target data objects.

[0094] As can be seen from the above analysis of Steps 11 and 12, in this embodiment, the processing system determines the accessed data object as the target data object, so as to subsequently determine whether the target data object needs to convert the storage medium for data migration. This can improve the accuracy and reliability of conversion and migration. Especially when the target data object is a partial data object in the preset storage medium, it can avoid the resources for processing all data objects and improve efficiency.

[0095] As can be seen from the above analysis, for the same data object, it may correspond to different types at different time periods. And the current type may be a cold data type or a hot data type. In some embodiments, S301 may include the following Steps 21 and 22, that is, the processing system can determine whether the current type is a cold data type or a hot data type based on the following Steps 21 and 22.

[0096] Step 21: Obtain the access-related information of the target data object, where the access-related information includes at least two of the following: queries per second, metadata access capability, number of IO requests per second, bandwidth, throughput, time interval between peak access and low access, access frequency, access latency.

[0097] Exemplarily, the target data object is the data object accessed in the preset storage medium within the corresponding time period. The access-related information is the information related to accessing the target data object.

[0098] It should be noted that the content listed in the above access-related information is only used to exemplarily illustrate the possible content of the access-related information, rather than being understood as the determination of the access-related information.

[0099] And in some embodiments, the access-related information may include content in a single dimension, such as throughput. While in other embodiments, the access-related information may also include content in multiple dimensions, such as bandwidth and throughput, etc.

[0100] Step 22: Determine the current type of the target data object according to the access-related information.

[0101] After obtaining the access-related information including one or more contents, the processing system can determine whether the current type of the target data object is a cold data type or a hot data type based on the access-related information.

[0102] As can be seen from the above analysis of Steps 21 and 22, in this embodiment, the processing system can obtain the access-related information including content in multiple dimensions and determine the current type based on the access-related information including content in multiple dimensions. This can improve the accuracy and reliability of the determined current type.

[0103] In some embodiments, for the content of different dimensions included in the access-related information, the processing system may assign corresponding weights to the content of different dimensions, and determine the current type based on the content of different dimensions and the corresponding weights.

[0104] In other embodiments, a prediction model may be pre-trained. The prediction model may determine the type of a data object based on the access-related information of the data object.

[0105] For example, taking the processing system training the prediction model as an example:

[0106] The processing system may obtain training data, which includes sample data objects, sample access-related information corresponding to the sample data objects, and type labels of the sample data objects (such as labels indicating whether the sample data object is a hot data type or a cold data type).

[0107] The processing system takes the sample access-related information as the input of the basic network model, and the basic network model determines the prediction result of whether the sample data object is a cold data type or a hot data type based on the sample access-related information. The processing system iteratively optimizes the basic network with the goal of minimizing the difference between the prediction result and the type label until a prediction model is obtained where the number of iterative optimizations reaches a preset number threshold or the loss value between the prediction result and the type label is less than a preset loss threshold.

[0108] Similarly, in this embodiment, the preset number threshold and the preset loss threshold are not limited and may be determined by the processing system based on requirements, historical records, experiments, etc.

[0109] In addition, during the training process, the processing system may also assign different weights to the sample access-related information of different dimensions of content to train a prediction model based on different weights.

[0110] S302: If the current type is different from the previous type, and the current type is a hot data type while the previous type is a cold data type, then a partial copy of the target data object stored in the first storage medium is migrated to the second storage medium.

[0111] The previous type can be understood as the type of the target data object determined in the previous time period.

[0112] Exemplarily, in combination with the above analysis, the processing system may execute the technical solution provided in this specification based on a preset time interval. Taking two adjacent time periods, the first time period as the current time period and the second time period as the previous time period as an example:

[0113] The type of the target data object obtained in the current time period is the current type, and the type of the target data object obtained in the previous time period is the previous type.

[0114] The processing system can compare the current type with the previous type to determine their consistency. The determination result may be that they are inconsistent, for example, the current type is a hot data type while the previous type is a cold data type; or, the current type is a cold data type while the previous type is a hot data type.

[0115] For the case where the determination result is that they are inconsistent, the processing system can determine that the target data object needs to be migrated. And if the determination result is specifically that the current type is a hot data type while the previous type is a cold data type, then perform the above S302. That is, part of the copies of the target data object that was previously a cold data type are stored from the first storage medium with relatively low cost to the second storage medium with relatively high performance.

[0116] For example, if there are a total of 3 copies of the target data object in the first storage medium, the processing system can migrate 1 or 2 of the copies from the first storage medium to the second storage medium.

[0117] The processing system migrates the target data object of the hot data type with relatively more read accesses from the first storage medium with relatively low performance to the second storage medium with relatively high performance. When the target data object is of the hot data type, it can meet the relatively more likely read requests and reduce the latency of reading the target data object, improving the data read performance.

[0118] In addition, during this data migration process, the processing system can only migrate part of the copies of the target data object from the first storage medium to the second storage medium to reduce the resource occupancy of the second storage medium and lower the storage cost. Moreover, since part of the copies of the target data object are still stored in the first storage medium, it can avoid reading from the first storage medium in case of failure to read the target data object from the second storage medium. That is, it can improve the effectiveness and reliability of reading the target data object.

[0119] Continuing with the above example and Figure 4 , the metadata service also includes an EasyTiering service. The EasyTiering service can include a hot conversion service and a cold conversion service.

[0120] The hot conversion service can be understood as a service in which a data object is converted from a cold data type to a hot data type and migrated from the first storage medium to the second storage medium.

[0121] The cold conversion service can be understood as a service in which a data object is converted from a hot data type to a cold data type and migrated from the second storage medium to the first storage medium.

[0122] Among them, the processing system can migrate some copies from the first storage medium to the second storage medium based on cache technology.

[0123] Based on the above analysis, when the judgment result is inconsistent, there may still be a possibility: the current type is a cold data type while the previous type is a hot data type.

[0124] For this possibility, the processing system can migrate all copies of the target data object stored in the second storage medium to the first storage medium.

[0125] Similarly, if there are 3 copies of the target data object stored in the second storage medium, when the target data object is transformed from a data object of a cold data type that is relatively frequently requested to a data object of a cold data type that is relatively rarely requested, the processing system can migrate the 3 copies from the second storage medium to the first storage medium respectively.

[0126] It can reduce the resource occupancy of the high-cost second storage medium to save storage costs. It can also release more high-performance storage resources to improve resource utilization.

[0127] In some other embodiments, if the current type is a cold data type while the previous type is a hot data type, the processing system can rewrite the data in the way of erasure code (EC) to migrate the target data object from the second storage medium to the first storage medium. In the case where there are multiple copies of the target data object stored in the second storage medium, the corresponding storage costs can be further reduced.

[0128] For example, in an (8 + 3) erasure code configuration, the processing system can divide the target data object into 8 data segments plus 3 parity segments, a total of 11 segments. As long as any 8 segments exist, the complete target data object can be restored. The storage efficiency in this configuration is approximately (8 / 11 ≈ 0.73). That is to say, the storage space required for 3 copies of the target data object now only needs about 1.38 (3 / 0.73) times the space. This significantly reduces the storage costs.

[0129] That is to say, by using erasure code for data migration, the processing system can greatly reduce the required storage space without affecting data reliability, thereby effectively reducing storage costs.

[0130] Continue to combine the above example and Figure 5It can be seen that the processing system can migrate a target data object whose current type is a hot data type and whose previous type is a cold data type from the first storage medium to the second storage medium based on the heat transfer service. The processing system can also migrate a target data object whose current type is a cold data type and whose previous type is a hot data type from the second storage medium to the first storage medium based on the cold transfer service.

[0131] Combined with the above analysis, it can be seen that in some embodiments, if the current type is different from the previous type, and specifically in the case where the target data object is converted from a cold data type to a hot data type, the processing system can migrate some copies of the target data object in the first storage medium to the second storage medium; for the case where the target data object is converted from a hot data type to a cold data type, the processing system can migrate all copies of the target data object in the second storage medium to the first storage medium.

[0132] In other embodiments, if the current type is different from the previous type, the processing system can migrate at least some copies of the target data object stored in the previous storage medium to the current storage medium.

[0133] Wherein, if the current type is a hot data type and the previous type is a cold data type, the previous storage medium is the first storage medium and the current storage medium is the second storage medium. Correspondingly, the processing system can migrate at least some copies stored in the first storage medium to the second storage medium.

[0134] If the current type is a cold data type and the previous type is a hot data type, the previous storage medium is the second storage medium and the current storage medium is the first storage medium. Correspondingly, the processing system can migrate at least some copies stored in the second storage medium to the first storage medium.

[0135] That is to say, in this embodiment, for the case where the target data object is converted from a cold data type to a hot data type, or for the case where the target data object is converted from a hot data type to a cold data type, the processing system can transfer some copies or all copies.

[0136] It should be noted that the number of target data objects may be multiple. Correspondingly, if one target data object corresponds to one migration task, the number of migration tasks of the processing system may also be multiple.

[0137] In some embodiments, the processing system can create multiple working nodes, such as multiple processes, to execute each migration task in parallel through multiple processes.

[0138] Exemplarily, the tiered storage service may include a transfer task management service. The processing system may create a transfer task (such as a warm transfer task or a cold transfer task) based on the transfer task management service. The transfer task management service may assign the created transfer task to the corresponding worker node.

[0139] The tiered storage service may further include a worker node management service. The worker node management service is used to manage each worker node. For example, the processing system may control each worker node to execute its corresponding migration task based on the worker node management service.

[0140] In some embodiments, when the worker node executes the migration task, it may report the task status to the metadata service until the reported task status is completed, then the processing system may determine that the cold transfer or warm transfer of the corresponding data object has been completed.

[0141] In some embodiments, after migrating a partial copy of the target data object from the first storage medium to the second storage medium, the processing method provided in this specification may further include the following steps 31 to 33:

[0142] Step 31: Obtain an IO request for the target data object.

[0143] Exemplarily, a user may initiate an IO request for the target data object to perform operations such as reading / writing / modifying / deleting the target data object.

[0144] The obtaining of the IO request may be implemented by the following examples:

[0145] In one example, the processing system may be connected to a collection device and receive the IO request collected and sent by the collection device.

[0146] In another example, the processing system may provide a tool for loading the IO request, and the user may transmit the IO request to the processing system through the tool for loading the IO request.

[0147] Among them, the tool for loading the IO request may be an interface for connecting to an external device, such as an interface for connecting to other storage devices, and obtaining the IO request transmitted by the external device through this interface; the tool for loading the IO request may also be a display device. For example, the processing system may output an interface for loading the IO request function on the display device, and the user may import the IO request into the processing system through this interface.

[0148] Step 32: Obtain the target storage address of the target data object in the second storage medium.

[0149] Exemplarily, after migrating a target data object from a first storage medium to a second storage medium, the processing system may record the storage location of the target data object in the second storage medium (i.e., the target storage address).

[0150] Correspondingly, in the case of obtaining an IO request, the processing system may obtain the target storage address.

[0151] In some embodiments, the processing system may store the correspondence between the target data object and its target storage address in the form of a directory tree.

[0152] Combined with the above example and Figure 4 , the directory tree service is also included in the processing system metadata service layer. Based on the directory tree service, the processing system can query the target storage address of the target data object.

[0153] In some embodiments, an offset is carried in the IO request. The offset can be understood as which specific part (or a certain length) of the content in the target data object that the user needs to read.

[0154] Among them, the directory tree can be understood as a tree structure for managing each data object and the corresponding directory. It can correspond one directory to one data object, or correspond one directory to each data object. The correspondence between the data object and the storage address, as well as the correspondence between the offset and the specific storage address, are recorded in the directory tree.

[0155] Correspondingly, based on the directory tree service, the processing system can obtain the target storage address of the target data object, and can determine the address of the data object with the specific corresponding length from the target storage address according to the offset.

[0156] In some embodiments, the management of the data object in the above example in the directory tree service, such as the correspondence between the data object and the address, etc., can be called data object management. And combined with the above example, if the data object is a file, it can be called file management. If the data object is a file block, it can be called file block management.

[0157] Continuing to combine the above example, the directory tree service may also include inode management, lock management, user management, etc.

[0158] Among them, the inode management can be understood as a data structure for storing the metadata of the data object. For example, it may include the name and content of the data object. One data object can correspond to one inode. The lock management can be understood as controlling the behavior when multiple users or processes access the same data object simultaneously based on the lock mechanism to prevent data corruption or inconsistent states. The user management may include user identification and authentication, etc.

[0159] Step 33: Obtain and output the target data object from the target storage address.

[0160] Continuing with the above example, after the processing system determines the target storage address of the target data object, it can read the corresponding target data object from the target storage address and feedback the target data object to the user.

[0161] Based on the analysis of the above Steps 31 to 33, in this embodiment, the processing system can improve the accuracy and reliability of obtaining the target data object by obtaining the target storage address of the target data object in the second storage medium and obtaining the target data object from the second storage medium based on the target storage address. Moreover, the high performance of the second storage medium can be utilized to reduce latency and improve the user experience.

[0162] In some embodiments, the processing method provided in this specification may further include the following Steps 41 and 42:

[0163] Step 41: If there is no target data object in the target storage address, obtain the initial storage address of the target data object in the first storage medium.

[0164] Exemplarily, combining the above analysis, in the case where the target data object is converted from the cold data type to the hot data type, the processing system migrates some copies stored in the first storage medium to the second storage medium, and the processing system stores the address of some copies stored in the second storage medium (i.e., the target storage address).

[0165] Therefore, after the processing system obtains an IO request, it first obtains a copy from the target storage address. If the target data object is not found in the target storage address due to data loss or other reasons, the processing system then obtains the address of the copy stored in the first storage medium and not migrated to the second storage medium (i.e., the initial storage address).

[0166] Step 42: Obtain and output the target data object from the initial storage address.

[0167] Correspondingly, after obtaining the initial storage address, the processing system can obtain the copy stored in the first storage medium and not migrated to the second storage medium from the initial storage address to obtain and feedback the target data object to the user.

[0168] Combining the above analysis of Steps 41 and 42, in this embodiment, the processing system first obtains the target data object from the second storage medium, and if not obtained, then obtains the target data object from the first storage medium. It can achieve low-latency data reading as much as possible and also achieve the reliability of finally obtaining the target data object.

[0169] Based on the above analysis, it can be seen that a data object may include contents of different granularities, such as a data object may be a relatively coarse-grained file or a relatively fine-grained file block.

[0170] Exemplarily, in some embodiments, the processing method provided in this specification further includes the following steps 51 to 53:

[0171] Step 51: Obtain a write request for a file to be written.

[0172] Exemplarily, the file to be written is a file that the user wishes the processing system to write and store in a preset storage medium.

[0173] It can be understood that, in order to avoid cumbersome description, the same or similar features as those in the above example will not be described in detail in this embodiment. For example, the implementation principle of the processing system obtaining a write request can refer to the description of the processing system obtaining an IO request in the above example.

[0174] Step 52: If the write request is not interrupted during the process of writing the file to be written, the file to be written is stored in a preset storage medium, wherein the target data object includes the file to be written.

[0175] Step 53: If the write request is interrupted during the process of writing the file to be written, the target file block corresponding to the interruption in the file to be written is stored in a preset storage medium, wherein the target data object includes the target file block.

[0176] Exemplarily, when the processing system writes the file to be written to the preset storage medium based on the write request, there may be two situations: Situation 1: The request is not interrupted and the writing is completed in one time. Situation 2: The request is interrupted and the writing is completed multiple times.

[0177] For situation 1, the processing system may execute step 52. If the request is not interrupted, the processing system continues to obtain the corresponding data stream until the file to be written is written as a whole to the preset storage medium. That is, in the preset storage medium, the file to be written is stored as a whole at the corresponding location.

[0178] For situation 2, the processing system may execute step 53. If an interruption is requested, then when the interruption is requested, the processing system obtains and writes the data stream obtained before the interruption until the interruption. Therefore, the processing system may store the file block corresponding to the data stream in the corresponding position of the preset storage medium. After requesting recovery, the processing system obtains the data stream after the recovery request, and stores the corresponding file block in the corresponding position of the preset storage medium (different from the position at the time of interruption).

[0179] For example, combined with Figure 6As described above, during the process in which the processing system obtains a write request to write a file to be written to a preset storage medium, the write request is interrupted twice. The file block written from the start of writing to the first interruption is file block 1, and the storage address of file block 1 is storage address 1; the file block written from the first interruption to the second interruption is file block 2, and the storage address of file block 2 is storage address 2.

[0180] Combined with the above example, in the case where the target data object is a file block, the above data object management includes the mapping management between files and file blocks (such as which specific files include which file blocks), as well as the management of the target storage address and the initial storage address of the file block, so as to achieve seamless migration of the target data object.

[0181] Combined with the above analysis of steps 51 to 53, it can be seen that in this embodiment, the processing system can store the file to be written as a file or a file block according to whether the write request is interrupted, which can improve the flexibility and diversity of data storage.

[0182] In some embodiments, if the storage space occupied by a file or a file block reaches a preset space threshold, the processing system can divide the file or the file block to obtain more refined file blocks and store them.

[0183] Similarly, in this embodiment, the preset space threshold is not limited and can be determined by the processing system based on requirements, historical records, experiments, etc.

[0184] Taking the file block as an example, continuing to combine the above example and Figure 6 , the file block written from the second interruption to the completion of writing is file block 3. Since the storage space required by file block 3 reaches the preset space threshold. Therefore, the processing system splits file block 3 into file block 31 and file block 32, and stores file block 31 and file block 32 at different storage addresses respectively. For example, the storage address of file block 31 is storage address 31; the storage address of file block 32 is storage address 32. Among them, file block 3 can also be called a combined file block.

[0185] This embodiment does not limit the way in which the processing system splits a file into file blocks, or splits a relatively large file block into relatively small file blocks. For example, it can be split by an average split method, or a split method based on requirements can be adopted. It can improve the effectiveness and reliability of data storage, and can also improve data storage performance.

[0186] Combined with the above analysis, it can be seen that the above example mainly scans the target data objects accessed during the current time period from the preset storage medium to determine whether the target data objects need to be transferred to cold or hot, and specifically how to transfer to cold or hot.

[0187] In some other embodiments, for other data objects that have not been accessed during the current time period, the processing system may determine the continuous duration for which the other data objects have not been accessed.

[0188] For a data object whose continuous duration reaches a preset duration threshold and which is stored in the second storage medium, the processing system may migrate the data object from the second storage medium to the first storage medium. In this way, the hot-to-cold transfer of the data object is completed, thereby achieving the technical effects of releasing high-quality resources and reducing storage costs.

[0189] Similarly, in this embodiment, the preset duration threshold is not limited and can be determined by the processing system based on requirements, historical records, experiments, and other means.

[0190] It should be noted that the above examples are only used to exemplarily illustrate the possible implementation manners of the processing method in this specification, and should not be construed as a limitation on the implementation manners of the processing method in this specification. Exemplarily, on the basis of the above technical concept, some of the above technical features can be combined to obtain a new embodiment; new technical features can also be added on the basis of the above examples to obtain a new embodiment; some technical features can also be reduced on the basis of the above examples to obtain a new embodiment; some of the technical features in the above examples can be replaced with other technical features; the order of some of the technical features in the above examples can also be adjusted to obtain a new embodiment, and so on, which will not be listed one by one here.

[0191] According to the above technical concept, this specification also provides a computer-readable non-transitory storage medium, in which at least one instruction set is stored. When the at least one instruction set is executed by a processor, the steps of the processing method described in this specification are implemented.

[0192] In some possible embodiments, various aspects of this specification can also be implemented in the form of a program product, which includes program code. When the program product runs on the processing system 200, the program code is used to cause the processing system 200 to execute the steps of the processing method described in this specification. The program product for implementing the above method can use a portable compact disc read-only memory (CD-ROM) to include program code and can run on the processing system 200. However, the program product of this specification is not limited to this. In this specification, the readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system. The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium can include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, and this readable medium can send, propagate, or transmit a program used by or in combination with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. The program code for performing the operations of this specification can be written in any combination of one or more programming languages, and the programming languages include object-oriented programming languages - such as Java, C++, etc., and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the processing system 200, partially on the processing system 200, executed as an independent software package, partially on the processing system 200 and partially on a remote processing system, or entirely on the remote processing system 200.

[0193] It should be noted that in the technical solution of this specification, the collection, storage, use, processing, transmission, provision, and disclosure of relevant information of users (such as access-related information, etc.) comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0194] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than in the embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require a particular order or a sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0195] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art can understand that this specification is intended to encompass various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0196] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" mean that the specific features, structures, or characteristics described in connection with that embodiment may be included in at least one embodiment of this specification. Thus, it should be emphasized and understood that two or more references to "an embodiment" or "one embodiment" or "alternative embodiments" in various parts of this specification do not necessarily all refer to the same embodiment. Additionally, the specific features, structures, or characteristics may be appropriately combined in one or more embodiments of this specification.

[0197] It should be understood that in the foregoing description of the embodiments of this specification, for the purpose of helping to understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, drawing, or its description. However, this does not mean that the combination of these features is necessary. Those skilled in the art may well mark out some of the devices as separate embodiments for understanding when reading this specification. That is to say, the embodiments in this specification can also be understood as the integration of multiple sub - embodiments. And the content of each sub - embodiment is also valid when it has fewer features than all the features of a single foregoing disclosed embodiment.

[0198] Each patent, patent application, publication of patent application, and other materials cited herein, such as articles, books, specifications, publications, documents, literature, etc. (excluding any historical examination documents related thereto), are hereby incorporated by reference for all purposes relevant hereto, e.g., in the specification and claims of this application. However, in the event of any inconsistency or conflict between the descriptions, definitions, and / or terms of the above materials and those used in this application, the descriptions, definitions, and / or terms used in this application shall prevail.

[0199] Finally, it should be understood that the embodiments of the application disclosed herein are illustrative of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely illustrative and not restrictive. Those skilled in the art may adopt alternative configurations in accordance with the embodiments in this specification to implement the application in this specification. Therefore, the embodiments of this specification are not limited to the embodiments precisely described in the application.

Claims

1. A method for processing data objects, comprising: Determining the current type of a target data object in a preset storage medium, where the current type is a cold data type or a hot data type; And If the current type is different from the previous type and the current type is the hot data type while the previous type is the cold data type, then migrating a partial copy of the target data object stored in the first storage medium to the second storage medium; Wherein, the first storage medium is a storage medium corresponding to the cold data type, the second storage medium is a storage medium corresponding to the hot data type, the preset storage medium includes the first storage medium and the second storage medium, and the cost and performance of the first storage medium are respectively lower than those of the second storage medium.

2. The method according to claim 1, wherein The method further includes: Obtaining an IO request for the target data object; Obtaining the target storage address of the target data object in the second storage medium; and Obtaining and outputting the target data object from the target storage address.

3. The method according to claim 2, wherein The method further includes: If the target data object is not in the target storage address, then obtaining the initial storage address of the target data object in the first storage medium; and Obtaining and outputting the target data object from the initial storage address.

4. The method according to any one of claims 1 to 3, wherein, The migrating a partial copy of the target data object stored in the first storage medium to the second storage medium includes: Based on caching technology, migrating the partial copy from the first storage medium to the second storage medium.

5. The method according to any one of claims 1 to 3, wherein The method further includes: Obtaining a write request for a file to be written; and If the write request is not interrupted during the process of writing the file to be written, then storing the file to be written in the preset storage medium, where the target data object includes the file to be written.

6. The method according to claim 5, wherein, The method further includes: If the write request is interrupted during the process of writing the file to be written, then storing the target file block corresponding to the interruption in the file to be written in the preset storage medium, where the target data object includes the target file block.

7. The method according to any one of claims 1 to 3, wherein The method further includes: Scanning the data objects in the preset storage medium based on a preset time interval to obtain the data objects accessed in the preset storage medium within the current time interval; and Determining the data objects accessed in the preset storage medium as the target data objects.

8. The method according to any one of claims 1 to 3, wherein The determining the current type of the target data object in the preset storage medium includes: Obtaining the access-related information of the target data object, where the access-related information includes at least two of: queries per second rate, metadata access capability, number of IO requests per second, bandwidth, throughput, time interval between peak access and trough access, access frequency, access latency; and Determining the current type of the target data object according to the access-related information.

9. The method according to any one of claims 1 to 3, wherein, The method further includes: If the current type is different from the previous type, and the current type is the hot data type while the previous type is the cold data type, then all copies of the target data object stored in the second storage medium are migrated to the first storage medium.

10. A processing system for data objects, comprising: At least one memory storing at least one instruction set for processing data objects; At least one processor communicatively connected to the at least one memory, wherein when the at least one processor runs, it reads the at least one instruction set and executes the method according to any one of claims 1 to 9 according to the instructions of the at least one instruction set.

Images