Data processing method and device, computer equipment and storage medium
Through neural network models, predict data popularity and effective duration, and generate data migration plans, solving the problem of inaccurate data migration in the existing technology, improving storage space utilization and reducing costs.
Patent Information
- Application Number
- CN202411080241.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, the accuracy of data migration is poor, resulting in low storage space utilization and high storage system cost.
By obtaining the data element information and access request information of the target data object, using the neural network model to predict data popularity and effective time, generating a data migration plan, and data migration in the storage cluster based on the prediction results.
Improve the accuracy of data migration, improve storage space utilization, and reduce storage system costs.
Smart Images

Figure CN119088293A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment and storage medium. Background Art
[0002] The data storage system is one of the important systems in the Internet. It saves the data in the Internet through physical or logical storage media for subsequent use. The data storage system can be divided into block storage, file storage and object storage.
[0003] Storage space utilization is an important indicator to measure the capacity of data storage systems. It describes the ratio of the actual used space of storage devices to the total space. Improving storage space utilization is of great significance in improving storage efficiency and reducing storage system costs.
[0004] In the related art, the storage space utilization rate in the storage system is improved by using data tiering and migration methods. However, the accuracy of data migration in the related art is poor. Summary of the invention
[0005] The embodiments of the present disclosure provide a data processing method, apparatus, computer equipment and storage medium, which can improve the accuracy of data migration.
[0006] According to one aspect of the present disclosure, a data processing method is provided, the method comprising:
[0007] Acquire data meta information of a target data object and first access request information within a first time period;
[0008] Inputting the data meta information and the first access request information into a first neural network model to perform data heat prediction, and obtaining data heat information of the target data object in a second time period, where the second time period is a time period after the first time period and adjacent to the first time period;
[0009] Inputting the data metadata and the first access request information into a second neural network model to predict the data validity period, and obtaining the data validity period corresponding to the target data object;
[0010] Generate a data migration plan for the target data object in a target data storage cluster based on the data meta information, the data popularity information, and the data validity period, wherein the data storage cluster includes at least two data storage nodes with different data storage capabilities;
[0011] The target data object is migrated in the target data storage cluster according to the data migration plan.
[0012] According to one aspect of the present disclosure, there is provided a data processing device, comprising:
[0013] An acquisition unit, used to acquire data meta information of a target data object and first access request information within a first time period;
[0014] a first prediction unit, configured to input the data meta information and the first access request information into a first neural network model to perform data heat prediction, and obtain data heat information of the target data object in a second time period, where the second time period is a time period after the first time period and adjacent to the first time period;
[0015] A second prediction unit, configured to input the data metadata and the first access request information into a second neural network model to predict the data validity period, and obtain the data validity period corresponding to the target data object;
[0016] A generating unit, configured to generate a data migration plan for the target data object in a target data storage cluster based on the data metadata, the data popularity information, and the data validity period, wherein the data storage cluster includes at least two data storage nodes with different data storage capabilities;
[0017] A migration unit is used to migrate the target data object in the target data storage cluster according to the data migration plan.
[0018] Optionally, in some embodiments, the present application further provides a first model training device, wherein the first model training device comprises:
[0019] A first acquisition unit is configured to acquire, from the data access log of the target data storage cluster, first sample data metadata information of a plurality of first sample data objects, first sample access request information of each of the first sample data objects in a plurality of first sample time periods, and second sample access request information of each of the first sample data objects in a plurality of second sample time periods, wherein the second sample time period is a time period after the first sample time period and adjacent to the first sample time period;
[0020] A first calculation unit, configured to calculate the access resource consumption of each of the first sample data objects in the second sample time period according to the second sample access request information;
[0021] A second calculation unit, configured to calculate a data heat label of each of the first sample data objects in the second sample time period based on the access resource consumption and the first sample data metadata;
[0022] The first updating unit is used to update the parameters of the first neural network model using the first sample data metadata and the first sample access request information of each of the first sample data objects as model inputs and the corresponding data heat labels as model output labels.
[0023] Optionally, in some embodiments, the second computing unit includes:
[0024] A first extraction subunit, configured to extract the sample data size and the number of copies of the first sample data object from the first sample data metadata;
[0025] The first calculation subunit is used to calculate the ratio of the access resource consumption to the product of the sample data size and the number of copies, and obtain the data heat label of each of the first sample data objects in the second sample time period.
[0026] Optionally, in some embodiments, the first computing unit includes:
[0027] a second extraction subunit, configured to extract sample access type information and sample access data volume information from the second sample access request information, and to extract storage type information of the first sample data object from the first sample data metadata information;
[0028] The first acquisition sub-unit is used to obtain amplification ratio information for amplifying the second sample access request information, and calculate the access resource consumption of each first sample data object in the second sample time period based on the amplification ratio information, the sample access type information, the sample access data volume information and the storage type information.
[0029] Optionally, in some embodiments, the present application further provides a second model training device, wherein the second model training device comprises:
[0030] A second acquisition unit is used to acquire second sample data metadata information of multiple second sample data objects, third sample access request information of a preset number of times before a sample moment, and a sample data valid time length label of the second sample data object at the sample moment from the data access log of the target data storage cluster;
[0031] The second updating unit is used to update the parameters of the second neural network model using the second sample data metadata and the third sample access request information as model inputs and the corresponding sample data valid time label as model output label.
[0032] Optionally, in some embodiments, the second updating unit includes:
[0033] A third extraction subunit is used to extract the sample data attribute information and the creation time information of the second sample data object from the second sample data metadata;
[0034] A second calculation subunit, configured to calculate a first time difference based on the creation time information and the sample time;
[0035] a fourth extraction subunit, configured to extract access time information from each of the third sample access request information, and calculate a plurality of second time differences based on the access time information and the sample time;
[0036] An updating subunit is used to update the parameters of the second neural network model using the sample data attribute information, the first time difference and the multiple second time differences as model inputs and the corresponding sample data effective duration label as model output label.
[0037] Optionally, in some embodiments, the data processing device further includes:
[0038] A third acquisition unit, used to acquire the deployment time of the first neural network model and the second neural network model;
[0039] a fourth acquisition unit, configured to acquire first updated training sample data and second updated training sample data from a data access log of the target data storage cluster within a third time period adjacent to the current time when the time interval between the deployment time and the current time is greater than a preset time threshold;
[0040] An updating unit is used to update the model parameters of the first neural network model based on the first updated training sample data, and to update the model parameters of the second neural network model based on the second updated training sample data.
[0041] Optionally, in some embodiments, the generating unit comprises:
[0042] The second acquisition subunit is used to acquire storage medium information corresponding to each storage node in the target data storage cluster;
[0043] A generating subunit is used to generate a data migration plan for the target data object in a target data storage cluster based on the data metadata, the data heat information, the storage medium information and the data validity period.
[0044] Optionally, in some embodiments, the generating subunit comprises:
[0045] A construction module, configured to construct an optimization function for the total cost of data storage in the target data storage cluster based on the data meta information, the data heat information, and the storage medium information;
[0046] A first determination module, configured to determine, according to the optimization function, a target storage state of each data object in different storage nodes when the total data storage cost is the lowest;
[0047] The second determination module is used to determine a data migration plan for the target data object in the target data storage cluster according to the target storage state and the data validity period.
[0048] Optionally, in some embodiments, the second determining module includes:
[0049] A calculation submodule, configured to calculate a data heat threshold for data migration of the target data object based on an access request processing capability and a data storage capability corresponding to each storage node, and a unit capacity cost ratio between different storage nodes;
[0050] A determination submodule is used to determine a data migration plan for the target data object according to the data heat threshold, the data heat information, the data validity period, and the meta information of the target data object.
[0051] Optionally, in some embodiments, the migration unit comprises:
[0052] A third acquisition subunit is used to acquire in real time the utilization rate of data access request processing resources in each storage node in the target data storage cluster;
[0053] A determination subunit, configured to determine a migration time for performing data migration on the target data object according to the utilization rate;
[0054] The migration subunit is used to perform data migration on the target data object according to the data migration plan based on the migration time.
[0055] According to one aspect of the present disclosure, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the data processing method as described above when executing the computer program.
[0056] According to one aspect of the present disclosure, a storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method as described above is implemented.
[0057] According to one aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. The computer program is read and executed by a processor of a computer device, so that the computer device executes the data processing method as described above.
[0058] The data processing method provided by the embodiment of the present disclosure obtains data meta information of a target data object and first access request information within a first time period; inputs the data meta information and the first access request information into a first neural network model to perform data heat prediction, and obtains data heat information of the target data object within a second time period, wherein the second time period is a time period after the first time period and adjacent to the first time period; inputs the data meta information and the first access request information into a second neural network model to perform data effective duration prediction, and obtains data effective duration corresponding to the target data object; generates a data migration plan for the target data object in a target data storage cluster based on the data meta information, the data heat information, and the data effective duration, wherein the data storage cluster includes at least two data storage nodes with different data storage capabilities; and migrates data of the target data object in the target data storage cluster according to the data migration plan.
[0059] Therefore, the data processing method provided by the present invention predicts the data heat and data validity period of the data in each data storage node of the data storage cluster through a pre-trained neural network model, and then guides the migration of data in different storage nodes based on the predicted accurate data heat and data validity period. Compared with the related technology that only guides data migration based on data heat, the accuracy of data migration can be improved.
[0060] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.
[0062] Figure 1 A system architecture diagram of a data processing method according to an embodiment of the present disclosure;
[0063] Figure 2 A schematic diagram of a data migration system in the related art;
[0064] Figure 3 A flowchart of a data processing method provided by an embodiment of the present disclosure;
[0065] Figure 4A A schematic diagram of a first time period and a second time period of an embodiment of the present disclosure;
[0066] Figure 4B Another schematic diagram of the first time period and the second time period of the embodiment of the present disclosure
[0067] Figure 5 A schematic diagram of cooling down and heating up a target data object disclosed in the present invention;
[0068] Figure 6 It is another system architecture diagram of the data processing method of the embodiment of the present disclosure;
[0069] Figure 7 Another flowchart of the data processing method provided by the present disclosure;
[0070] Figure 8 A module diagram of a data processing device according to an embodiment of the present disclosure;
[0071] Fig. 9 A terminal structure diagram for implementing various methods according to an embodiment of the present disclosure;
[0072] Fig.10 A server structure diagram for implementing various methods according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0073] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.
[0074] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:
[0075] Data tiering: Data tiering is a data management and organization method. Data tiering groups logically related data into hierarchical structures based on data storage (such as storage cost) and data usage (such as usage frequency) requirements, so that different storage technologies and storage strategies can be used at different levels. This method can not only improve data management efficiency and data utilization, but also effectively control storage costs.
[0076] Data migration: involves the selection, preparation, extraction, conversion and permanent change of data storage location. Data migration refers to the process of transferring data from one storage system, database or storage environment to another storage system, database or storage environment. Data migration moves rarely used or unused files to secondary storage systems (such as tapes or optical disks) for archiving. These files are usually image documents or historical information that need to be easily accessed at any time in the future.
[0077] Object storage (Clustered Object Storage, COS): Object storage is a way of storing data that treats data as an object. Each object has a unique identifier that allows a server or end user to retrieve the object. The object consists of a data body and metadata. The data body is the actual content stored in the object, such as text, pictures, audio or video files, etc.; the metadata is the basic information of the data body, including but not limited to the object size, creation time, storage type, etc. Metadata helps to describe and manage objects. Object storage is an object storage technology based on cluster architecture. It adopts a distributed architecture and organizes multiple storage nodes into a cluster to jointly undertake the tasks of data storage and access. This method is helpful for automating and simplifying data storage in cloud computing environments.
[0078] Cloud Object Storage (COS) service: Cloud object storage service is a distributed storage service that has no directory hierarchy, no data format restrictions, can accommodate massive amounts of data, and supports access via Hypertext Transfer Protocol (HTTP) or Hypertext Transfer Protocol Secure (HTTPS). Users can store and view data at any time over the Internet.
[0079] Distributed storage: Distributed storage is a computer data storage architecture that stores data in multiple computers or servers to achieve redundant backup and load balancing of data. Each computer or server can be regarded as a storage node. Each storage node can store a part of the data and provide data read and write operations. Each storage node can store and access data independently. Compared with traditional centralized storage, distributed storage can provide higher reliability, scalability and performance.
[0080] Tiered storage: Tiered storage is also called hierarchical storage management (HSM). Tiered storage is to store data in media of different tiers and automatically or manually migrate data between different media. At the same time, tiered storage is also a specific application and implementation of information lifecycle management. In fact, placing storage media of the same cost and efficiency between different tiers for data migration and replication is not an effective data storage method in terms of practicality and cost. Therefore, different storage media are used between different tiers in order to meet both performance and capacity requirements at the same cost. The difference in this storage medium is mainly in access speed and capacity. Media with fast access speed usually have high storage unit cost (cost per unit storage capacity) and relatively low capacity. Correspondingly, media with slow access speed are usually used to meet capacity and cost requirements, that is, to obtain larger capacity at the same cost. Therefore, tiered storage is actually a storage technology that performs operations such as data migration between the media layer of the high-speed and small-capacity tier and the media layer of the low-speed and large-capacity tier.
[0081] like Figure 2 As shown, it is a schematic diagram of a data migration system in the related art. The data migration system 200 in the related art includes a hierarchical consistent hash ring 210, a detection module 220 and a scheduling module 240. The hierarchical consistent hash ring 210 divides data objects into three categories according to capacity: small data, medium data, and big data. Among them, big data is always cold data, small data is always hot data, and medium data switches between cold, hot, and warm data according to the access frequency. The hierarchical consistent hash ring 210 stores hot data in a solid-state hard disk storage cluster 230 with a higher storage level, and both cold data and warm data are stored in a traditional disk storage cluster 250 with a lower storage level.
[0082] The detection module 220 is used to detect the storage capacity utilization rate of the solid-state hard disk storage cluster 230. When the storage capacity utilization rate of the solid-state hard disk storage cluster 230 exceeds a preset threshold, the detection module 220 determines the cold data based on the time difference between the current time and the last access time of each data, and the scheduling module 240 migrates the cold data to the traditional disk storage cluster 250 until the storage capacity utilization rate of the solid-state hard disk storage cluster 230 is lower than the preset threshold, thereby realizing downgrade migration scheduling.
[0083] When reading a storage object, the data temperature is updated according to the difference between the current access time and the last access time. When the detection module detects that there is data in the traditional disk storage cluster 230 whose temperature flag is switched to the hot flag, the hot data is migrated to the solid-state drive storage cluster 230 through the scheduling module 240 to implement upgrade migration scheduling.
[0084] In the data migration system 200 in the related art, when performing data tiered storage, data heat is determined only based on the data size of the data object and the access time of the data object, and the definition of data heat is inaccurate. When performing data migration, the timing of data migration is determined based on the manually set storage cluster capacity threshold and storage capacity utilization threshold. When the access characteristics of the data objects in the storage cluster change, the manually set threshold cannot be adjusted adaptively, and the generation of the data migration plan is guided only based on the data heat, resulting in inaccurate data migration.
[0085] Inaccurate definition of data heat and inaccurate data migration in related technologies result in uneven distribution of data on hot and cold nodes in the data storage cluster, and part of the storage space in some storage nodes cannot be utilized, ultimately leading to low storage space utilization of the data storage cluster.
[0086] In order to solve the problem in the related art that inaccurate data migration plan formulation leads to insufficient rationality of data migration and thus low storage space utilization of the data storage system, the present application provides a data processing method to improve the accuracy of data migration.
[0087] System architecture and scenario description of the application of the embodiments of the present disclosure
[0088] Figure 1 It is a system architecture diagram applied by the data processing method according to the embodiment of the present disclosure, and includes: a terminal 110, a data migration control device 120, a server 130, and the like.
[0089] The terminal 110 includes a variety of device forms such as desktop computers, laptop computers, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, home theater terminals, dedicated terminals, intelligent voice interaction devices, smart home appliances or aircraft. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network and work together using a common display device to form a terminal 110. The terminal 110 can communicate with the Internet in a wired or wireless manner to exchange data. In the disclosed embodiment, the terminal 110 can access the server 130 through the data migration control device 120.
[0090] The server 130 refers to a computer system that can provide certain services to the terminal 110. In the application scenario of the present embodiment, multiple servers 130 refer to multiple storage nodes in a data storage cluster, and each storage node is a server 130. Compared with ordinary terminals 110, the server 130 has higher requirements in terms of stability, security, performance, etc. The server 130 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. In the embodiment of the present disclosure, the server 130 specifically provides a storage function, that is, the server 130 here can be a database node of a distributed database, and the present disclosure has a cluster of multiple servers 130 forming a distributed database.
[0091] The data migration control device 120 is a proxy server or a proxy terminal, which can control and manage the server 130. The data processing method in the embodiment of the present disclosure can be implemented in the data migration control device 120. The data migration control device 120 obtains the data meta information of the target data object and the first access request information in the first time period; then, the data migration control device 120 can input the data meta information and the first access request information into the first neural network model to predict the data heat, and obtain the data heat information of the target data object in the second time period; the data meta information and the first access request information can be input into the second neural network model to predict the data effective duration, and obtain the data effective duration corresponding to the target data object; further, the data migration control device 120 can generate a data migration plan for the target data object in the target data storage cluster based on the data meta information, the data heat information and the data effective duration; finally, the target data object is migrated in the target data storage cluster according to the data migration plan, that is, the data migration control device 120 can control the target data object to migrate data between the servers 130 in the data storage cluster.
[0092] The data processing method provided in the embodiments of the present disclosure is essentially a data migration method, and data migration may include database migration, storage system migration, application migration, and cloud migration. Taking storage system migration as an example, the data processing method provided in the present disclosure can be applied to various cloud object storage systems and tiered storage systems. For example, when the data processing method provided in the present disclosure is applied to a cloud object storage system, the data objects stored in the cloud object storage system may be data corresponding to a certain application, including but not limited to application running data, application usage object data, and application configuration data.
[0093] The data processing method provided by the present disclosure is applied to a cloud object storage system as an example. A plurality of data objects are stored in the cloud object storage system. The data objects include data bodies, such as documents, pictures, videos, etc., and also include data meta information, such as object size, storage type, etc. Taking a picture as an example, a picture named "example.jpg", the picture name is the name of the target data object, and the data meta information of this picture may include the object size, such as 5MB, the object creation time is "2023-07-08, 9:00:00", etc. When the data processing method provided by the embodiment of the present disclosure is applied to the data migration task in the cloud object storage system, the data meta information of the picture object and the access request information within five minutes before the current moment can be obtained, such as the accessed data size "5MB" and the access request time corresponding to multiple access requests within the past five minutes. Then, the data heat of the picture object can be predicted based on the data meta information of the picture object and the access request information within the past five minutes, and the data heat information of the picture object within five minutes after the current moment can be obtained; the data valid duration of the picture data can also be predicted based on the data meta information of the picture object and the access request information within the past five minutes. Furthermore, a data migration plan may be generated based on the data metadata, data popularity information, and data validity period of the image object, and data of the image object may be migrated between different storage nodes in the data storage cluster according to the data migration plan.
[0094] The above examples do not limit the scope of protection of this case.
[0095] General description of the disclosed embodiments
[0096] The present disclosure provides a data processing method, which aims to improve the accuracy of data migration and ultimately improve the storage space utilization of the data storage cluster. Figure 3 As shown, according to one embodiment of the present disclosure, the data processing method includes:
[0097] Step 310: Acquire data metadata information of a target data object and first access request information within a first time period.
[0098] In the disclosed embodiment, there may be differences in storage media between multiple storage nodes in the data storage cluster, such as differences in storage capacity of different storage nodes, or differences in the ability of different storage nodes to process access requests, which may affect the storage resource utilization of the storage node when data of different heats are stored in different storage nodes. Specifically, when storing data in a storage node, it is constrained by the following two indicators: the remaining storage capacity of the storage node and the input / output (I / O) resource utilization. The hot data in the storage node is accessed at a high frequency, generating a large number of data access requests, thereby occupying a large number of I / O resources (such as the number of input / output operations per second (IOPS), throughput, etc.), making the I / O resource utilization high. At this time, even if there is still remaining storage space in the storage node, the storage node will no longer continue to write data. It can also be understood that the amount of I / O requests exceeds the processing capacity of the storage node, and the storage node cannot process more data access requests and cannot continue to write data, so that part of the storage space in the storage node cannot be fully utilized, which ultimately leads to low storage space utilization. The data processing method provided by the embodiment of the present disclosure can be applied to migrate data of different popularity between different storage nodes in a data storage cluster to improve storage space utilization.
[0099] As disclosed above, the data processing method provided in the embodiments of the present disclosure can be applied to a cloud object storage system to migrate data stored in the cloud object storage system between different storage nodes to improve the storage space utilization of the cloud object storage system. The target data object can be a basic unit in the cloud object storage system, and any type of data can be stored in the target data object, such as a database, a data packet, an application, or a common file (document, video, picture, etc.). In some embodiments, the target data object can be all data objects stored in the data storage cluster, and each data object can include a data body and data metadata.
[0100] The data body is the content actually stored in the target data object, such as the running code of the application or a video file, document file, etc. The data meta information is the basic information of the data body. The data meta information of the target data object may include the object uniform resource locator (object uniform resource locator, objURL). The object uniform resource locator may include a bucket (Bucket). The bucket is the carrier of the target data object. The bucket name is a name defined by the user after obtaining the AppID assigned by the system. The user can store multiple objects in the bucket. The naming of the bucket consists of two parts: the bucket name (Bucket Name) and the application identifier (Application Identifier, AppID). The two are connected by a hyphen "-", for example "examplebucket-xxxxxx". AppID is a unique identifier assigned to each user account by the cloud service provider. It is also one of the account identifiers in the cloud object storage system. AppID is generally a string of numbers automatically generated by the system. The data metadata of the target data object may also include the object identifier (Object ID), object name (Object Name), object creation time (Object Create Time, create Time), object size (Object Size, objSize), and storage type (Storage Type) of the target data object. Among them, the object identifier is the unique identifier of the target data object in the storage bucket, which is used to locate and access the target data object; the object size is the size of the data body in the target data object, for example, if the data body in the target data object is a picture, and the size of the picture is 5MB, then the object size is 5MB; the object creation time is the time when the data body in the target data object is created. The storage type of the target data object may include standard storage (Standard), infrequent storage (Infrequent Access), and archive storage (Archive); the difference between different storage types lies in the different redundancy mechanisms, which may include replicas and erasure coding (EC), and the storage redundancy corresponding to different redundancy mechanisms is also different. For example, standard storage and low-frequency storage are used to store data with high access frequency and can provide high access speed. Therefore, standard storage and low-frequency storage use a copy redundancy mechanism. For archive storage, it is used to store data with low storage costs. Therefore, archive storage uses erasure codes as a redundancy mechanism.
[0101] Furthermore, the first access request information of the target data object within the first time period can also be obtained. The first time period is a period of time before the current moment when the data heat information is evaluated, and is a period of time adjacent to the current moment, so as to ensure the rationality and effectiveness of the obtained first access request information and improve the accuracy of the data heat information prediction. The length of the first time period can be five minutes or ten minutes, which is not limited here.
[0102] The first access request information is the detailed information of the access request initiated by the user to the data in the target data object through the application programming interface (REST API) at the terminal. REST API is an interface used by two computer systems to exchange information securely through the Internet. Specifically, the first access request information may include access time (Reference Time, RefTime), access type (Reference Type, RefType), and access size (Reference Size, RefSize). Among them, the access time may be the time when the user sends the access request to the storage node in the data storage cluster. The access type may include acquisition (GET), upload (PUT) and deletion (DELETE), etc. The acquisition request is used to obtain the data object from the storage node, the upload request is used to update or replace the data object in the storage node, and the deletion request is used to delete the data object in the storage node. Different access types reflect the different resource consumption required for input / output (I / O) operations, specifically the resource consumption required for read and write operations; the access size may be the size of the data corresponding to the access request.
[0103] Among them, the data meta information of the target data object can be obtained from the data meta information library of each storage node in the data storage cluster; the first access request information of the target object in the first time period can be obtained from the data access log of the data storage cluster. In the embodiment of the present disclosure, the data meta information of the target data object and the first access request information of the target data object in the first time period are obtained, and the data heat information and data validity period of the target data object in the second time period can be further predicted based on the data meta information and the first access request information, wherein the second time period is a period of time after the current moment, and the duration of the second time period can be the same as the duration of the first time period. The specific prediction process will be described below.
[0104] Step 320: Input the data metadata and the first access request information into the first neural network model to perform data heat prediction, and obtain data heat information of the target data object in the second time period.
[0105] The disclosed embodiment proposes a method for estimating the heat of data based on a neural network model, which can be referred to as a first neural network model or a data heat prediction model. The first neural network model can accurately predict the heat of data in a second time period after the current moment based on data metadata information and data access request information in a first time period before the current moment.
[0106] The data element information input by the first neural network model is the object uniform resource identifier (objURL) and the object size (objSize), and the first access request information input is the access time (RefTime) and the access size (RefSize). The input data and output results of the first neural network model will be further explained later.
[0107] Specifically, the first neural network model includes embedding and deep neural networks (DNN). Among them, Embedding refers to the process of converting different types of input data (such as text, images, sounds, etc.) into numerical vectors. Numerical vectors usually contain multiple dimensions, and each dimension represents a certain abstract feature or attribute of the input data. In the embodiment of the present disclosure, Embedding is used to convert the data metadata information of the input target data object and the first access request information within the first time period to obtain a numerical vector, so that the vector can be input into the deep neural network for prediction. A deep neural network is an artificial neural network model composed of multiple neuron layers, which can extract the abstract features of the input data layer by layer, and finally output the data heat information of the target data object.
[0108] Furthermore, the second time period is the data heat information after the current moment of data heat evaluation. The data heat information in the second time period is used to characterize the access frequency and access requirements of the target data object in the second time period. The access requirements may include the amount of data accessed and the speed of responding to access requests. Figure 4A The second time period t2 is a time period after the first time period t1 and adjacent to the first time period t1. The duration of the first time period t1 can be equal to the duration of the second time period t2. For example, the data metadata information of the target data object and the first access request information of the target data object within five minutes before the current moment for data heat evaluation can be used to predict the data heat information within five minutes after the current moment. Please refer to Figure 4B The second time period t4 is a time period after the first time period t3 and adjacent to the first time period t3, and the first time period t3 may overlap with the second time period t4.
[0109] Next, the training process of the first neural network model provided by the embodiment of the present disclosure will be described in detail. The training process of the first neural network model and the second neural network model can be trained in the model training server. Compared with the data migration control device, the model training server has better data computing capabilities and model training capabilities. The training process of the first neural network model includes but is not limited to the following steps:
[0110] Acquire, from the data access log of the target data storage cluster, first sample data metadata information of a plurality of first sample data objects, first sample access request information of each first sample data object in a plurality of first sample time periods, and second sample access request information of each first sample data object in a plurality of second sample time periods;
[0111] Calculate the access resource consumption of each first sample data object in the second sample time period according to the second sample access request information;
[0112] Calculate the data heat label of each first sample data object in the second sample time period based on the access resource consumption and the first sample data metadata;
[0113] The first sample data metadata and the first sample access request information of each first sample data object are used as model inputs, and the corresponding data heat labels are used as model output labels to update the parameters of the first neural network model.
[0114] Specifically, the target data storage cluster may include multiple storage nodes, and there are media differences between the multiple storage nodes, and target data objects with different data heats can be stored in layers. Among them, the data in the data object can be divided into hot data and cold data according to the predicted data heat information, and the hot data can be stored in the hot node, and the cold data can be stored in the cold node. Hot and cold storage nodes can be divided according to the storage performance of the storage node and the performance of processing access requests. For example, the hot and cold properties of the storage node are determined according to the performance of processing I / O operations (i.e., read and write operations). The node that can frequently and efficiently process I / O operations is a hot node, and the storage node that processes I / O operations slowly is a cold node.
[0115] The properties of storage nodes in different data storage clusters are different. Therefore, for a certain target data storage cluster, the data metadata information and the first access request information of the data object stored in each storage node can be obtained from the data access log of the target data storage cluster to train the first neural network model. As a result, the data heat information predicted by the first neural network model is closely related to the storage node properties in the target data storage cluster to improve the accuracy of data heat prediction. Furthermore, the accuracy of data migration can be improved by guiding the generation of data migration plans between different storage nodes in the data storage cluster through the predicted data heat information.
[0116] The data access log of the target data storage cluster is a text file that records the operating status of the storage nodes in the data storage cluster and the user access behavior. It includes detailed access operation information when the user accesses the data objects in the storage node, including the access request time, access request path, response time, and the number of accesses within a period of time.
[0117] After constructing the first neural network model as described in the above steps, training sample data for training the first neural network model can be further obtained. The training sample data of the first neural network model may include first sample data metadata of multiple first sample data objects, first sample access request information of each first sample data object in multiple first sample time periods, and second sample access request information of each first sample data object in multiple second sample time periods. The first sample time period is adjacent to the second sample time period, and the first sample time period and the second sample time period are equal in length. The first sample access request information in the first sample time period is used to extract access features of the first sample data object, and the second sample access request information in the second sample time period is used to calculate the I / O resource consumption of the first sample data object in the second sample time period.
[0118] In some embodiments, first sample access request information of multiple first sample data objects in a data storage cluster within multiple first sample time periods may be obtained, and second sample access request information within a second sample time period adjacent to each first sample time period may be correspondingly obtained.
[0119] Based on this, the above calculation of the access resource consumption of each first sample data object in the second sample time period according to the second sample access request information specifically includes:
[0120] Extracting sample access type information and sample access data volume information from the second sample access request information, and extracting storage type information of the first sample data object from the first sample data metadata information;
[0121] Obtain amplification ratio information for amplifying the second sample access request information, and calculate the access resource consumption of each first sample data object in the second sample time period according to the amplification ratio information, sample access type information, sample access data volume information and storage type information.
[0122] Specifically, the embodiment of the present disclosure calculates the access resource consumption by establishing an I / O resource consumption model. The access resource consumption represents the demand of the first sample data object for the underlying I / O resources during the second sample time period. The demand for the underlying I / O resources refers to the demand of the storage node for the input / output (I / O) resources of the data storage system when performing data read and write operations. I / O resources may include disk space, disk input / output operations per second (IOPS), and network bandwidth. The embodiment of the present disclosure defines the access resource consumption as DiskIOPS, and establishes the following I / O resource consumption model to calculate the access resource consumption:
[0123] DiskIOPS=F(RefType,RefSize,StorageType,AmplifyFactor)
[0124] Among them, the main variables of the I / O resource consumption model include RefType, RefSize, StorageType and AmplifyFactor. RefType represents the sample access type information, which may include GET, PUT and DELETE. RefSize represents the sample access data volume information, which is the size of the data volume for each access request. For example, a first sample data object may store a data body with a data size of 8MB. In a certain access request, only 4MB of data can be accessed. At this time, RefSize is 4MB. For another example, a first sample data object may store a data body with a data size of 2MB and a data body with a data size of 3MB. In a certain access request, the two data bodies in the first sample data object are accessed at the same time. At this time, RefSize is 5MB.
[0125] StorageType represents the storage type of the first sample data object, such as standard storage, etc. StorageType can be extracted from the first sample data metadata. AMplifyFactor represents the amplification ratio information for amplifying the second sample access request information. The amplification ratio information is the amplification ratio of the storage node in the data storage cluster to the read and write requests, such as the ratio of the amount of data actually written by the storage node to the actual amount of data requested by the user. F represents the functional relationship between the dependent variable DiskIOPS and the four main variables. The access resource consumption model provided in the embodiment of the present disclosure can associate the access characteristics of the data object with the I / O resource consumption of the storage node, and predict the data heat information on this basis, which can improve the evaluation accuracy of the data heat information.
[0126] In some embodiments, after the aforementioned step calculates the access resource consumption of each first sample data object in the second sample time period, the data heat label of each first sample data object in the second sample time period can be calculated based on the access resource consumption and the first sample data metadata, specifically including:
[0127] Extracting the sample data size and the number of copies of the first sample data object from the first sample data metadata;
[0128] The ratio of the access resource consumption to the product of the sample data size and the number of copies is calculated to obtain the data heat label of each first sample data object in the second sample time period.
[0129] In the embodiment of the present disclosure, the data heat label of each first sample data object in the second sample time period is defined as the access resource consumption per unit storage capacity in the second time period. Specifically, the sample data size (objSize) of the first sample data object can be extracted from the first sample data metadata, and the sample data size is the size of all sample data stored in the first sample data object. The specific definition formula of the data heat label is as follows:
[0130]
[0131] Among them, DiskIOPS(upload(t),download(t)) represents the access resource consumption of the first sample data object in the second sample time period, objSize represents the sample data size of the first sample data object, ReplicationFactor represents the number of copies of the first sample data object, and the number of copies represents the redundancy of the data object. For example, if the number of copies is 4, the redundancy of the data object is 4, and Hotness(obj,t) represents the data heat label of the first sample data object in the second sample time period.
[0132] Furthermore, by calculating the ratio of the access resource consumption of each first sample data object in the second sample time period to the product of the sample data size of each sample data object and the number of copies, the data heat label of each first sample data object in the second sample time period is obtained. The disclosed embodiment analyzes the relationship between data heat information and the underlying storage I / O resource consumption by establishing a mathematical model, so that the data migration plan generated according to the guidance of the data heat information can achieve the purpose of optimizing the I / O resources in the data storage cluster, that is, when the data migration plan is subsequently formulated according to the data heat information, the storage resource utilization rate can be improved.
[0133] In some embodiments, the first sample data metadata and the first sample access request information of each first sample data object are used as model input features, and the corresponding data heat label is used as the model output label to update the parameters of the first neural network model. Specifically, the input features and output labels of the first neural network model are shown in Table 1 below:
[0134] Table 1 Input features and output labels of the first neural network model
[0135]
[0136] Among them, the input features of the first neural network model include the first sample data metadata of the first sample data object and the first sample access request information of the first sample data object within the first time period. Specifically, the first sample data metadata of the first sample data object includes objURL and objSize. The attributes of objURL and objSize are both model input features (Feature). objURL includes AppID, bucket name (bucket name) and object name. The object name is used to find the object that the user wants to access in the bucket. The object name is actually the object key (Object Key) used to uniquely identify the name of an object, similar to the file name in the file system. AppID, bucket name, and object name are all string types. objSize is the sample data size of a first sample data object, and the data type is a floating point number.
[0137] Furthermore, the first sample access request information of the first sample data object in the first time period includes kRefTime and kRefSize, both of which are input features. Among them, kRefTime represents the time difference between each access time and the current time in the k accesses in the first sample time period, there are k time differences in total, and the data type of kRefTime is an array; kRefSize represents the amount of data accessed by the k accesses in the first sample time period, there are k access sizes in total, and the data type of kRefSize is also an array.
[0138] The output label of the first neural network model is the data heat label Hotness, and the data type is a floating point number, which represents the data heat of the first sample data object in the second sample time period. Finally, the first sample data metadata information and the first sample access request information of each first sample data object are used as model input, and the corresponding data heat label is used as the model output label to update the parameters of the first neural network model. According to the trained first neural network model, the embodiment of the present disclosure can perform data heat prediction based on the data metadata information of the target data object and the first access request information in the first time period, and obtain the data heat information of the target data object in the second time period.
[0139] The disclosed embodiment provides a method for estimating the heat of a data object using a data heat estimation model, by obtaining data metadata information and access request information of a data object in a time period before a certain moment in the data access log of a target data storage cluster, and obtaining access request resource consumption information in a time period after the moment to train the data heat estimation model, so that the trained data heat estimation model is more reasonable and accurate, thereby improving the accuracy of data heat estimation for data objects. Therefore, more accurate data heat information can more accurately guide data stratification and data migration, and improve storage space utilization.
[0140] In addition, in addition to guiding the formulation of a data migration plan based on data heat information, the embodiment of the present disclosure also guides the generation of a data migration plan based on the effective duration of data to further improve the accuracy of data migration. Next, the prediction method of the effective duration of data provided by the embodiment of the present disclosure will be described in detail.
[0141] Step 330: Input the data metadata and the first access request information into the second neural network model to predict the data validity period, and obtain the data validity period corresponding to the target data object.
[0142] The time period from data generation to data extinction is represented by the data life cycle. Different from the data life cycle, in the disclosed embodiment, the data validity period refers to how long it will take for the data to expire from the predicted moment. Data extinction means that the data is deleted from the storage node of the data storage cluster. The data validity period can characterize the use value of the data in the future. When formulating a data migration plan, the disclosed embodiment not only considers the data popularity information of the target data object, but also considers the data validity period. On the one hand, a more sophisticated data migration plan can be formulated, and on the other hand, unnecessary migration overhead caused by frequent migration of data with a short data validity period can be avoided.
[0143] The disclosed embodiment proposes a method for estimating the effective duration of data based on a neural network model, and the neural network model can be called a second neural network model or a data effective duration prediction model. The second neural network model can accurately predict the effective duration of data in a second time period after the current moment based on data metadata information and data access request information in a first time period before the current moment.
[0144] Among them, in the actual training and prediction process, the data meta information and the first access request information input by the second neural network model are not exactly the same as those of the first neural network model mentioned in the aforementioned steps. The input data meta information of the second neural network model is the object uniform resource identifier (objURL), and the input first access request information is the access time (RefTime) and the object creation time (createTime).
[0145] In some embodiments, the effective duration corresponding to the target object data is predicted by a second neural network model, wherein the training process of the second neural network model specifically includes the following steps:
[0146] Obtaining second sample data metadata information of multiple second sample data objects, third sample access request information of a preset number of times before the sample moment, and sample data valid duration labels of the second sample data objects at the sample moment from the data access log of the target data storage cluster;
[0147] The parameters of the second neural network model are updated by taking the second sample data metadata and the third sample access request information as model inputs and taking the corresponding sample data effective time label as the model output label.
[0148] Specifically, the second neural network model can use a gradient boosting model (GBM). The sample time is any time before the current prediction time, which can be randomly selected, and there can be multiple sample times. The multiple second sample data objects can be data that has been deleted by the data storage cluster when the sample data is acquired, that is, sample data objects whose data life cycle has expired when the sample data is acquired.
[0149] The disclosed embodiment obtains the second sample data object that has already died when the sample data is obtained from the data access log of the target data storage cluster. Further, the second sample data metadata of the second sample data object can be obtained, and the extinction time of each second sample data object can be obtained. Any sample time is selected before the extinction time of the second sample data object, and the third sample access request information of the second sample data object before the sample time is obtained, and the sample data effective duration label of the second sample data object at the sample time is calculated based on the extinction time of the second sample data object and the sample time, so as to obtain the sample data effective duration label of the second neural network model.
[0150] In some embodiments, by sampling and analyzing the access logs of the target data storage cluster, it can be learned that there are obvious differences in the data life cycles corresponding to different sample data objects. The life cycle of most sample data objects is less than 30 days, and there are also some sample data objects with a life cycle of more than 180 days. Therefore, in order to reduce the difference in the values of the sample data effective duration labels, the value after taking the logarithm of the sample data effective duration is used as the data effective duration output label of the second neural network model.
[0151] Furthermore, the third sample access information of each second sample data object for a preset number of times before each sample moment and the sample data effective duration label of each second sample data object at the sample moment are obtained. For each sample moment, the second sample data metadata of multiple second sample data objects and the third sample access request information of each second sample data object for a preset number of times before the sample moment are used as input features of the second neural network model, and the sample data effective duration label of each second sample data object at the sample moment is used as the output label of the model to update the model parameters of the second neural network model.
[0152] In some embodiments, the second sample data metadata and the third sample access request information are used as model inputs, and the corresponding sample data effective time length label is used as the model output label to update the parameters of the second neural network model, specifically including the following steps:
[0153] Extracting sample data attribute information and creation time information of the second sample data object from the second sample data metadata;
[0154] A first time difference is calculated based on the creation time information and the sample time;
[0155] Extracting access time information from each third sample access request information, and calculating a plurality of second time differences based on the access time information and the sample time;
[0156] The sample data attribute information, the first time difference and the multiple second time differences are used as model inputs, and the corresponding sample data effective time length labels are used as model output labels to update the parameters of the second neural network model.
[0157] Specifically, the sample data attribute information of the second sample data object includes the AppID corresponding to the second sample data object, the bucket name of the bucket where the second sample data object is located, and the object name of the second sample data object. The creation time information is the time when the second sample data object is created.
[0158] Furthermore, the first time difference is calculated based on the creation time information and the sample time. The access time information of multiple accesses is extracted from the third sample access request information of each second sample data object before the sample time, and the difference between the multiple access time information and the sample time is calculated to obtain the second time difference. The sample data attribute information, the first time difference and the multiple second time differences are used as the model input, and the corresponding sample data effective duration label is used as the model output label to update the parameters of the second neural network model. The input features and output labels of the second neural network model are shown in Table 2 below:
[0159] Table 2 Input features and output labels of the second neural network model
[0160]
[0161] Specifically, the second neural network model's input feature is the second sample data meta information of the second sample data object, and the third sample access request information of a preset number of times before the sample moment. Among them, the second sample data meta information of the second sample data object includes objURL and createTime. objURL is an input feature, the data type is a string, including the AppID of the second sample data object, the bucket name where the second sample data object is located, and the object name of the second sample data object. Here, it has the same meaning as the objURL of the first sample data object input to the first neural network model, and will not be repeated; createTime is an input feature, the data type is a floating point number, which represents the time difference between the creation time of the second sample data object and the sample moment, that is, the first time difference. The third sample access request information of a preset number of times before the sample moment includes kRefTime, the attribute is an input feature, the type is an array, and the array contains k second time differences. The second time difference is the time difference between the access time before the sample moment and the sample moment.
[0162] Furthermore, the output label of the second neural network model is timeToLive, which represents the effective duration label of the sample data, and the data type is a floating point number. As disclosed above, the effective duration label of the sample data can be the logarithm of the effective duration of the sample data obtained by taking the logarithm of the effective duration of the sample data, or the original value of the effective duration of the sample data can be used as the effective duration label of the sample data.
[0163] The second neural network model provided in the embodiment of the present disclosure can predict the data validity period of the target data object after the current moment based on the data metadata information of the target data object, the object creation time, and the access request information of the target data object before the data validity period prediction moment (the current moment), thereby being able to determine how long after the current moment the target data object will not be deleted by the storage node, thereby avoiding the storage node from frequently migrating data objects with short data validity periods and reducing migration costs.
[0164] The load status of the storage nodes in the data storage cluster is constantly changing. For example, the type and frequency of the access request operations initiated by the user end to the storage node, as well as the amount of data accessed, change, resulting in changes in the data objects stored in the storage node and the amount of data stored in the storage node. Correspondingly, the data metadata, access characteristics and data storage capacity of the data objects stored in the storage nodes will also change accordingly, ultimately resulting in changes in the data popularity of the data objects. Therefore, in order to ensure the effectiveness of the prediction results of the first neural network model and the second neural network model, it is necessary to collect new training sample data in real time to incrementally train and update the parameters of the first neural network model and the second neural network model, and adaptively adjust the model parameters according to changes in the load status.
[0165] Based on this, in some embodiments, the data processing method provided by the present disclosure further includes:
[0166] Obtaining deployment time of the first neural network model and the second neural network model;
[0167] When the time interval between the deployment time and the current time is greater than a preset time threshold, obtaining the first updated training sample data and the second updated training sample data from the data access log of the target data storage cluster in a third time period close to the current time;
[0168] The model parameters of the first neural network model are updated based on the first updated training sample data, and the model parameters of the second neural network model are updated based on the second updated training sample data.
[0169] Specifically, the deployment time indicates the time when the first neural network training model and the second neural network training model are deployed to the data migration control device 120. When the first neural network model and the second neural network model are trained, the training sample data in a period of time before the deployment work is started are used. The time of obtaining the training sample data is far from the current time. During the period from sampling the training sample to the completion of the model deployment, the data objects stored in the data storage cluster will change, and the storage status and load status of the storage nodes in the data storage cluster will also change. Therefore, if the model parameters obtained by training the training sample data in the previous period of time are still used for prediction at this time, the prediction results will be inaccurate.
[0170] The disclosed embodiment obtains the time interval between the deployment time and the current time, and compares the time interval with a preset time threshold, which can be customized and is not limited here. When the time interval between the deployment time and the current time is greater than the preset time threshold, the first updated training sample data and the second updated training sample data can be obtained from the data access log of the target data storage cluster in a third time period adjacent to the current time. The duration of the third time period can be customized and is not limited here.
[0171] Furthermore, the first updated training sample data is used to update the model parameters of the first neural network model, and the second updated training sample data is used to update the model parameters of the second neural network model. The first updated training sample data and the second updated training sample data can be obtained from the data access log of the target data storage cluster, and the sample data metadata and access request information of some sample data objects within the third time period can be obtained. The first updated training sample data may include a vector composed of features such as objURL, objSize, kRefTime, kRefSize, and a Hotness label, for example<objURL,objSize,kRefTime,kRefSize,Hotness> ; The second updated training sample data may include features such as objURL, createTime, kRefTime and a vector composed of the timeToLive label, for example<objURL,createTime,kRefTime,timeToLive> .
[0172] It is understandable that after each update, the neural network model can be redeployed to the data migration control device 120. The disclosed embodiment can obtain the deployment time of the first neural network model and the second neural network model in real time, and adaptively perform incremental training and parameter update on the two, so that the disclosed embodiment can adaptively update the model parameters according to the dynamic changes of the load.
[0173] In the aforementioned steps, the embodiment of the present disclosure provides a method for predicting data heat information and data validity period. In addition to guiding the generation of a data migration plan through data heat information and data validity period, the embodiment of the present disclosure also considers the differences in storage media between different storage nodes when solving the migration priority and the target storage node of the migration when performing data migration on the target data object, thereby further improving the accuracy of data migration. Based on this, step 340 in the data processing method provided by the embodiment of the present disclosure is introduced.
[0174] Step 340: Generate a data migration plan for the target data object in the target data storage cluster based on the data metadata information, the data popularity information, and the data validity period.
[0175] Specifically, the training sample data of the first neural network model and the second neural network model can be obtained from the target data storage cluster to train the first neural network training model and the second neural network model. Therefore, the data heat information predicted by the first neural network model and the data validity period predicted by the second neural network model are closely related to the storage node attributes in the target data storage cluster. Furthermore, by guiding the generation of a data migration plan between different storage nodes in the target data storage cluster through the predicted data heat information and data validity period, the accuracy of data migration can be improved.
[0176] In some embodiments, generating a data migration plan for a target data object in a target data storage cluster based on data metadata, data popularity information, and data validity period specifically includes the following steps:
[0177] Obtain storage medium information corresponding to each storage node in the target data storage cluster;
[0178] A data migration plan for the target data object in the target data storage cluster is generated based on data metadata information, data popularity information, storage medium information, and data validity period.
[0179] For example, the storage media used by different storage nodes may include solid state disks (SSDs) and hybrid hard disks (HDDs), etc., and may also include other storage media (such as virtual disks). Different storage media have different storage media information, such as different data storage capabilities, different data storage costs, and different data processing performance, so that the storage capabilities (i.e., storage capacity), the number of input and output operations, and the ability to process access requests (I / O capabilities) of the corresponding storage nodes in the data storage cluster will also be different. In addition, the data storage costs required per unit of media corresponding to different storage media are also different.
[0180] Therefore, when formulating a data migration plan, you also need to consider the storage media information corresponding to each storage node in the data storage cluster to ensure that after the data is migrated between different storage nodes, the target storage node can provide corresponding storage space for the migrated data and meet the corresponding data access requirements, as well as reduce the data storage cost required for data migration.
[0181] Based on this, the embodiment of the present disclosure generates a data migration plan for the target data object in the target data storage cluster based on data metadata information, data popularity information, storage medium information, and data validity period, specifically including:
[0182] Based on constructing an optimization function for the total cost of data storage in the target data storage cluster;
[0183] Determine the target storage state of each data object in different storage nodes when the total data storage cost is lowest according to the optimization function;
[0184] The data migration plan of the target data object in the target data storage cluster is determined according to the target storage status and the data validity period.
[0185] Specifically, the embodiment of the present disclosure solves the problem of solving the data migration plan as an optimization problem, and the optimization goal of the optimization problem is to minimize the total data storage cost required by the data storage cluster when performing data migration. The optimization goal corresponding to the optimization function of the total data storage cost is as follows:
[0186]
[0187] In general, the total cost of ownership (TCO) is used to evaluate and quantify the total cost that an organization or enterprise needs to bear when using cloud services, including the cost of using cloud services, network costs, storage (including data backup) costs, and data migration costs. In the embodiment of the present disclosure, the estimated TCO refers to the data storage cost required for all storage nodes in the data storage cluster when performing data migration, and the optimization goal is to minimize the data storage cost. Where n represents a total of n storage nodes in the target data storage cluster, and i represents the i-th storage node in the data storage cluster. #StorageMedia i Indicates the storage media requirement of the i-th storage node in the target data storage cluster. The storage media requirement can be the number of storage media required by each storage node or the storage capacity and storage performance that the storage media can provide. MediaCost i It refers to the unit storage medium cost corresponding to the storage medium of the i-th storage node in the data storage cluster.
[0188] The constraints of the data storage cost optimization function of the target data storage cluster provided by the embodiment of the present disclosure are as follows:
[0189] where#StorageMedia∝max(IONeed,StorageNeed)
[0190] Specifically, IONeed represents the I / O demand of the data storage cluster, which refers to the number of input and output operations that the storage nodes in the data storage cluster need to perform within a certain period of time. StorageNeed represents the data storage demand of the data storage cluster, that is, the data storage capacity that the storage nodes in the data storage cluster need to provide within a given time. max(IONeed, StorageNeed) represents the maximum value between IONeed and StorageNeed. The storage media demand #StorageMedia of the data storage cluster is proportional to max(IONeed, StorageNeed). #StorageMedia represents the total amount of storage media required by all storage nodes in the entire data storage cluster.
[0191] Furthermore, the specific calculation formula of IONeed is as follows:
[0192] IONeed=∑ObjectIO
[0193] The I / O demand of the data storage cluster is equal to the sum of the I / O demand of each data object in the data storage cluster. In the embodiment of the present disclosure, the I / O demand of the target data object (ie ObjectIO) may be the DiskIOPS of the data object calculated above.
[0194] The calculation formula for the data storage requirement StorageNeed of the data storage cluster is as follows:
[0195] StorageNeed=∑(ObjectSize*ReplicationFctor)
[0196] The data storage demand StorageNeed is equal to the sum of the product of the data object size and the data object redundancy of each data object in the data storage cluster. The embodiment of the present disclosure provides a method for solving the total data storage cost when different target data objects are hierarchically stored and migrated between different storage nodes in the target data storage cluster by combining optimization problems. The optimization goal of the optimization function is to minimize the total data storage cost when different data objects are hierarchically stored and migrated between different storage nodes in the target data storage cluster, taking into account the data heat information of different target data objects.
[0197] Specifically, when solving the total data storage cost, the present disclosure takes into account the data heat information of the target data object, which is manifested in taking into account the IONeed of the data storage cluster, that is, taking into account the number of input and output operations required by different storage nodes, reflecting the consideration of data heat information. When optimizing the total data storage cost, the embodiment of the present disclosure takes into account the data object size of the target data object, that is, takes into account the data metadata information.
[0198] Furthermore, the target storage state of each data object in different storage nodes is determined according to the optimization function when the total data storage cost is the lowest. The output result of the optimization function of the total data storage cost is the target storage state of each target data object in the target data storage cluster in different storage nodes when the total data storage cost is the lowest. The target storage state of each target data object refers to the best storage node where each target data object should be stored when the I / O requirements of the data object in the data storage cluster are met and the total data storage cost of all storage nodes is the lowest. Among them, the target storage state of each data object in different storage nodes can then be the storage node where the data object is currently located and the target storage node to be migrated to.
[0199] In some embodiments, after the above steps determine the target storage status of each data object in different storage nodes, a data migration plan for the target data object in the target data storage cluster can be determined according to the target storage status and the data validity period, specifically including:
[0200] The data heat threshold for data migration of the target data object is calculated based on the access request processing capacity and data storage capacity corresponding to each storage node and the unit capacity cost ratio between different storage nodes;
[0201] The data migration plan of the target data object is determined according to the data heat threshold, data heat information, data validity period and the metadata of the target data object.
[0202] The disclosed embodiment determines the data heat threshold for data migration of data objects in each node according to the storage capacity and the ability to process access requests of each storage node in the data storage cluster. Specifically, the data heat threshold for data migration of data objects is calculated by the following formula:
[0203] Threshold = k*(IOPS / Capacity)
[0204] Among them, Threshold is the data heat threshold, IOPS indicates the number of input and output operations that the storage node can process per second, which is used to characterize the access request processing capability of each storage node, Capacity indicates the data storage capacity of each storage node, which can be expressed by the unit medium storage capacity of each storage node, and k indicates the unit capacity cost ratio between different storage nodes.
[0205] Taking the storage media used by storage nodes as SSD and HDD as an example, assuming that the access request processing capacity per unit time of HDD (the number of input and output operations that can be processed per unit time, that is, the unit IO capacity) is IOPS_HDD, the unit storage capacity is Capacity_HDD, and the unit capacity cost ratio of SSD to HDD is set to k. The data heat threshold for the migration of target data objects between different storage nodes in the target data storage cluster is as follows:
[0206] Threshold=k*(IOPS_HDD / Capacity_HDD)
[0207] Furthermore, the data objects that need to be migrated are determined based on the data heat threshold and the data heat information of the data objects, and then the target storage nodes for the migration of each data object can be determined based on the target storage status of each data object in different storage nodes determined in the aforementioned steps. Then, the migration priority of multiple data objects that need to be migrated can be determined based on the data heat information and the data validity period, and each data object in the target data storage cluster is migrated according to the migration priority. Among them, the migration priority can be determined based on the ratio of the data heat information to the data validity period, for example, it can be determined based on Hotness / timeToLive, or it can be determined based on Hotness / log(timeToLive).
[0208] When performing data migration between storage nodes for different data objects, the disclosed embodiment compares the data heat information of the data object with the data heat threshold for data migration, thereby determining the data object that needs to be migrated. Furthermore, the migration priority between multiple data objects is determined based on the data validity period and the data heat information, which can avoid migrating data objects with shorter data validity periods. Then, the target storage node for data object migration can be determined based on the target storage state of each data object determined in the aforementioned steps, and the data object migration is migrated from the current storage node to the target storage node, thereby improving the accuracy of data migration.
[0209] Taking the storage media used by storage nodes as HDD and SSD as examples, the data migration plan for data objects provided by the embodiment of the present disclosure is explained. The I / O processing speed of the storage node using SSD is faster and the storage capacity is smaller; the I / O processing speed of the storage node using HHD is slower and the storage capacity is larger. Different data migration decisions may include: for data objects whose data heat information stored in SSD is less than Threshold, migrate to the storage nodes corresponding to HDD in sequence according to the migration priority; for data objects whose data heat information stored in SSD is equal to or greater than Threshold, do not migrate data; for data objects whose data heat information stored in HHD is equal to or less than Threshold, do not migrate data; for data objects whose data heat information stored in HDD is greater than Threshold, migrate to the storage node corresponding to SSD according to the migration priority. The migration priority definition and different data migration decisions are shown in Table 3 below:
[0210] Table 3 Migration priority definitions and different data migration decisions
[0211]
[0212] Step 350: Migrate the target data object in the target data storage cluster according to the data migration plan.
[0213] In some embodiments, a data migration plan is obtained according to the aforementioned steps, and data migration of the target data object in the target data storage cluster can be performed according to the data migration plan. Figure 5 , the migration of data from the first storage node to the second storage node is called a data cooling operation. The I / O processing capability of the second storage node is inferior to that of the first storage node. The migration of data from the second storage node to the first storage node is called a data warm-up operation. Since the data migration operation will occupy the input and output operations and bandwidth and other resources of the storage node, large-scale and frequent data migration during the high-frequency period of user access requests will seriously affect the overall performance of the data storage cluster. Therefore, the embodiment of the present disclosure performs data migration operations in the time period when I / O resources are less occupied, so as to reduce the impact of data migration on the overall performance of the data storage cluster. The migration of data is divided into three stages: reading data-writing data-updating metadata. The data object to be migrated can be read from the original storage node. Furthermore, the target data object can be written to the target storage node for migration, and the corresponding data metadata can be updated to complete the data migration action.
[0214] Specifically, step 350 includes but is not limited to the following steps:
[0215] Obtaining in real time the utilization rate of data access request processing resources in each storage node in the target data storage cluster;
[0216] Determine the migration time for data migration of the target data object according to the utilization rate;
[0217] Based on the migration time, data migration is performed on the target data object according to the data migration plan.
[0218] The data migration process provided by the embodiment of the present disclosure may include: determining the start time of data migration, determining the target data object to be migrated and the target storage node of migration, and determining the end time of data migration.
[0219] In some embodiments, the utilization rate of data access request processing resources of each storage node can be obtained in real time, and the utilization rate of each storage node is compared with a preset cooling start threshold (C_start) to determine the start time of data migration. The cooling start threshold can be set to 90%, which is not limited here; the utilization rate of each storage node is compared with a preset cooling end threshold (C_end) to determine the end time of data migration (data cooling). The cooling end threshold can be set to 85%, which is not limited here.
[0220] Furthermore, when performing data cooling, if it is detected that the utilization of a storage node exceeds the starting threshold, a target data object with a data heat lower than the Threshold can be selected, and the target data object can be migrated to the target storage node according to the migration priority defined in the previous steps to implement the data cooling process. When it is detected that the utilization of the storage node is lower than the preset termination threshold, the data cooling process ends.
[0221] In other embodiments, the highest number of input / output operations of each storage node within a period of time, i.e., the I / O peak value, can also be obtained in real time. The number of input / output operations per second (IOPS) of each storage node within the time period is calculated based on the highest number of input / output operations and the corresponding duration. The IOPS of each storage node is compared with the preset warm-up start threshold (IOPS_start) to determine the start time of data migration (warm-up). The warm-up start threshold can be set to 60%, which is not limited here; the IOPS of each storage node is compared with the preset warm-up end threshold (IOPS_end) to determine the end time of data migration. The warm-up end threshold can be set to 55%, which is not limited here.
[0222] When performing data recovery, if it is detected that the IOPS of a storage node within a period of time exceeds the preset recovery start threshold, the target data object with data heat greater than the Threshold can be selected, and the target data object can be migrated to the target storage node according to the migration priority defined in the previous steps to realize the data recovery process; when it is detected that the IOPS of the storage node within a period of time is lower than the preset recovery termination threshold, the data recovery process is terminated.
[0223] Based on the migration time, the target data object is migrated according to the data migration plan. The data migration operation is that the original storage node deletes the target data object, the target storage node writes the target data object, and the data metadata information of the target data object is updated to complete the data migration.
[0224] Please refer to Figure 6 , Figure 6 Another system architecture diagram of the data processing method of the embodiment of the present disclosure. The user terminal 110 sends access request information to the data migration control device 120 through the object service interface. The data migration control device 120 can obtain the data metadata information of the target data object and the access request information within the first time period, and extract features of the data metadata information of the target data object and the access request information within the first time period through the target data object feature extraction module 610, and input the extracted two aspects of features into the access characteristic analysis module 640 to predict the access characteristics in the future period, and input them into the data migration plan solution module 620 to solve the data migration plan. The access characteristic prediction module 640 includes a first neural network model and a second neural network model. The data heat information of the target data object is predicted based on the first neural network model, and the data validity period of the target data object is predicted based on the second neural network model.
[0225] Furthermore, the predicted data heat information and data effective duration are input into the data migration plan solving module 620, and the data migration plan is solved in combination with the data metadata information of the target data object and the storage medium information of the storage node, and the solved data migration plan is sent to the data migration execution module 630. The data migration execution module 630 performs data migration to the storage nodes in the data storage cluster according to the data migration plan. In this process, the training sample data can also be continuously obtained from the data access log of the target data storage cluster to update the first neural network model and the second neural network model.
[0226] The embodiments of the present disclosure are described in detail in combination with specific application scenarios.
[0227] like Figure 7As shown, it is another flow chart of the data processing method provided by the present disclosure. This embodiment will take the cloud object storage system scenario as an example and combine the execution subject of each step to introduce the data processing method in detail. The method specifically includes the following steps:
[0228] Step 701: The model training server obtains first training sample data and second training sample data from the data access log of the target data storage cluster.
[0229] The cloud object storage system contains multiple data objects. The data objects can be data corresponding to a certain application, or any other type of data, such as text and video. In this embodiment, the data object is an application data object. As the model training process requires high-speed processing and response speed of huge computing resources, the model training server needs to have strong computing power and storage capacity, as well as a stable network connection and a reliable software environment. The model training server can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc., which are not limited here.
[0230] The model training server obtains first training sample data and second training sample data from the data access log of the cloud object storage system. The first training sample data and the second training sample data both include data metadata information of the application data object and access request information within a period of time before the current moment.
[0231] Specifically, the data metadata information of the first training sample data includes objURL and objSize, objURL includes the AppID of the application data object, the name of the storage bucket where it is located, and the name of the application data object, and objSize is the size of the data stored in the application data object; the access request information of the first training sample data includes the time difference kRefTime from the k access times in the previous period before the current moment to the current moment, and the size kRefSize of the data volume of the k access requests in the previous period before the current moment.
[0232] The data metadata information of the second training sample data includes objURL and createTime. objURL includes the AppID of the application data object, the name of the storage bucket where it is located, and the name of the application data object. CreateTime is the time difference between the creation time of the application data object and the current moment. The access request information of the second training sample data includes the time difference kRefTime from the k access times in the previous period before the current moment to the current moment.
[0233] Step 702: The model training server trains a data heat prediction model based on the first training sample data, and trains a data effective duration prediction model based on the second training sample data.
[0234] In this embodiment, the model training server trains the data heat prediction model based on the first training sample data. The input of the model is the data metadata information of the first sample training data and the access request information of the first sample training data in the first time period before the current moment. The output is the data heat label in the second time period after the first time period and adjacent to the first time period, so as to update the parameters of the data heat prediction model. The data heat prediction model can predict the data heat information of the application data object in the second time period based on the data metadata information of the application data object and the access request information in the first time period.
[0235] Furthermore, the model training server trains the data effective duration prediction model based on the second training sample data, the input of the model is the data metadata information of the second sample training data and the access request information of the second sample training data in the first time period before the current moment, and the output is the data effective duration label, so as to update the parameters of the data effective duration prediction model. The data effective duration prediction model can predict the data effective duration of the application data object starting from the current prediction moment according to the data metadata information of the application data object and the access request information in the first time period.
[0236] Step 703: The model training server sends the model data of the data heat prediction model and the data effective duration prediction model to the data migration control device for deployment.
[0237] In the above steps, the model training server trains a data heat prediction model based on the first training sample data, and trains a data effective time prediction model based on the second training sample data, and then sends the data heat prediction model and the data effective time prediction model to the data migration control device, which deploys the data heat prediction model and the data effective time prediction model.
[0238] The data migration control device may be a terminal or a server, which is not limited here. In order to ensure the deployment accuracy and smoothness of the model deployment process, the data migration control device may have a larger storage space and a faster data transmission speed, and have better network connection stability.
[0239] Step 704: The data migration control device obtains data meta information of multiple data objects in the target data storage cluster and access request information in a first time period before the current moment.
[0240] In this embodiment, the data migration control device can access, manage and migrate data in the cloud object storage system. The data migration control device obtains data metadata information of multiple application data objects in the cloud object storage system, including AppID, the name of the storage bucket, the name of the application data object, the size of the data body stored in the application data object, objSize, the time difference createTime between the creation time of the application data object and the current moment; and access request information in a first time period before the current moment, which may include the time difference kRefTime from the k access times in the previous time period before the current moment to the current moment, and the size kRefSize of the amount of data of the k access requests in the previous time period before the current moment.
[0241] Step 705: The data migration control device inputs the data metadata information and the access request information into the data heat prediction model to obtain the output data heat information of each data object in the second time period after the current moment.
[0242] Furthermore, the data migration control device can input the data metadata information of the application data object (including AppID, the name of the storage bucket where it is located, the application data object name and the data body size objSize stored in the application data object), and the access request information (including kRefTime and kRefSize) into the data heat prediction model to obtain the output data heat information Hotness of each data object in the cloud object storage system in the second time period after the current moment.
[0243] Step 706: The data migration control device inputs the data metadata information and the access request information into the data effective duration prediction model to obtain output data effective duration information of each data object.
[0244] Specifically, the data migration control device can input the data metadata information of the application data object (including AppID, the name of the storage bucket where it is located, the name of the application data object, and the time difference createTime between the creation time of the application data object and the current time), as well as the access request information (including kRefTime) into the data validity period prediction model, and obtain the output data validity period information timeToLive of each application data object in the cloud data storage cluster within the second time period after the current time.
[0245] Step 707: The data migration control device obtains node attribute information of each storage node in the target data storage cluster.
[0246] After obtaining the data heat information of each data object in the second time period after the current moment based on the data heat prediction model and obtaining the data effective duration information of each data object through the data effective duration prediction model, the data migration control device further obtains the node attribute information of each storage node in the cloud data storage cluster. Specifically, the node attribute information may include the storage capacity, storage capacity utilization, access request processing capability (such as IOPS) and unit storage medium cost of each storage node.
[0247] By obtaining the storage capacity and access request processing capability of each storage node, it is ensured that after the data object is migrated to the target storage node, the target storage node can meet the data access mode required by the data object, such as high-frequency access.
[0248] Step 708: The data migration control device determines a data migration target according to the data heat information and effective duration information of each data object and the node attribute information of each storage node.
[0249] Specifically, the data migration target is the target storage state of each application data object in the cloud object storage system, the application data objects that need to be migrated, and the migration priority. The disclosed embodiment obtains the node attribute information of each storage node, and optimizes and solves the target storage state of each application data object when the total data storage cost of each application data object stored in different storage nodes in the cloud object storage system is minimized according to the node attribute information of each storage node and the data heat information of the application data object. The data migration control device compares the data heat information of each data object with the data heat threshold for data migration, and determines whether each application data object needs to be migrated according to the node attribute information of each storage node where each application data object is currently located. The specific data migration decision method has been introduced in the above steps and will not be repeated here. Furthermore, the migration priority of multiple application data objects that need to be migrated is determined based on the data validity period and data heat information of the application data object, and the application data objects are migrated between different storage nodes in the cloud object storage system according to the priority.
[0250] Step 709: The data migration control device obtains the access resource utilization of each storage node, and determines a data migration plan according to the data migration target and the access resource utilization.
[0251] Specifically, the data migration plan includes the start time of data migration, the application data objects to be migrated, and the end time of data migration. The data migration control device can obtain the access resource utilization of each storage node in real time, compare the access resource utilization with the preset utilization threshold, and determine the start time and end time of data migration according to the comparison result. The specific process has been introduced in the above steps and will not be repeated here. Further, the data migration control device determines the data migration plan according to the data migration target and the start time and end time of data migration.
[0252] Step 710: The data migration control device sends the data migration plan to each storage node, so that each storage node performs corresponding data processing according to the data migration plan.
[0253] Finally, the data migration control device sends the data migration plan to each storage node in the cloud object storage system. Each storage node writes or deletes the corresponding application data according to the data migration plan, and updates the data metadata information of the application data to complete the data migration.
[0254] Description of the apparatus and device of the present disclosure
[0255] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the characterization of arrows, these steps are not necessarily executed in sequence according to the order of arrow characterization. Unless there is a clear description in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.
[0256] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target content such as target content attribute information or attribute information sets, the permission or consent of the target content will be obtained first, and the collection, use and processing of these data will comply with the relevant laws, regulations and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the attribute information of the target content, it will obtain the separate permission or separate consent of the target content through a pop-up window or jump to a confirmation page. After clearly obtaining the separate permission or separate consent of the target content, the necessary target content-related data used to enable the normal operation of the embodiment of the present application will be obtained.
[0257] Figure 8The data processing device 800 provided in the embodiment of the present disclosure is a schematic diagram of the structure of the data processing device 800. The data processing device 800 includes:
[0258] An acquisition unit 801 is used to acquire data metadata information of a target data object and first access request information within a first time period;
[0259] A first prediction unit 802 is used to input the data metadata information and the first access request information into a first neural network model to perform data heat prediction, and obtain data heat information of the target data object in a second time period, where the second time period is a time period after the first time period and adjacent to the first time period;
[0260] The second prediction unit 803 is used to input the data metadata and the first access request information into a second neural network model to predict the data validity period, and obtain the data validity period corresponding to the target data object;
[0261] A generating unit 804 is used to generate a data migration plan of a target data object in a target data storage cluster based on the data metadata, the data popularity information and the data validity period, wherein the data storage cluster includes at least two data storage nodes with different data storage capabilities;
[0262] The migration unit 805 is used to migrate the target data object in the target data storage cluster according to the data migration plan.
[0263] Optionally, in some embodiments, the present application further provides a first model training device, the first model training device comprising:
[0264] A first acquisition unit is used to acquire, from a data access log of a target data storage cluster, first sample data metadata information of a plurality of first sample data objects, first sample access request information of each first sample data object in a plurality of first sample time periods, and second sample access request information of each first sample data object in a plurality of second sample time periods, wherein the second sample time period is a time period after the first sample time period and adjacent to the first sample time period;
[0265] A first calculation unit, configured to calculate the access resource consumption of each first sample data object in a second sample time period according to the second sample access request information;
[0266] A second calculation unit, configured to calculate a data heat label of each first sample data object in a second sample time period based on the access resource consumption and the first sample data metadata;
[0267] The first updating unit is used to update the parameters of the first neural network model using the first sample data metadata and the first sample access request information of each first sample data object as model input and the corresponding data heat label as model output label.
[0268] Optionally, in some embodiments, the second computing unit includes:
[0269] A first extraction subunit, configured to extract the sample data size and the number of copies of the first sample data object from the first sample data metadata;
[0270] The first calculation subunit is used to calculate the ratio of the access resource consumption to the product of the sample data size and the number of copies, and obtain the data heat label of each first sample data object in the second sample time period.
[0271] Optionally, in some embodiments, the first computing unit includes:
[0272] a second extraction subunit, configured to extract sample access type information and sample access data volume information from the second sample access request information, and to extract storage type information of the first sample data object from the first sample data metadata information;
[0273] The first acquisition subunit is used to obtain amplification ratio information for amplifying the second sample access request information, and calculate the access resource consumption of each first sample data object in the second sample time period according to the amplification ratio information, sample access type information, sample access data volume information and storage type information.
[0274] Optionally, in some embodiments, the present application further provides a second model training device, the second model training device comprising:
[0275] A second acquisition unit is used to acquire second sample data metadata information of multiple second sample data objects, third sample access request information of a preset number of times before the sample moment, and a sample data valid time length label of the second sample data object at the sample moment from the data access log of the target data storage cluster;
[0276] The second updating unit is used to update the parameters of the second neural network model using the second sample data metadata and the third sample access request information as model inputs and the corresponding sample data effective time label as the model output label.
[0277] Optionally, in some embodiments, the second updating unit includes:
[0278] A third extraction subunit is used to extract sample data attribute information and creation time information of the second sample data object from the second sample data metadata;
[0279] A second calculation subunit, configured to calculate a first time difference based on the creation time information and the sample time;
[0280] a fourth extraction subunit, configured to extract access time information from each third sample access request information, and calculate a plurality of second time differences based on the access time information and the sample time;
[0281] The updating subunit is used to update the parameters of the second neural network model using the sample data attribute information, the first time difference and multiple second time differences as model inputs and the corresponding sample data effective time length labels as model output labels.
[0282] Optionally, in some embodiments, the data processing device further includes:
[0283] A third acquisition unit, used to acquire the deployment time of the first neural network model and the second neural network model;
[0284] a fourth acquisition unit, configured to acquire the first updated training sample data and the second updated training sample data from the data access log of the target data storage cluster within a third time period adjacent to the current time when the time interval between the deployment time and the current time is greater than a preset time threshold;
[0285] An updating unit is used to update the model parameters of the first neural network model based on the first updated training sample data, and to update the model parameters of the second neural network model based on the second updated training sample data.
[0286] Optionally, in some embodiments, the generating unit comprises:
[0287] The second acquisition subunit is used to acquire storage medium information corresponding to each storage node in the target data storage cluster;
[0288] A generation subunit is used to generate a data migration plan for a target data object in a target data storage cluster based on data metadata information, data heat information, storage medium information, and data validity period.
[0289] Optionally, in some embodiments, the generating subunit comprises:
[0290] A construction module is used to construct an optimization function for the total cost of data storage in a target data storage cluster based on data metadata information, data heat information, and storage medium information;
[0291] A first determination module is used to determine, according to an optimization function, a target storage state of each data object in different storage nodes when the total data storage cost is lowest;
[0292] The second determination module is used to determine a data migration plan of a target data object in a target data storage cluster according to a target storage state and a data validity period.
[0293] Optionally, in some embodiments, the second determining module includes:
[0294] A calculation submodule, used to calculate a data heat threshold for data migration of a target data object based on the access request processing capability and data storage capability corresponding to each storage node and the unit capacity cost ratio between different storage nodes;
[0295] The determination submodule is used to determine the data migration plan of the target data object according to the data heat threshold, data heat information, data validity period and meta information of the target data object.
[0296] Optionally, in some embodiments, the migration unit comprises:
[0297] A third acquisition subunit is used to obtain in real time the utilization rate of data access request processing resources in each storage node in the target data storage cluster;
[0298] A determination subunit is used to determine a migration time for performing data migration on a target data object according to the utilization rate;
[0299] The migration subunit is used to migrate data of the target data object according to the data migration plan based on the migration time.
[0300] Reference Fig. 9 , Fig. 9 The terminal structure diagram for implementing each method according to an embodiment of the present disclosure includes: a radio frequency (RF) circuit 910, a memory 915, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a wireless fidelity (WiFi) module 970, a processor 980, and a power supply 990. Those skilled in the art can understand that Fig. 9 The terminal structure shown does not constitute a limitation on the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0301] The RF circuit 910 may be used for receiving and sending signals during information transmission or communication. In particular, after receiving downlink information from the base station, the information is sent to the processor 980 for processing. In addition, the uplink data is sent to the base station.
[0302] The memory 915 may be used to store software programs and modules. The processor 980 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 915 .
[0303] The input unit 930 may be used to receive input digital or character information and generate key signal input related to the setting and function control of the content terminal. Specifically, the input unit 930 may include a touch panel 931 and other input devices 932.
[0304] The display unit 940 may be used to display input information or provided information and various menus of the content terminal. The display unit 940 may include a display panel 941.
[0305] The audio circuit 960 , the speaker 961 , and the microphone 962 may provide an audio interface.
[0306] In this embodiment, the processor 980 included in the terminal can execute the data processing method of the previous embodiment.
[0307] The terminals of the embodiments of the present disclosure include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The embodiments of the present invention can be applied to various scenarios, including but not limited to content recommendation, data screening, etc.
[0308] Fig.10 A server structure diagram for implementing each method according to an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1022 (for example, one or more processors) and memory 1032, and one or more storage media 1030 (for example, one or more mass storage devices) storing application programs 1042 or data 1044. Among them, the memory 1032 and the storage medium 1030 can be short-term storage or permanent storage. The program stored in the storage medium 1030 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1022 can be configured to communicate with the storage medium 1030 and execute a series of instruction operations in the storage medium 1030 on the server.
[0309] The server may also include one or more power supplies 1026, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1058, and / or one or more operating systems 1041, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0310] The central processor 1022 in the server can be used to execute the data processing method of the embodiment of the present disclosure.
[0311] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the data processing methods of the aforementioned embodiments.
[0312] The embodiment of the present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device executes and implements the above-mentioned data processing method.
[0313] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar contents, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0314] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated content, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated content before and after is in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0315] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.
[0316] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0317] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0318] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0319] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.
[0320] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.
[0321] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire data meta information of a target data object and first access request information within a first time period; Inputting the data meta information and the first access request information into a first neural network model to perform data heat prediction, and obtaining data heat information of the target data object in a second time period, where the second time period is a time period after the first time period and adjacent to the first time period; Inputting the data metadata and the first access request information into a second neural network model to predict the data validity period, and obtaining the data validity period corresponding to the target data object; Generate a data migration plan for the target data object in a target data storage cluster based on the data meta information, the data popularity information, and the data validity period, wherein the data storage cluster includes at least two data storage nodes with different data storage capabilities; The target data object is migrated in the target data storage cluster according to the data migration plan.
2. The method according to claim 1, characterized in that The training process of the first neural network model includes the following steps: Acquire, from the data access log of the target data storage cluster, first sample data metadata information of a plurality of first sample data objects, first sample access request information of each of the first sample data objects in a plurality of first sample time periods, and second sample access request information of each of the first sample data objects in a plurality of second sample time periods, wherein the second sample time period is a time period after the first sample time period and adjacent to the first sample time period; Calculate the access resource consumption of each of the first sample data objects in the second sample time period according to the second sample access request information; Calculate a data heat label of each of the first sample data objects in the second sample time period based on the access resource consumption and the first sample data metadata; The first sample data metadata and the first sample access request information of each of the first sample data objects are used as model inputs, and the corresponding data heat labels are used as model output labels to update the parameters of the first neural network model.
3. The method according to claim 2, characterized in that The calculating the data heat label of each of the first sample data objects in the second sample time period based on the access resource consumption and the first sample data metadata includes: Extracting the sample data size and the number of copies of the first sample data object from the first sample data metadata; The ratio of the access resource consumption to the product of the sample data size and the number of copies is calculated to obtain a data heat label of each of the first sample data objects in the second sample time period.
4. The method according to claim 2, characterized in that: The calculating the access resource consumption of each of the first sample data objects in the second sample time period according to the second sample access request information includes: extracting sample access type information and sample access data volume information from the second sample access request information, and extracting storage type information of the first sample data object from the first sample data metadata information; Obtain amplification ratio information for amplifying the second sample access request information, and calculate the access resource consumption of each first sample data object in the second sample time period based on the amplification ratio information, the sample access type information, the sample access data volume information and the storage type information.
5. The method according to claim 1, characterized in that The training process of the second neural network model comprises the following steps: Acquire second sample data metadata information of multiple second sample data objects, third sample access request information of a preset number of times before a sample moment, and sample data valid duration labels of the second sample data objects at the sample moment from the data access log of the target data storage cluster; The parameters of the second neural network model are updated by taking the second sample data metadata and the third sample access request information as model inputs and taking the corresponding sample data effective time label as model output label.
6. The method according to claim 5, characterized in that The method of taking the second sample data metadata and the third sample access request information as model inputs and taking the corresponding sample data valid duration label as model output label to update the parameters of the second neural network model includes: extracting sample data attribute information and creation time information of the second sample data object from the second sample data metadata; Calculate a first time difference based on the creation time information and the sample time; Extracting access time information from each of the third sample access request information, and calculating a plurality of second time differences based on the access time information and the sample time; The sample data attribute information, the first time difference and the multiple second time differences are used as model inputs, and the corresponding sample data effective duration label is used as the model output label to update the parameters of the second neural network model.
7. The method according to claim 1, characterized in that The method further comprises: Obtaining deployment time of the first neural network model and the second neural network model; When the time interval between the deployment time and the current time is greater than a preset time threshold, obtaining first updated training sample data and second updated training sample data from a data access log of the target data storage cluster in a third time period adjacent to the current time; The model parameters of the first neural network model are updated based on the first updated training sample data, and the model parameters of the second neural network model are updated based on the second updated training sample data.
8. The method according to claim 1, characterized in that The generating a data migration plan for the target data object in the target data storage cluster based on the data meta information, the data heat information and the data effective duration includes: Obtain storage medium information corresponding to each storage node in the target data storage cluster; A data migration plan for the target data object in a target data storage cluster is generated based on the data meta information, the data heat information, the storage medium information, and the data validity period.
9. The method according to claim 8, characterized in that The generating a data migration plan for the target data object in the target data storage cluster based on the data meta information, the data heat information, the storage medium information, and the data effective duration includes: Constructing an optimization function for the total cost of data storage in the target data storage cluster based on the data meta information, the data heat information and the storage medium information; Determine, according to the optimization function, a target storage state of each data object in different storage nodes when the total data storage cost is lowest; A data migration plan for the target data object in the target data storage cluster is determined according to the target storage state and the data validity period.
10. The method according to claim 9, characterized in that The step of determining a data migration plan for the target data object in the target data storage cluster according to the target storage state and the data validity period includes: Calculate the data heat threshold for data migration of the target data object based on the access request processing capacity and data storage capacity corresponding to each storage node and the unit capacity cost ratio between different storage nodes; A data migration plan for the target data object is determined according to the data heat threshold, the data heat information, the data validity period, and the meta information of the target data object.
11. The method according to claim 1, characterized in that: The migrating the target data object in the target data storage cluster according to the data migration plan includes: Acquire the utilization rate of data access request processing resources in each storage node in the target data storage cluster in real time; Determining a migration time for performing data migration on the target data object according to the utilization rate; Based on the migration time, data migration is performed on the target data object according to the data migration plan.
12. A data processing device, characterized in that: The device comprises: An acquisition unit, used to acquire data meta information of a target data object and first access request information within a first time period; a first prediction unit, configured to input the data meta information and the first access request information into a first neural network model to perform data heat prediction, and obtain data heat information of the target data object in a second time period, where the second time period is a time period after the first time period and adjacent to the first time period; A second prediction unit, configured to input the data metadata and the first access request information into a second neural network model to predict the data validity period, and obtain the data validity period corresponding to the target data object; A generating unit, configured to generate a data migration plan for the target data object in a target data storage cluster based on the data metadata, the data popularity information, and the data validity period, wherein the data storage cluster includes at least two data storage nodes with different data storage capabilities; A migration unit is used to migrate the target data object in the target data storage cluster according to the data migration plan.
13. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.
14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the data processing method according to any one of claims 1 to 11 is implemented.
15. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device executes the data processing method according to any one of claims 1 to 11.
