Data reading and writing method and device and computing device cluster
By obtaining data description information to judge its popularity and optimizing the storage location, the problem of low data storage location allocation efficiency in the cold start stage of computing equipment is solved, reducing data read and write delays and improving storage efficiency.
Patent Information
- Application Number
- CN202310845797.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-10
- Publication Date
- 2025-01-10
AI Technical Summary
During the cold start phase of the computing device, it is difficult to effectively determine the storage location of the data, resulting in a long delay in reading and writing of the data.
By obtaining the description information of the data, we judge whether the data is hot data, and write the data to high-speed storage or low-speed storage according to its heat. We use the I/O interface of the file system to carry the description information to classify hot and cold data, and optimize the storage location allocation.
It reduces the data read and write delay in the cold start stage, improves storage efficiency and accuracy, and improves the flexibility of data management.
Smart Images
Figure CN119292995A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of storage, and in particular to a method, apparatus and computing device cluster for reading and writing data. Background Art
[0002] With the continuous development of storage technology, many computing devices have deployed high-speed local non-volatile memory express (NVMe) devices and dedicated buffer nodes for functional items, which has evolved the storage environment of computing devices into a multi-layer storage environment, including multi-layer storage and burst buffers. This storage structure is called deep memory and storage hierarchy (DMSH). DMSH greatly improves the write performance between the application layer and the storage layer, but the difficulty of reading data has increased significantly. When reading data, it needs to be cached from the underlying parallel file system to the memory, and it needs to move between different levels of storage during the process, which has a high data reading latency.
[0003] To eliminate the data reading delay, HFetch hierarchical data prefetcher collects statistical information of each data segment by monitoring the file system, and obtains the events that occur in each layer of the multi-layer storage. The control center uses the number of events that occur in the collected storage and the time when the event occurs to calculate the hotness of the data stored in the storage, and stores the data in each layer of the multi-layer storage according to the hotness of the data. Hot data is stored in the storage layer close to the application layer, and cold data is stored in the storage layer far away from the application layer. While realizing data hierarchical storage, it reduces the data reading delay.
[0004] Although the HFetch layered data processing method reduces the delay in reading data, it requires the historical data stored in the data to be used as the basis for hot and cold calculations of the data. Therefore, the HFetch layered data processing method needs to run for a period of time before it can take effect. How to further reduce the data reading and writing delay in the cold start phase has become an urgent problem to be solved. Summary of the invention
[0005] The embodiments of the present application provide a method, apparatus and computing device cluster for reading and writing data, which are used to solve the problem that the data storage location is difficult to determine during the cold start phase, and further reduce the delay of data reading and writing.
[0006] A first aspect of an embodiment of the present application provides a method for reading and writing data, including: obtaining first data and description information of the first data; determining whether the first data is hot data based on the description information of the first data; if so, writing the first data to a first storage; if not, writing the first data to a second storage, wherein a data reading and writing speed of the first storage is greater than a data reading and writing speed of the second storage.
[0007] In an embodiment of the present application, after obtaining the first data and the description information of the first data, the description information of the first data is used as a basis for determining whether the first data is hot data. If the first data is hot data, the first data is written to the first storage. If the first data is not hot data, the first data is written to the second storage, wherein the data reading and writing speed of the first storage is greater than the data reading and writing speed of the second storage. The description information of the first data obtained at the same time as the first data is obtained is used to determine whether the first data is hot data, and the storage address of the first data is determined. This eliminates the problem that it is difficult to store the data in the storage corresponding to the data heat according to the heat of the data during the cold start phase, and further reduces the delay in data reading and writing.
[0008] In a possible implementation of the first aspect, the description information of the first data includes a data temperature of the first data; and judging whether the first data is hot data according to the description information of the first data includes: judging whether the data temperature of the first data is greater than or equal to a first threshold.
[0009] In the embodiment of the present application, when judging whether the first data is hot data according to the description information of the first data, and the description information of the first data includes the data temperature of the first data, it can be determined whether the first data is hot data by judging whether the data temperature of the first data is greater than a first threshold. By carrying the data temperature of the first data in the description information of the first data, it helps to judge whether the first data is hot data, and then allocates the storage location of the first data according to whether the first data is hot data, thereby improving the accuracy of data storage location allocation in the cold start phase.
[0010] In a possible implementation of the first aspect, the description information of the first data includes an estimated file size of the first data; judging whether the first data is hot data according to the description information of the first data includes: judging whether the estimated file size of the first data is less than a second threshold.
[0011] In an embodiment of the present application, when determining whether the first data is hot data based on the description information of the first data, and when the description information of the first data includes the estimated file size of the first data, it is possible to help confirm whether the first data is hot data by determining whether the estimated file size of the first data is less than a second threshold, and then allocate the storage location of the first data based on whether the first data is hot data, thereby improving the accuracy of data storage location allocation in the cold start phase.
[0012] In a possible implementation of the first aspect, the description information of the first data includes a file type of the first data; and judging whether the first data is hot data according to the description information of the first data includes: judging whether the file type of the first data is included in the first file type.
[0013] In an embodiment of the present application, when determining whether the first data is hot data based on the description information of the first data, and when the description information of the first data includes the file type of the first data, it is possible to determine whether the file type of the first data is included in the first file type to help confirm whether the first data is hot data, and then allocate the storage location of the first data based on whether the first data is hot data, thereby improving the accuracy of data storage location allocation in the cold start phase.
[0014] In a possible implementation of the first aspect, the description information of the first data also includes an access mode of the first data; the method also includes: obtaining a signal for reading a first sub-data, the first sub-data being included in the first data; and reading the first sub-data into a memory according to the access mode.
[0015] In the embodiment of the present application, after the first data is stored in the designated storage, the first sub-data can also be read to the memory according to the access mode of the first data after the signal for reading the first sub-data is obtained, wherein the first sub-data is included in the first data, and the description information of the first data includes the access mode of the first data. By adding the access mode of the first data to the description information of the first data, the access mode to be used when the first data needs to be read is agreed in advance, which provides the possibility for pre-fetching of data, thereby further improving the efficiency of data reading and writing.
[0016] In a possible implementation of the first aspect, the access mode includes sequential access, stride access and random access. In the embodiment of the present application, in the process of reading the first sub-data into the memory, sequential access can be used to read, stride access can be used to read, and random access can be used to pre-fetch the first sub-data into the memory, further improving the efficiency of the scheme in reading data.
[0017] In a possible implementation of the first aspect, the description information of the first data also includes read and write information of the first data; the method also includes: acquiring a read signal of a first sub-data, the first sub-data being included in the first data; and processing the first sub-data according to the read and write information of the first data.
[0018] In the embodiment of the present application, the description information of the first data may further include the read and write information of the first data, and the read and write information of the first data indicates the operation corresponding to the number of read and write times of the first data. When the signal for reading the first sub-data is obtained and the first sub-data is included in the first data, the first sub-data may be processed according to the read and write information of the first data. This provides a further basis for data management and improves the utilization efficiency of storage space.
[0019] In a possible implementation of the first aspect, the first sub-data is processed according to the read-write information of the first data, including: when the read-write information of the first data is read once or write once, pre-fetching the first sub-data to the memory and deleting the first sub-data in the memory after completing the reading and writing of the first sub-data; when the read-write information of the first data is write multiple times and read multiple times or read multiple times, pre-fetching the first sub-data to the memory.
[0020] In the embodiment of the present application, the first sub-data is processed according to the read-write information of the first data in a specific manner: when the read-write information of the first data is read once or write once, the first sub-data is pre-fetched to the memory and after the read-write of the first sub-data is completed, the first sub-data in the memory is deleted; when the read-write information of the first data is write multiple reads or read multiple times, the first sub-data is pre-fetched to the memory. By defining different read-write information in different situations, it is determined whether to delete the first sub-data stored in the memory after use, thereby further improving the storage utilization efficiency.
[0021] In a possible implementation of the first aspect, the method also includes: obtaining access information of second sub-data, the second sub-data being included in the first data; calculating the data temperature of the second sub-data based on the access information of the second sub-data; if the first data is cold data and the data temperature of the second sub-data is greater than a first threshold, deleting the second sub-data from the second storage, and writing the second sub-data to the first storage and memory; if the first data is hot data and the data temperature of the second sub-data is less than or equal to the first threshold, deleting the second sub-data from the first storage and memory, and writing the second sub-data to the second storage.
[0022] In the embodiment of the present application, after storing the first data, access information of the second sub-data can also be obtained, the second sub-data is included in the first data, and the data temperature of the second sub-data is calculated according to the access information of the second sub-data. If the first data is cold data and the data temperature of the second sub-data is greater than the first threshold, the second sub-data is deleted from the second storage, and the second sub-data is written to the first storage and memory; if the first data is hot data and the data temperature of the second sub-data is less than or equal to the first threshold, the second sub-data is deleted from the first storage and memory, and the second sub-data is written to the second storage. In the case where the hot and cold types of the first data are different from those of the second sub-data, the second sub-data in the first storage and memory or the second storage is deleted and the second sub-data is stored in the second storage or the first storage and memory, thereby realizing efficient management of the storage location of the data according to the hot and cold conditions of the data.
[0023] A second aspect of the present application provides a data reading and writing device, comprising:
[0024] An acquiring unit, used to acquire first data and description information of the first data;
[0025] a judging unit, configured to judge whether the first data is hot data according to the description information of the first data;
[0026] A writing unit, configured to write the first data into the first storage if yes;
[0027] The writing unit is further used to write the first data into the second storage if no, and the data reading and writing speed of the first storage is greater than the data reading and writing speed of the second storage.
[0028] In a possible implementation manner of the second aspect, the description information of the first data includes a data temperature of the first data;
[0029] The judging unit is specifically configured to judge whether the data temperature of the first data is greater than or equal to a first threshold.
[0030] In a possible implementation of the second aspect, the description information of the first data includes an estimated file size of the first data;
[0031] The judging unit is specifically configured to judge whether the estimated file size of the first data is less than or equal to a second threshold.
[0032] In a possible implementation of the second aspect, the description information of the first data includes a file type of the first data;
[0033] The judging unit is specifically configured to judge whether the file type of the first data is included in the first file type.
[0034] In a possible implementation of the second aspect, the description information of the first data further includes an access mode of the first data;
[0035] The acquisition unit is further used to acquire and read a first sub-data signal, where the first sub-data is included in the first data;
[0036] The acquisition unit is further used to read the first sub-data into the memory according to the access mode.
[0037] In a possible implementation of the second aspect, the access mode includes sequential access, stride access and random access.
[0038] In a possible implementation of the second aspect, the description information of the first data further includes read and write information of the first data;
[0039] The acquisition unit is further used to acquire and read a first sub-data signal, where the first sub-data is included in the first data;
[0040] The acquisition unit is further used to process the first sub-data according to the read and write information of the first data.
[0041] In a possible implementation manner of the second aspect, the acquiring unit is specifically configured to:
[0042] When the read / write information of the first data is read once or write once, pre-fetching the first sub-data to the memory and deleting the first sub-data in the memory after completing the reading / writing of the first sub-data;
[0043] When the read / write information of the first data is write-multiple-read or read-multiple-times, the first sub-data is pre-fetched into the memory.
[0044] In a possible implementation manner of the second aspect, the acquiring unit is further configured to acquire access information of second sub-data, where the second sub-data is included in the first data;
[0045] The device also includes a calculation unit, the calculation unit being configured to calculate the data temperature of the second sub-data according to the access information of the second sub-data;
[0046] The writing unit is further configured to delete the second sub-data from the second storage and write the second sub-data to the first storage and the memory if the first data is cold data and the data temperature of the second sub-data is greater than a first threshold;
[0047] The writing unit is further used to delete the second sub-data from the first storage and the memory, and write the second sub-data to the second storage if the first data is hot data and the data temperature of the second sub-data is less than or equal to the first threshold.
[0048] A third aspect of an embodiment of the present application provides a device for reading and writing data, including a processor and a memory, the memory is used to store instructions, and the processor is used to execute instructions. When the processor executes the instructions, it executes the method in the first aspect or any possible implementation of the first aspect.
[0049] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, including a program, which, when executed on a computer, enables the computer to execute a method as described in the first aspect or any possible implementation of the first aspect.
[0050] A fifth aspect of an embodiment of the present application provides a computing device cluster, including at least one data reading and writing device, each data reading and writing device including a processor and a memory;
[0051] The processor of at least one data reading and writing device is used to execute instructions stored in the memory of at least one data reading and writing device, so that the computing device cluster executes the method in the aforementioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 A structural diagram of a data movement framework provided in an embodiment of the present application;
[0053] Figure 2 A schematic diagram of a method for reading and writing data provided in an embodiment of the present application;
[0054] Figure 3 Another schematic diagram of the data reading and writing method provided in the embodiment of the present application;
[0055] Figure 4 A schematic diagram of a structure of a data reading and writing device provided in an embodiment of the present application;
[0056] Figure 5 Another structural schematic diagram of the data reading and writing device provided in the embodiment of the present application;
[0057] Figure 6 A schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;
[0058] Figure 7 Another structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0060] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0061] As computing devices are constantly upgraded, the storage devices they are equipped with are also constantly upgraded. Since data for different purposes have different storage requirements, more and more computing devices have deployed dedicated buffer nodes for functional items on the basis of deploying high-speed local non-volatile memory express (NVMe) devices, evolving the originally simple storage environment into a multi-layer storage environment. In order to eliminate the read delay caused by this multi-layer storage environment, the HFetch hierarchical data prefetcher is also used to monitor the events that occur to each data in the storage system, calculate the hotness of the data based on the events that occur to each data in the storage system, and store the data in each level of the multi-layer storage according to the hotness of the data.
[0062] For example, when the stored data includes monitoring data generated by the monitor and log files generated by the application running, the HFetch hierarchical data prefetcher monitors the events occurring in the monitoring data and log files in the storage system and finds that the monitoring data has never been read once it is generated, while the log files are continuously read and written in units of minutes. Therefore, the HFetch hierarchical data prefetcher considers the monitoring data to be cold data, while the log files are hot data. The monitoring data can be stored in a storage space away from the application layer, while the log files can be stored in a storage space or memory close to the application layer. Alternatively, the monitoring data can be stored in a low-speed storage space, and the log files can be stored in a high-speed storage space. Among them, the low-speed storage space is a storage space with a low read and write speed such as a hard disk, and the high-speed storage space can be a storage space with a high read and write speed such as an NVMe device.
[0063] However, although the HFetch hierarchical data processing method reduces the latency of reading data, it is necessary to analyze the hotness and coldness of the data based on the historical events that occurred in the storage system to determine how to reasonably allocate the data storage address. How to further solve the problem of data storage location allocation during the cold start phase of computing devices has become an urgent problem to be solved.
[0064] Based on the above problems, the present application proposes that the idle bits in the I / O interface of the file system can be used to carry the description information of the file, and the description information of the file can be used to classify the file into hot and cold data, and the hot data can be stored in high-speed storage and the cold data can be stored in low-speed storage. The method of classifying the files into hot and cold data using the description information carried by the files can realize the allocation of hot and cold data in the cold start phase of the computing device, and improve the efficiency of data storage.
[0065] For example, see Figure 1 , Figure 1 A structural diagram of a data movement framework provided in an embodiment of the present application.
[0066] The data movement framework includes the application layer, file system layer and storage layer.
[0067] The file system layer connects the application layer and the storage layer and is used to transfer the data generated by the application layer to the storage layer for storage.
[0068] The storage layer includes local memory, burst buffer and storage nodes. Storage nodes can be a variety of storage hardware with different storage capabilities, such as solid state disk (SSD) and hard disk drive (HDD). SSD is a high-speed storage node and HDD is a low-speed storage node.
[0069] The local memory is used to temporarily store the data generated by the application layer, and transfer the data generated by the application layer to the burst buffer, which is then stored in the SSD or HDD according to the storage policy assigned by the file system layer.
[0070] Specifically, after generating data, the application layer transmits a write request to the file system layer, and uses the idle bits of the file system I / O interface to send Hints information to the storage node. The file system I / O interface module analyzes the hotness and coldness of the data based on the Hints information, and the redirection module assigns a storage strategy to the data based on the hotness and coldness of the file, specifying that the data is written to the SSD or HDD. During the operation of the data movement framework, the information statistics module of the file system layer periodically collects statistical information in the data movement framework to further maintain the hotness and coldness information of the data. For example, information such as the number of times the data is read and written and the time when the data is read and written.
[0071] During the cold start phase, the data movement framework uses the data description information to determine whether the data is hot or cold, allocates a storage strategy for the data based on the hot or cold nature of the data, and writes the data into the SSD or HDD, thereby realizing the allocation of hot and cold data during the cold start phase of the computing device and improving the efficiency of data storage.
[0072] First, the method of reading and writing data in the case of cold start provided by this application is introduced. Figure 2 , Figure 2 A schematic diagram of a method for reading and writing data provided in an embodiment of the present application.
[0073] 201. Obtain first data and description information of the first data;
[0074] During a cold start phase of the storage system, first data and description information of the first data are obtained, wherein the description information of the first data includes at least one of an estimated file size of the first data, a file type of the first data, or a data temperature of the first data.
[0075] Optionally, the description information of the first data may further include an access mode of the first data and / or read / write information of the first data.
[0076] Exemplarily, the access mode of the first data includes sequential access (sequential), stride access (reverse_sequential) and random access (random).
[0077] The read and write information of the first data includes read once (read_once), write once (write_once), read modify (read_mody) or write modify (write_mody).
[0078] Specifically, the description information of the first data may be Hints information added to the idle bits in the I / O interface of the file system, and different information of the first data may be transmitted using different bits according to a pre-agreed rule, which is not limited here.
[0079] It is understandable that the description of the description information of the first data here is only an example. In actual applications, the form and content of the description information of the first data should be set in combination with specific application scenarios, and no limitation is made here.
[0080] 202. Determine whether the first data is hot data according to description information of the first data;
[0081] After the first data and the description information of the first data are acquired, it is determined whether the first data is hot data according to the description information of the first data.
[0082] Exemplarily, when the description information of the first data includes the data temperature of the first data, it is determined whether the data temperature of the first data is greater than a first threshold.
[0083] If it is greater, the first data is hot data, and step 203 is executed.
[0084] If it is less than or equal to, the first data is considered to be cold data, and step 204 is executed.
[0085] Exemplarily, when the description information of the first data includes an estimated file size of the first data, it is determined whether the estimated file size of the first data is greater than a second threshold.
[0086] If it is less than, the first data is hot data, and step 203 is executed.
[0087] If it is greater than or equal to, the first data is considered to be cold data, and step 204 is executed.
[0088] Exemplarily, when the description information of the first data includes the file type of the first data, it is determined whether the file type of the first data is included in the first file type. For example, the first file type includes log files, transaction information, hot search results, or artificial intelligence model parameter information.
[0089] If so, the first data is considered to be hot data, and step 203 is executed.
[0090] If not, the first data is considered to be cold data, and step 204 is executed.
[0091] When the description information of the first data includes both the data temperature of the first data and the estimated file size of the first data, or the data temperature of the first data and the file type of the first data, or the data temperature of the first data, the estimated file size of the first data and the file type of the first data, the data temperature of the first data is preferentially used as the basis for determining whether the first data is hot data. In other cases, the description information of the first data can be set in priority in combination with a specific application scenario, which is not limited here.
[0092] It is understandable that the description of the first data as the basis for determining whether the first data is cold data or hot data is only an example. In actual applications, the hot and cold classification of the first data should be determined in combination with specific application scenarios. When there are many types of storage hardware, the possible hot and cold classifications of the first data can be divided into hot data, warm data and cold data according to the data temperature, and the hot data is stored in an SSD with high read and write efficiency, the warm data is stored in a disk array, and the cold data is stored in an HDD. In specific cases, it can be set in combination with specific application scenarios, and there is no limitation here.
[0093] 203. Write the first data into the first storage;
[0094] When the first data is hot data, the first data is written into a first storage, wherein the first storage is a high-speed storage device, such as an SSD.
[0095] Optionally, when the first data is hot data, in a specific application scenario, when writing the first data to the first storage, it is necessary to first cache the first data to the memory, and then migrate the first data from the memory to the first storage. After completing the migration of the first data from the memory to the first storage, the first data in the memory can be selectively deleted or retained.
[0096] 204. Write the first data into the second storage;
[0097] When the first data is cold data, the first data is written to the second storage, wherein the second storage is a low-speed storage device, such as a HDD.
[0098] Specifically, when the first data is cold data, the first data is cached in the memory, and then the first data is migrated from the memory to the second storage. After the migration of the first data from the memory to the second storage is completed, the first data in the memory is deleted.
[0099] Optionally, when the read / write information of the first data is write_once, the first data does not need to be stored.
[0100] When the read / write information of the first data is any one of read_once, read_mody or write_mody, the first data is stored in the second storage.
[0101] It can be understood that the description of different processing methods for the first data when the read and write information of the first data is different is only an example, and should be set according to specific circumstances in actual applications, and is not limited here.
[0102] 205. Obtain and read a first sub-data signal;
[0103] After the first data is written into the first storage or the second storage, a first sub-data signal is obtained and read, wherein the first sub-data is included in the first data.
[0104] Optionally, the signal for reading the first sub-data may carry description information of the first sub-data, where the description information of the first sub-data includes at least one of an estimated file size of the first sub-data, a file type of the first sub-data, or a data temperature of the first sub-data.
[0105] Optionally, the description information of the first sub-data may further include the access mode of the first sub-data and / or the read-write information of the first sub-data.
[0106] It can be understood that the description of reading the description information of the first sub-data signal carrying the first sub-data is only an example, and in actual application, it should be set in combination with specific application scenarios.
[0107] 206. Read the first sub-data into the memory according to the description information of the first data.
[0108] After acquiring the first sub-data reading signal, the first sub-data is read into the memory according to the description information of the first data.
[0109] Exemplarily, when the first data is stored in the first storage, the first sub-data is read from the first storage to the memory.
[0110] If the first data is stored in the first storage and the memory and the read / write information of the first data is write_once or read_once, the first sub-data is read from the first storage to the memory, and after the processing of the first sub-data is completed, the first sub-data is deleted from the memory.
[0111] Furthermore, after the processing of the first sub-data is completed, the processed first sub-data is stored in the first storage or the second storage according to the actual heat of the processed first sub-data, and there is no limitation here.
[0112] If the first data is stored in the first storage, and the read / write information of the first data is write_mody or read_mody, the first sub-data is read from the first storage to the memory.
[0113] If the first data is stored in the first storage and the memory, since the first data is stored in the memory, there is no need to read the first sub-data.
[0114] If the first data is stored in the first storage and the memory and the read / write information of the first data is write_once or read_once, after the processing of the first sub-data is completed, the first sub-data is deleted from the memory.
[0115] If the first data is stored in the first storage, and the read / write information of the first data is write_mody or read_mody, no operation needs to be performed.
[0116] It can be understood that the description of different processing methods for the first data when the read and write information of the first data is different is only an example, and should be set according to specific circumstances in actual applications, and is not limited here.
[0117] If the first data is stored in the second storage and the read / write information of the first data is write_once or read_once, the first sub-data is read from the second storage to the memory, and after the local first sub-data is processed, the first sub-data in the memory is deleted.
[0118] If the first data is stored in the second storage, the read / write information of the first data is write_mody or read_mody and the access mode of the first data is sequential access, the first sub-data is read from the second storage to the memory.
[0119] If the first data is stored in the second storage, the read / write information of the first data is write_mody or read_mody and the access mode of the first data is stride access with a step length of n, the first sub-data is read from the second storage to the memory in a stride access manner. Specifically, if the access address of the first sub-data is A1, while reading the first sub-data from the address A1 of the second storage, the data of A1+n is directly pre-fetched from the address A1+n of the second storage to the memory for subsequent use.
[0120] It is understandable that the description of the specific method of reading the first sub-data into the memory according to the description information of the first data is only an example. In actual application, it should be set in combination with the specific application scenario and is not limited here.
[0121] In the embodiment of the present application, the access mode and read-write information of the first data are transmitted by using the description information of the first data, which makes it possible to improve the reading and writing efficiency of the data.
[0122] In the embodiment of the present application, after obtaining the first data and the description information of the first data, it is determined whether the first data is hot data according to the description information of the first data. When the first data is hot data, the first data is written into the first storage and memory; when the first data is cold data, the first data is written into the second storage. The description information of the first data carried when the first data is transmitted is used as the basis for determining whether the first data is hot data, and the determination of whether the first data is hot data is performed, and the storage location of the first data is allocated according to the determination result, which solves the problem of low efficiency in data storage location allocation during the cold start phase of the device.
[0123] Next, we will introduce the maintenance of data during operation. Figure 3 , Figure 3 Another schematic diagram of the data reading and writing method provided in an embodiment of the present application.
[0124] 301. Obtain access information of second sub-data;
[0125] When the first data has been stored in the first storage and memory or the second storage, access information of the second sub-data can be obtained during the operation of the device. The second sub-data is included in the first data, and the first sub-data and the second sub-data can be the same or different, which is not limited here.
[0126] Exemplarily, the access information of the second sub-data includes the number of accesses k to the second sub-data, the time t when the access information of the second sub-data is obtained, the time of the i-th access to the second sub-data, and the number of times the second sub-data is referenced n, where n≥1.
[0127] It is understandable that the description of the specific content of the access information of the second sub-data here is only an example. In actual application, it should be set in combination with the specific application scenario and the requirement for calculating the data temperature of the second sub-data, and no limitation is made here.
[0128] 302. Calculate the data temperature of the second sub-data according to the access information of the second sub-data;
[0129] After the access information of the second sub-data is obtained, the data temperature of the second sub-data is calculated according to the access information of the second sub-data, and the score of the second sub-data is calculated according to the following first formula.
[0130] First formula:
[0131] Where s is the second sub-data to be scored, k is the number of times the second sub-data is accessed, t is the time when the access information of the second sub-data is obtained, is the time when the second sub-data is accessed for the i-th time, and n is the number of times the second sub-data is referenced, where n≥1.
[0132] If the second sub-data is a frequently accessed data segment, or a recently accessed data segment, or a data segment with multiple references, its score will be higher, that is, the second sub-data will be hotter and suitable to be placed in the second storage with faster reading and writing speed.
[0133] It should be noted that the description of the first formula here is only an example. In actual applications, other formulas can be used to calculate the heat of the second sub-data, which is not limited here.
[0134] 303. Determine whether the data temperature of the second sub-data is greater than a first threshold;
[0135] After calculating the data temperature of the second sub-data according to the access information of the second sub-data, it is determined whether the data temperature of the second sub-data is greater than a first threshold value. If the data temperature of the second sub-data is greater than the first threshold value, the second sub-data is hot data; if the data temperature of the second sub-data is less than or equal to the first threshold value, the second sub-data is cold data.
[0136] If the second sub-data is hot data and the first sub-data is hot data, no operation is required.
[0137] If the second sub-data is cold data and the first data is cold data, no operation is required.
[0138] If the second sub-data is hot data and the first sub-data is cold data, step 304 is executed.
[0139] If the second sub-data is cold data and the first sub-data is hot data, step 305 is executed.
[0140] 304. When the first data is cold data, delete the second sub-data stored in the second storage, and write the second sub-data into the first storage;
[0141] When the first data is cold data and the second sub-data is hot data, the second sub-data stored in the second storage is deleted, and the second sub-data is written into the first storage and the memory.
[0142] 305. When the first data is hot data, delete the second sub-data stored in the first storage, and write the second sub-data into the second storage.
[0143] When the first data is hot data and the second sub-data is cold data, the second sub-data stored in the first storage is deleted, and the second sub-data is written into the second storage.
[0144] In the embodiment of the present application, the access information of the second sub-data is periodically collected to calculate the data temperature of the second sub-data, and it is determined whether the data temperature of the second sub-data is greater than the first threshold value. If it is greater, the second sub-data is hot data, and if it is less than or equal to the first threshold value, the second sub-data is cold data. The second sub-data is stored in a suitable storage according to the current storage location of the second sub-data and the hotness and coldness of the second sub-data. The data can be stored in a storage with suitable read and write efficiency in a timely manner according to the usage of the data, thereby improving the flexibility of data storage.
[0145] The above is an introduction to application scenarios in which storage hardware is relatively simple in an application environment. When the storage hardware includes many types of storage hardware, the data can be divided into cold data, hot data, and warm data on the basis of the above solution, and data of different temperatures can be stored in storage devices with different read and write capabilities. When data needs to be divided into cold data, warm data, and hot data for data classification and storage, cold data, warm data, and hot data can be divided by setting a threshold value of data temperature; cold data, warm data, and hot data can also be divided by setting different data type sets; cold data, warm data, and hot data can also be divided by setting a threshold value of the expected file size of the data. There is no restriction here. After determining that the data is cold data, warm data, or hot data, the data is stored in the corresponding storage space.
[0146] The above is an introduction to the data reading and writing method proposed in the embodiment of the present application. The following is an introduction to the data reading and writing device provided in the embodiment of the present application:
[0147] See also Figure 4 , Figure 4 A schematic diagram of the structure of a data reading and writing device provided in an embodiment of the present application.
[0148] Devices for reading and writing data, including:
[0149] An acquisition unit 401 is used to acquire first data and description information of the first data;
[0150] A judging unit 402, configured to judge whether the first data is hot data according to description information of the first data;
[0151] A writing unit 403, configured to write the first data into the first storage;
[0152] The writing unit 403 is further used to write the first data into the second storage if no, and the data reading and writing speed of the first storage is greater than the data reading and writing speed of the second storage.
[0153] Optionally, the description information of the first data includes data temperature of the first data;
[0154] The judging unit 402 is specifically configured to judge whether the data temperature of the first data is greater than or equal to a first threshold.
[0155] Optionally, the description information of the first data includes an estimated file size of the first data;
[0156] The judging unit 403 is specifically configured to judge whether the estimated file size of the first data is less than or equal to a second threshold.
[0157] Optionally, the description information of the first data includes a file type of the first data;
[0158] The judging unit 403 is specifically configured to judge whether the file type of the first data is included in the first file type.
[0159] Optionally, the description information of the first data further includes an access mode of the first data;
[0160] The acquisition unit 401 is further used to acquire and read a first sub-data signal, where the first sub-data is included in the first data;
[0161] The acquisition unit 401 is further configured to read the first sub-data into the memory according to the access mode.
[0162] Optionally, access modes include sequential access, strided access, and random access.
[0163] Optionally, the description information of the first data further includes read and write information of the first data;
[0164] The acquisition unit 401 is further used to acquire and read a first sub-data signal, where the first sub-data is included in the first data;
[0165] The acquisition unit 401 is further configured to process the first sub-data according to the read and write information of the first data.
[0166] Optionally, the acquiring unit 401 is specifically configured to:
[0167] When the read / write information of the first data is read once or write once, pre-fetching the first sub-data to the memory and deleting the first sub-data in the memory after completing the reading / writing of the first sub-data;
[0168] When the read / write information of the first data is write-time read or read-multiple-times, the first sub-data is pre-fetched into the memory.
[0169] Optionally, the acquiring unit 401 is further configured to acquire access information of second sub-data, where the second sub-data is included in the first data;
[0170] The apparatus further includes a calculation unit 404, the calculation unit 404 being configured to calculate the data temperature of the second sub-data according to the access information of the second sub-data;
[0171] The writing unit 403 is further configured to delete the second sub-data from the second storage and write the second sub-data to the first storage and the memory if the first data is cold data and the data temperature of the second sub-data is greater than the first threshold;
[0172] The writing unit 403 is further configured to delete the second sub-data from the first storage and the memory, and write the second sub-data to the second storage if the first data is hot data and the data temperature of the second sub-data is less than or equal to the first threshold.
[0173] Among them, the acquisition unit 401, the judgment unit 402, the writing unit 403 and the calculation unit 404 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of module A is introduced below by taking module A as an example. Similarly, the implementation of module B and module C can refer to the implementation of module A.
[0174] As an example of a software functional unit, the acquisition unit 401 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, module A may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.
[0175] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0176] As an example of a hardware functional unit, the acquisition unit 401 may include at least one computing device, such as a server, etc. Alternatively, the acquisition unit 401 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
[0177] The multiple computing devices included in the acquisition unit 401 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition unit 401 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition unit 401 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0178] It should be noted that, in other embodiments, the acquisition unit 401 can be used to execute any step in the method of data reading and writing, the judgment module 402 can be used to execute any step in the method of data reading and writing, the write unit 403 can be used to execute any step in the method of data reading and writing, and the calculation unit 404 can also be used to execute any step in the method of data reading and writing. The steps that the acquisition unit 401, the judgment unit 402, the write unit 403 and the calculation unit 404 are responsible for implementing can be specified as needed. The full functions of the data reading and writing device are realized by respectively implementing different steps in the method of data reading and writing by the acquisition unit 401, the judgment unit 402, the write unit 403 and the calculation unit 404.
[0179] See also Figure 5 , Figure 5 Another structural schematic diagram of the data reading and writing device provided in an embodiment of the present application.
[0180] The data reading and writing device 500 includes: a bus 502, a processor 504, a memory 506 and a communication interface 508. The processor 504, the memory 506 and the communication interface 508 communicate with each other through the bus 502. The data reading and writing device 500 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors 504 and memories 506 in the data reading and writing device 500.
[0181] The bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus 502 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 502 may include a path for transmitting information between various components of the data reading and writing device 500 (for example, the memory 506, the processor 504, and the communication interface 508).
[0182] The processor 504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0183] The memory 506 may include a volatile memory, such as a random access memory (RAM). The processor 504 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0184] The memory 506 stores executable program codes, and the processor 504 executes the executable program codes to respectively implement the functions of the aforementioned acquisition unit 401, the judgment unit 402, the writing unit 403, and the calculation unit 404, thereby implementing any one of the data storage methods. That is, the memory 506 stores instructions for executing any one of the data analysis methods.
[0185] Alternatively, the memory 506 stores executable codes, and the processor 504 executes the executable codes to respectively implement the functions of the aforementioned access points, thereby implementing any of the data storage methods. That is, the memory 506 stores instructions for executing any of the data storage methods.
[0186] The communication interface 508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the data reading and writing apparatus 500 and other devices or a communication network.
[0187] See also Figure 6 , Figure 6 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application.
[0188] The computing device cluster includes at least one data reading and writing device. The data reading and writing device 500 can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.
[0189] like Figure 6 As shown, the computing device cluster includes at least one data reading and writing device 500. The memory 506 in one or more data reading and writing devices 500 in the computing device cluster may store the same instructions for executing any one of the data storage methods.
[0190] In some possible implementations, the memory 506 of one or more data reading and writing devices 500 in the computing device cluster may also store partial instructions for executing any of the data storage methods. In other words, the combination of one or more data reading and writing devices 500 may jointly execute instructions for executing any of the data storage methods.
[0191] It should be noted that the memory 506 in different data reading and writing devices 500 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the cloud management platform. That is, the instructions stored in the memory 506 in different data reading and writing devices 500 can realize the functions of one or more units of the acquisition unit 401, the judgment unit 402, the writing unit 403 and the calculation unit 404.
[0192] In some possible implementations, one or more data storage devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network.
[0193] See also Figure 7 , Figure 7 Another structural diagram of a computing device cluster provided in an embodiment of the present application.
[0194] like Figure 7 As shown, two data reading and writing devices 500A and 500B are connected via a network. Specifically, they are connected to the network via communication interfaces in various computing devices. In this type of possible implementation, the memory 506 in the data reading and writing device 500A stores instructions for executing the functions of the acquisition unit 401 and the judgment unit 402. At the same time, the memory 506 in the data reading and writing device 500B stores instructions for executing the functions of the writing unit 403 and the computing unit 404.
[0195] It should be understood that Figure 7 The function of the data reading and writing device 500A shown in FIG. 5 can also be completed by multiple data reading and writing devices 500. Similarly, the function of the data reading and writing device 500B can also be completed by multiple data reading and writing devices 500.
[0196] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Figure 6 and Figure 7 The connection mode of the computing device cluster is different in that the memory 506 in one or more data reading and writing devices 500 in the computing device cluster may store the same instructions for executing any one of the data storage methods.
[0197] In some possible implementations, the memory 506 of one or more data reading and writing devices 500 in the computing device cluster may also store partial instructions for executing any of the data storage methods. In other words, the combination of one or more data reading and writing devices 500 may jointly execute instructions for executing any of the data storage methods.
[0198] It should be noted that the memory 506 in different data reading and writing devices 500 in the computing device cluster may store different instructions for executing part of the functions of the data reading and writing devices.
[0199] An embodiment of the present application also provides a computer-readable storage medium, including computer-readable instructions. When the computer-readable instructions are executed on a computer, the computer executes any one of the implementation methods shown in the aforementioned method embodiments.
[0200] The embodiments of the present application also provide a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed on a computer, the computer executes any one of the implementation methods shown in the aforementioned method embodiments.
[0201] The present application also provides a chip or chip system, which may include a processor. The chip may also include a memory (or storage module) and / or a transceiver (or communication module), or the chip is coupled to a memory (or storage module) and / or a transceiver (or communication module), wherein the transceiver (or communication module) can be used to support the chip for wired and / or wireless communication, and the memory (or storage module) can be used to store a program or a set of instructions, and the processor calls the program or the set of instructions to implement the above method embodiment, and the operation performed by the terminal or network device in any possible implementation of the method embodiment. The chip system may include the above chip, and may also include the above chip and other discrete devices, such as a memory (or storage module) and / or a transceiver (or communication module).
[0202] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0203] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0204] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0205] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0206] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.
Claims
1. A method for reading and writing data, characterized in that: include: Acquire first data and description information of the first data; Determining whether the first data is hot data according to the description information of the first data; If yes, write the first data into the first storage; If not, the first data is written into a second storage, and the data reading and writing speed of the first storage is greater than the data reading and writing speed of the second storage.
2. The method according to claim 1, characterized in that The description information of the first data includes the data temperature of the first data; The determining whether the first data is hot data according to the description information of the first data includes: It is determined whether a data temperature of the first data is greater than or equal to a first threshold.
3. The method according to claim 1, characterized in that The description information of the first data includes an estimated file size of the first data; The determining whether the first data is hot data according to the description information of the first data includes: It is determined whether the estimated file size of the first data is less than or equal to a second threshold.
4. The method according to claim 1, characterized in that: The description information of the first data includes the file type of the first data; The determining whether the first data is hot data according to the description information of the first data includes: It is determined whether the file type of the first data is included in the first file type.
5. The method according to any one of claims 1 to 4, characterized in that: The description information of the first data also includes an access mode of the first data; The method further comprises: Acquire and read a first sub-data signal, where the first sub-data is included in the first data; The first sub-data is read into a memory according to the access mode.
6. The method according to claim 5, characterized in that The access modes include sequential access, stride access and random access.
7. The method according to any one of claims 1 to 4, characterized in that: The description information of the first data also includes read and write information of the first data; The method further comprises: Acquire and read a first sub-data signal, where the first sub-data is included in the first data; The first sub-data is processed according to the read and write information of the first data.
8. The method according to claim 7, characterized in that The processing of the first sub-data according to the read-write information of the first data includes: When the read / write information of the first data is read once or write once, pre-fetching the first sub-data to the memory and deleting the first sub-data in the memory after completing the reading and writing of the first sub-data; When the read / write information of the first data is write multiple times or read multiple times, the first sub-data is pre-fetched into the memory.
9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Acquire access information of second sub-data, where the second sub-data is included in the first data; calculating the data temperature of the second sub-data according to the access information of the second sub-data; If the first data is cold data and the data temperature of the second sub-data is greater than the first threshold, deleting the second sub-data from the second storage, and writing the second sub-data to the first storage and the memory; If the first data is hot data and the data temperature of the second sub-data is less than or equal to the first threshold, the second sub-data is deleted from the first storage and the memory, and the second sub-data is written to the second storage.
10. A device for reading and writing data, characterized in that: include: An acquiring unit, configured to acquire first data and description information of the first data; a judging unit, configured to judge whether the first data is hot data according to the description information of the first data; A writing unit, configured to, if yes, write the first data into a first storage; The writing unit is further used for writing the first data into a second storage if no, and the data reading and writing speed of the first storage is greater than the data reading and writing speed of the second storage.
11. The device according to claim 10, characterized in that The description information of the first data includes the data temperature of the first data; The judging unit is specifically configured to judge whether the data temperature of the first data is greater than or equal to a first threshold.
12. The device according to claim 10, characterized in that The description information of the first data includes an estimated file size of the first data; The determining unit is specifically configured to determine whether the estimated file size of the first data is less than or equal to a second threshold.
13. The device according to claim 10, characterized in that The description information of the first data includes the file type of the first data; The determining unit is specifically configured to determine whether the file type of the first data is included in the first file type.
14. The device according to any one of claims 10 to 13, characterized in that: The description information of the first data also includes an access mode of the first data; The acquisition unit is further used to acquire and read a first sub-data signal, where the first sub-data is included in the first data; The acquisition unit is further configured to read the first sub-data into a memory according to the access mode.
15. The device according to claim 14, characterized in that The access modes include sequential access, stride access and random access.
16. The device according to any one of claims 10 to 13, characterized in that The description information of the first data also includes read and write information of the first data; The acquisition unit is further used to acquire and read a first sub-data signal, where the first sub-data is included in the first data; The acquisition unit is further used to process the first sub-data according to the read and write information of the first data.
17. The device according to claim 16, characterized in that The acquisition unit is specifically used for: When the read / write information of the first data is read once or write once, pre-fetching the first sub-data to the memory and deleting the first sub-data in the memory after completing the reading and writing of the first sub-data; When the read / write information of the first data is write-multiple-read or read-multiple-times, the first sub-data is pre-fetched into the memory.
18. The device according to any one of claims 10 to 17, characterized in that The acquisition unit is further used to acquire access information of second sub-data, where the second sub-data is included in the first data; The device further includes a calculation unit, the calculation unit being configured to calculate the data temperature of the second sub-data according to the access information of the second sub-data; The writing unit is further configured to, if the first data is cold data and the data temperature of the second sub-data is greater than the first threshold, delete the second sub-data from the second storage, and write the second sub-data to the first storage and the memory; The writing unit is further configured to delete the second sub-data from the first storage and the memory, and write the second sub-data to the second storage if the first data is hot data and the data temperature of the second sub-data is less than or equal to the first threshold.
19. A device for reading and writing data, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store instructions, and the processor is used to execute the instructions. When the processor executes the instructions, the method according to any one of claims 1 to 9 is executed.
20. A computer-readable storage medium, characterized in that: The invention comprises a program which, when being run on a computer, causes the computer to execute the method according to any one of claims 1 to 9.
21. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.