Data storage method and related equipment

Abstract

The embodiment of the invention discloses a data storage method and related equipment. The related equipment can comprise a data storage device, electronic equipment, a computer program product and a computer readable storage medium. According to the embodiment of the invention, after at least one piece of to-be-stored data and the data attribute information of the to-be-stored data are obtained, the data capacity of the to-be-stored data is identified in the data attribute information; the to-be-stored data is transferred to a second storage space of a second storage medium through a first storage space of the first storage medium, the writing speed of the first storage medium is larger than that of the second storage medium, and when the data capacity exceeds a preset data capacity threshold value, the to-be-stored data is written into the second storage space of the second storage medium; according to the scheme, the storage efficiency of data storage can be improved. The embodiment of the invention can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and the like.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a data storage method and related equipment. The related equipment may include a data storage device, an electronic device, a computer program product, and a computer-readable storage medium. Background Art

[0002] In recent years, with the rapid development of Internet technology, data storage has become increasingly convenient. For data such as I / O from various applications, current data storage methods can achieve data storage by writing data to the storage space (storage pool) of fast storage media (such as SSDs (solid-state storage media)) in the front end, and then transferring the data from the storage space of the fast storage media to the storage space (storage pool) of slower storage media (such as HDDs (mechanical storage media)).

[0003] During the research and practice of the prior art, the inventors of the present application discovered that the lifespan of fast storage media is relatively short, and that each IO operation needs to be transferred from an SSD to an HDD, which often occupies a large amount of network bandwidth and SSD bandwidth, thereby resulting in low storage efficiency of data storage. Summary of the Invention

[0004] An embodiment of the present application provides a data storage method and related equipment. The related equipment may include a data storage device, an electronic device, a computer program product, and a computer-readable storage medium, which can improve the storage efficiency of data storage.

[0005] A data storage method, comprising:

[0006] Acquire at least one piece of data to be stored and data attribute information of the data to be stored;

[0007] identifying a data capacity of the data to be stored in the data attribute information, wherein the data capacity indicates a data size of the data to be stored;

[0008] When the data capacity does not exceed a preset data capacity threshold, transferring the data to be stored from the first storage space of the first storage medium to the second storage space of the second storage medium, the writing speed of the first storage medium being greater than the writing speed of the second storage medium;

[0009] When the data capacity exceeds a preset data capacity threshold, the data to be stored is written into the second storage space of the second storage medium.

[0010] Accordingly, an embodiment of the present application provides a data storage device, including:

[0011] an acquiring unit, configured to acquire at least one piece of data to be stored and data attribute information of the data to be stored;

[0012] an identification unit, configured to identify a data capacity of the data to be stored in the data attribute information, wherein the data capacity indicates a data size of the data to be stored;

[0013] a first storage unit, configured to transfer the data to be stored from the first storage space of the first storage medium to the second storage space of the second storage medium when the data capacity does not exceed a preset data capacity threshold, wherein a write speed of the first storage medium is greater than a write speed of the second storage medium;

[0014] The second storage unit is configured to write the data to be stored into the second storage space of the second storage medium when the data capacity exceeds a preset data capacity threshold.

[0015] In some embodiments, the first storage unit can be specifically used to write the data to be stored into the first storage space of the first storage medium through the target cache space; write the data to be stored cached in the target cache space into the second storage space of the second storage medium; when all the data to be stored are stored in the second storage space, delete the data to be stored in the first storage space.

[0016] In some embodiments, the first storage unit can be specifically used to aggregate the data to be stored to obtain aggregated data to be stored; asynchronously write the aggregated data to be stored into the internal cache space corresponding to the second storage space; when it is detected that all the aggregated data to be stored are stored in the internal cache space, asynchronously write the aggregated data to be stored in the internal cache space into the second storage space.

[0017] In some embodiments, the first storage unit may be specifically configured to detect at least one stored data in the internal cache space to obtain current cache data; and delete the current cache data in the target cache space.

[0018] In some embodiments, the first storage unit can be specifically used to obtain the data storage record corresponding to the second storage space when an abnormality is detected; based on the data storage record, identify the data location of the stored data in the second storage space; and according to the data location, stop writing the data in the target cache space in the second storage space.

[0019] In some embodiments, the data storage device may further include a first reading unit, which may be specifically used to determine the data storage status of the data to be read according to the first data reading request when receiving a first data reading request for the data to be read; based on the data storage status, identify the data storage location of the data to be read in the target cache space, the internal cache space, the first storage space and the second storage space; and read the data to be read according to the data storage location.

[0020] In some embodiments, the first reading unit can be specifically used to determine that the data storage location of the data to be read is the target cache space when the data storage status is not asynchronously written to the internal cache space; when the data storage status is not asynchronously written to the second storage space, determine that the data storage location of the data to be read is the first storage space; when the data storage status is asynchronously written to the second storage space, determine that the data storage location of the data to be read is the second storage space.

[0021] In some embodiments, the second storage unit can be specifically used to generate at least one data index corresponding to the data to be stored according to the data capacity of the data to be stored; and based on the data index, write the data to be stored in parallel into the second storage space of the second storage medium.

[0022] In some embodiments, the second storage unit can be specifically used to divide the data to be stored into at least one data block according to the data capacity of the data to be stored, and generate a logical offset address corresponding to each data block based on the data attribute information; compress the data block to obtain a compressed data block, and determine the physical offset address of the compressed data block in the target data storage space; associate the logical offset address of each data block with the corresponding physical offset address of the compressed data block to obtain at least one data index of the data to be stored.

[0023] In some embodiments, the second storage unit can be specifically used to determine the metadata corresponding to the data to be stored based on the physical offset address, where the metadata indicates the location information of the data to be stored in the target data storage space; according to the data index, add an index header to the metadata and the data to be stored respectively; write the data to be stored with the index header added into the target data storage space, and write the metadata with the index header added into the original data storage space.

[0024] In some embodiments, the second storage unit can be specifically used to detect the persistence status of each data to be stored in the second storage space when there are multiple data to be stored written in parallel to the second storage space; based on the persistence status and data index, filter out at least one target data to be stored from the data to be stored, and delete the target data to be stored in the second storage space; restore the data index of the target data to be stored, and based on the restored data index, write the target data to be stored in parallel to the second storage space of the second storage medium.

[0025] In some embodiments, the second storage unit can be specifically used to, when detecting the presence of data to be stored that failed to be persisted and data to be stored that succeeded in persistence, sort the data to be stored according to the data index; based on the sorting result, screen out first candidate data to be stored from the data to be stored that failed to be persisted; based on the first candidate data to be stored and the sorting result, screen out second candidate data to be stored from the data to be stored that succeeded in persistence; based on the sorting result, use the data to be stored after the second candidate data to be stored in the data to be stored as the target data to be stored.

[0026] In some embodiments, the data storage device may further include a second reading unit, which may be specifically used to identify the original data reading interval corresponding to the second storage space in the second data reading request when a second data reading request is received; determine the target data storage interval corresponding to the original data reading interval based on the first data storage interval and the second data storage interval; and read the target data corresponding to the target data storage space in the second storage space to obtain the target reading data corresponding to the second data reading request.

[0027] In some embodiments, the second reading unit can be specifically used to identify the reading start position and the reading end position in the original data reading interval; based on the reading start position and the reading end position, screen out at least one candidate data storage interval that matches the data reading interval in the first data storage interval; and screen out a second data storage interval corresponding to the candidate data storage interval in the second data storage interval to obtain a target data storage interval.

[0028] In some embodiments, the second reading unit can be specifically used to identify the storage starting position and the storage end position in the first data storage interval, and determine the boundary position of the data index corresponding to the second storage space; compare the storage starting position with the reading starting position to obtain a first comparison result, and compare the storage end position with the reading end position to obtain a second comparison result; based on the first comparison result, the second comparison result and the boundary position, screen out at least one candidate data storage interval in the first data storage interval.

[0029] In some embodiments, the second reading unit can be specifically used to read the candidate data corresponding to the target data storage space in the second storage space, and strip the index header from the candidate data to obtain candidate read data; when the data type of the candidate read data is the metadata corresponding to the data to be stored, based on the metadata, the target read data is read in the original data storage space of the second storage space; when the data type of the candidate read data is the data to be stored, the candidate read data is used as the target read data.

[0030] In addition, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the data storage method provided in the embodiment of the present application.

[0031] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the steps in any one of the data storage methods provided in the embodiment of the present application.

[0032] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the data storage method provided in the embodiment of the present application.

[0033] In an embodiment of the present application, after obtaining at least one data to be stored and data attribute information of the data to be stored, the data capacity of the data to be stored is identified in the data attribute information. When the data capacity does not exceed a preset data capacity threshold, the data to be stored is transferred to the second storage space of the second storage medium through the first storage space of the first storage medium. The write speed of the first storage medium is greater than the write speed of the second storage medium. When the data capacity exceeds the preset data capacity threshold, the data to be stored is written into the second storage space of the second storage medium. This scheme can determine different data storage paths according to the size of the identified data to be stored. When the data capacity of the data to be stored exceeds the preset data capacity threshold, that is, when the data to be stored is large block data (IO), the first storage space of the first storage medium (fast storage medium) can be bypassed and directly written into the second storage space of the second storage medium (slow storage medium), thereby optimizing the life of the first storage medium. Moreover, for large block data, the bandwidth of the first storage medium and the network bandwidth can also be saved, thereby improving the storage efficiency of data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 This is a schematic diagram of a scenario of a data storage method provided in an embodiment of the present application;

[0036] Figure 2 This is a flow chart of a data storage method provided by an embodiment of the present application;

[0037] Figure 3 This is a schematic diagram of transferring data to be stored from a first storage space to a second storage space provided by an embodiment of the present application;

[0038] Figure 4 is a schematic diagram of reading data to be read corresponding to a first data read request provided in an embodiment of the present application;

[0039] Figure 5 Schematic diagram of the mapping relationship between the physical offset address and the logical offset address provided in the embodiment of the present application;

[0040] Figure 6 This is a schematic diagram of the IO writing process provided by an embodiment of the present application;

[0041] Figure 7 is a schematic diagram of the index recovery process provided by an embodiment of the present application;

[0042] Figure 8 is a schematic diagram of an index query provided by an embodiment of the present application;

[0043] Figure 9 This is a schematic diagram of the amplification problem in the index query provided by the embodiment of the present application;

[0044] Figure 10 2 is a schematic diagram of reading target read data corresponding to a second data read request provided by an embodiment of the present application;

[0045] Figure 11 This is a schematic diagram of the overall process of data storage provided by an embodiment of the present application;

[0046] Figure 12 This is another flowchart of the data storage method provided by an embodiment of the present application;

[0047] Figure 13 This is a schematic diagram of the structure of a data storage device provided in an embodiment of the present application;

[0048] Figure 14 This is another structural diagram of the data storage device provided in an embodiment of the present application;

[0049] Figure 15 This is another structural diagram of the data storage device provided in an embodiment of the present application;

[0050] Figure 16 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0052] The present invention provides a data storage method and related equipment, which may include a data storage device, an electronic device, a computer program product, and a computer-readable storage medium. The data storage device may be integrated into an electronic device, which may be a server, a terminal, or other device.

[0053] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not limit this.

[0054] For example, see Figure 1 Taking the data storage device integrated in an electronic device as an example, the electronic device can identify the data capacity of the data to be stored in the data attribute information after obtaining at least one data to be stored and data attribute information of the data to be stored. When the data capacity does not exceed the preset data capacity threshold, the data to be stored is transferred to the second storage space of the second storage medium through the first storage space of the first storage medium. The writing speed of the first storage medium is greater than the writing speed of the second storage medium. When the data capacity exceeds the preset data capacity threshold, the data to be stored is written into the second storage space of the second storage medium, thereby improving the storage efficiency of data storage.

[0055] The data storage method provided in the embodiments of the present application involves artificial intelligence (AI) technology. The embodiments of the present application can identify the data capacity in the data attribute information of the data to be stored. When the data capacity does not exceed a preset data capacity threshold, the data to be stored is transferred from the first storage space of the first storage medium to the second storage space of the second storage medium. When the data capacity exceeds the preset data capacity threshold, the data to be stored is written to the second storage space of the second storage medium, thereby improving the storage efficiency of data storage.

[0056] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0057] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0058] The embodiments of the present application may also involve cloud storage technology in cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or a local area network to achieve data computing, storage, processing and sharing.

[0059] Cloud technology, a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model, can form a resource pool for on-demand, flexible and convenient use. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark, requiring transmission to backend systems for logical processing. Data of different levels will be processed separately, and data from all industries will require strong system support, which can only be achieved through cloud computing.

[0060] Among them, cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0061] Currently, storage systems utilize a method for creating logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a specific storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID) of the data entity. The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can provide access to the data based on the storage location information of each object.

[0062] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0063] It can be understood that in the specific implementation of this application, when the following embodiments of this application are applied to specific products or technologies, such as the data to be stored of the object and the data attribute information of the data to be stored, permission or consent is required, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0064] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0065] This embodiment will be described from the perspective of a data storage device, which can be specifically integrated into an electronic device, which can be a server or a terminal; wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC), a wearable device, a virtual reality device or other smart devices that can perform data storage.

[0066] A data storage method, comprising:

[0067] Obtain at least one data to be stored and data attribute information of the data to be stored, identify the data capacity of the data to be stored in the data attribute information, and the data capacity indicates the data size of the data to be stored. When the data capacity does not exceed a preset data capacity threshold, transfer the data to be stored to the second storage space of the second storage medium through the first storage space of the first storage medium, and the write speed of the first storage medium is greater than the write speed of the second storage medium. When the data capacity exceeds the preset data capacity threshold, write the data to be stored into the second storage space of the second storage medium.

[0068] like Figure 2 As shown, the specific process of the data storage method is as follows:

[0069] 101. Obtain at least one piece of data to be stored and data attribute information of the data to be stored.

[0070] The data to be stored can be understood as data that needs to be stored, and the data here can include IO (input and output data) of various applications. The data to be stored can have various data types, for example, it can include raw data of various modalities, and it can also include raw data converted into computer-readable data. The so-called modality can be understood as a data type, for example, it can include text data, image data, audio and video data, or other types of data, etc.

[0071] The data attribute information may be understood as data representing the attributes of the data to be stored, for example, it may include the data identifier or object (Blob) of the data to be stored, and may also include relevant data indicating the size of the data to be stored.

[0072] There are multiple ways to obtain at least one data to be stored and data attribute information of the data to be stored, which can be specifically as follows:

[0073] For example, at least one data to be stored and data attribute information of the data to be stored uploaded by the terminal or client can be directly obtained, or at least one data to be stored uploaded by the terminal or client can be obtained and the data attribute information of the data to be stored can be identified in the data to be stored, or at least one data to be stored can be obtained on the network or data platform and the data attribute information can be identified in the data to be stored, or, when the data to be stored is large or the data capacity is large, a data storage request can also be received, which carries the data to be stored and the storage address of the data to be stored, and based on the storage address, at least one data to be stored and the data attribute information of the data to be stored can be obtained, and so on.

[0074] 102. Identify the data capacity of the data to be stored in the data attribute information.

[0075] The data capacity indicates the size of the data to be stored. The so-called data size can be understood as the size of the memory occupied by the data in the storage space. The unit of data size can be bytes.

[0076] There are multiple ways to identify the data capacity of the data to be stored in the data attribute information, which can be specifically as follows:

[0077] For example, the size of the data to be stored can be counted in the data attribute information to obtain the data capacity of the data to be stored, or the size of the data to be stored can be directly identified in the data attribute information to obtain the data capacity of the data to be stored, and so on.

[0078] 103. When the data capacity does not exceed a preset data capacity threshold, transfer the data to be stored from the first storage space of the first storage medium to the second storage space of the second storage medium.

[0079] The first storage medium has a faster write speed than the second storage medium. Therefore, the first storage medium can be considered a fast storage medium, and the second storage medium can be considered a slow storage medium. There are many types of fast storage media, such as SSDs or other storage media with write speeds exceeding those of mechanical disks. There are also many types of slow storage media, such as HDDs or other storage media with write speeds slower than those of SSDs.

[0080] When the data capacity does not exceed the preset data capacity threshold, there are multiple ways to transfer the data to be stored from the first storage space of the first storage medium to the second storage space of the second storage medium, which may be as follows:

[0081] For example, the data to be stored can be written into the first storage space of the first storage medium through the target cache space, and the data to be stored cached in the target cache space can be written into the second storage space of the second storage medium. When all the data to be stored are stored in the second storage space, the data to be stored is deleted in the first storage space.

[0082] The target cache space may be a storage space for caching data to be stored, and the target cache space may be a cache (cache memory) or other types of cache. There may be multiple ways to write the data to be stored into the first storage space of the first storage medium through the target cache space. For example, the data to be stored may be written into the target cache space, and then the data to be stored in the target cache space may be written into the first storage space of the first storage medium. Alternatively, the target cache space may be used as a transitional storage space, and the data to be stored may be transitionally stored into the first storage space of the first storage medium through the transitional storage space, and so on.

[0083] It should be noted that writing the data to be stored into the target cache space and writing the data cached in the target cache space into the first storage space of the first storage medium can be done in parallel. In addition, after the data to be stored is written into the first storage space of the first storage medium, the data to be stored is still cached in the target cache space.

[0084] After the data to be stored is written to the first storage space of the first storage medium through the target cache space, the data to be stored cached in the target cache space can be written to the second storage space of the second storage medium. There are many ways to write the data to be stored cached in the target cache space to the second storage space of the second storage medium. For example, the data to be stored can be aggregated to obtain aggregated data to be stored, and the aggregated data to be stored can be asynchronously written to the internal cache space corresponding to the second storage space. When it is detected that all the aggregated data to be stored are stored in the internal cache space, the aggregated data to be stored in the internal cache space can be asynchronously written to the second storage space.

[0085] There are many ways to aggregate the data to be stored. For example, the data to be stored may be converted into large blocks of buffer IO (aggregated data) through a first storage medium, thereby obtaining aggregated data to be stored.

[0086] It should be noted that, for example, if the first storage medium is an SSD and the second storage medium is an HDD, due to the physical characteristics of the HDD, the HDD's random IO performance is poor, and the SSD is needed to adjust the load to be HDD-friendly. Therefore, by converting the data to be stored into large buffer IO blocks, the aggregated data to be stored can be made load-friendly for the second storage space of the second storage medium.

[0087] After the data to be stored are aggregated, the aggregated data to be stored can be asynchronously written into the internal cache space corresponding to the second storage space. The so-called internal cache space can be a buffer corresponding to the second storage space.

[0088] When it is detected that all the aggregated data to be stored are stored in the internal cache space, the aggregated data to be stored in the internal cache space can be asynchronously written to the second storage space. There are many ways to asynchronously write the aggregated data to be stored in the internal cache space to the second storage space. For example, the data identifier of the data to be stored can be identified in the data attribute information, and based on the data identifier, a target data index of each data to be stored in the aggregated data to be stored is generated. According to the target data index, the aggregated data to be stored in the internal cache space is written to the second storage space through sync (a writing method).

[0089] Taking the first storage space as SSD space and the second storage space as HDD space as an example, the process of transferring the data to be stored from the first storage space to the second storage space can be as follows: Figure 3 As shown, the foreground writes the data to be stored into the SSD space through the target cache space (cache). If the data is transferred from the SSD space to the HDD space, the conventional transfer often requires the operation of reading the data to be stored in the SSD space. In order to avoid reading data from the SSD space for transfer every time, this solution writes the data to be stored in the SSD into the HDD space asynchronously through the target cache space (cache). After the data is written to the SSD space and stored on the disk, it can return an acknowledgment (ack) to the storage object, and the background task will continuously asynchronously protocol the data to be stored in the cache to the HDD space. The process of asynchronously writing (applying) the data to be stored in the cache to the HDD space can be regarded as two stages. The first stage can be understood as asynchronously writing from the target cache space to the internal storage space (buffer) corresponding to the HDD space, and the second stage can be understood as syncing from the buffer corresponding to the HDD space to the HDD space (disk medium).

[0090] Optionally, to address the issue of space occupancy in the target cache space, after the data to be stored is aggregated to obtain the aggregated data to be stored, at least one data item already stored in the internal cache space can be detected to obtain the current cached data, and the current cached data can be deleted from the target cache space. In this way, the data cached in the target cache space can be released.

[0091] Optionally, in some embodiments, if an abnormal situation such as power failure occurs in the background after the data to be stored is written to the first storage space of the first storage medium through the target cache space, the abnormality can also be handled. There can be multiple ways to handle the abnormality. For example, when an abnormality is detected, the data storage record corresponding to the second storage space is obtained, and based on the data storage record, the data location of the stored data is identified in the second storage space, and according to the data location, the writing of the data in the target cache space in the second storage space is stopped.

[0092] The data location can be understood as the location corresponding to the data that has been persisted. Based on the data location, there are various ways to stop writing data from the second storage space to the target cache. For example, based on the data location, data that has not yet been persisted in the second storage space can be identified in the target cache space. In this case, it is necessary to stop writing the data that has not yet been persisted to the second storage space of the second storage medium.

[0093] Optionally, in some embodiments, there may be multiple ways to read the data to be stored whose data capacity is less than a preset data capacity threshold. For example, when a first data read request for the data to be read is received, the data storage status of the data to be read can be determined based on the first data read request, and based on the data storage status, the data storage location of the data to be read can be identified in the target cache space, the internal cache space, the first storage space, and the second storage space, and the data to be read can be read based on the data storage location.

[0094] The data storage status can be understood as the storage status of the data to be read in the first storage space and the second storage space. The data storage status may include not being written to the first storage space, being written to the first storage space, not being asynchronously written to the internal cache space, being written to the internal cache space, not being asynchronously written to the second storage space, and being written to the second storage space, etc. There are various ways to determine the data storage status of the data to be read based on the first data read request. For example, a data identifier of the data to be stored can be identified in the first data read request. Based on the data identifier, the data to be read can be queried in the internal cache space and the second storage space. The data storage status of the data to be read can be determined based on the query result.

[0095] After determining the data storage status of the data to be read, there are many ways to identify the data storage location of the data to be read in the target cache space, the internal cache space, the first storage space and the second storage space. For example, when the data storage status is not asynchronously written to the internal cache space, the data storage location of the data to be read is determined to be the target cache space; when the data storage status is not asynchronously written to the second storage space, the data storage location of the data to be read is determined to be the first storage space; when the data storage status is asynchronously written to the second storage space, the data storage location of the data to be read is determined to be the second storage space.

[0096] After the data storage location of the data to be read is identified in the target cache space, the internal cache space, the first storage space, and the second storage space, the data to be read can be read according to the data storage location. There are multiple ways to read the data to be read according to the data storage location. For example, when the data storage location is the target cache space, the data to be read is read in the target cache space based on the data identifier of the data to be read. When the data storage location is the first storage space, the data to be read can be read in the first storage space based on the data identifier of the data to be read. When the data storage location is the second storage space, the data to be read can be read in the second storage space based on the data identifier of the data to be read, and so on.

[0097] The first storage space is the SSD space, the second storage space is the HDD space, and the target cache space is the cache. After writing the data to the SSD space, the data is transferred from the SSD space to the HDD space through two-stage asynchronous writing (apply). During this transfer or writing process, when a first data read request is received, the process of reading the data to be read corresponding to the first data read request can be as follows: Figure 4 As shown, it is necessary to judge the data status of the data to be read. When the data to be read has not been asynchronously written to the internal cache space (that is, the buffer corresponding to the HDD space), the data to be read can be directly read in the cache. When the data to be read has completed the first stage of apply, that is, the data to be read has been asynchronously written to the buffer, but has not been asynchronously written to the HDD space, the data to be read can be directly read in the SSD space. When the data to be read has completed the second stage of apply, that is, the data to be read has been asynchronously written to the HDD space, the data to be read can be directly read in the HDD space.

[0098] 104. When the data capacity exceeds a preset data capacity threshold, write the data to be stored into the second storage space of the second storage medium.

[0099] For example, when the data capacity exceeds a preset data capacity threshold, at least one data index corresponding to the data to be stored can be generated according to the data capacity of the data to be stored, and based on the data index, the data to be stored can be written in parallel to the second storage space of the second storage medium.

[0100] The second storage space may include an original data storage space and a target data storage space. For example, if the data to be stored does not exceed a preset data capacity threshold as small-block data, and the data to be stored exceeds a preset data capacity threshold as large-block data, the original data storage space is used to store the location information of the small-block data and the large-block data in the target storage space (i.e., metadata corresponding to the large-block data), and the target data storage space is used to store the large-block data. Therefore, in the second storage space, the target data storage space is introduced so that the large-block data and the small-block data can be stored separately, thereby facilitating the queueing of the large-block data for processing.

[0101] Among them, the data index can be understood as a set of kv (key-value pairs) between the logical offset address (logicaloffset) of the data to be stored and the physical offset address (physical offset) where the data to be stored is actually stored in the second storage space. According to the data capacity of the data to be stored, there can be multiple ways to generate at least one data index corresponding to the data to be stored. For example, according to the data capacity of the data to be stored, the data to be stored can be divided into at least one data block, and based on the data attribute information, the logical offset address corresponding to each data block is generated, the data block is compressed to obtain the compressed data block, and the physical offset address of the compressed data block in the original data storage space is determined, and the logical offset address of each data block is associated with the corresponding physical offset address of the compressed data block to obtain at least one data index of the data to be stored.

[0102] There are multiple ways to divide the data to be stored into at least one data block based on the data capacity of the data to be stored. For example, the data to be stored can be divided into at least one data block of a preset data capacity based on the data capacity of the data to be stored. Alternatively, the number of indexes of the data index can be determined based on the data capacity of the data to be stored, and the data to be stored can be divided into data blocks corresponding to the number of indexes. For example, if the data capacity of the data to be stored is 2MB, the data to be stored can be divided into multiple data blocks of 64KB each. It should be noted that the preset data capacity can be set according to actual application.

[0103] After the data to be stored is divided into at least one data block, a logical offset address corresponding to each data block can be generated based on the data attribute information. There are various ways to generate the logical offset address corresponding to each data block. For example, the logical offset address corresponding to each data block can be identified in the data attribute information. Alternatively, a data identifier of the data to be stored can be identified in the data attribute information, a preset logical offset address corresponding to the data identifier can be obtained, and the logical offset address corresponding to each data block can be filtered out from the preset logical offset addresses.

[0104] There are many ways to compress the data block. For example, the data in the data block can be directly compressed to obtain a compressed data block, or at least one compressible data block can be screened out from the data block to obtain a data block to be compressed, and the data to be compressed can be compressed to obtain a compressed data block.

[0105] It should be noted that, during the process of compressing data blocks, some data blocks can be compressed while some data blocks cannot be compressed, and compressed data blocks may include compressed data blocks and non-compressible data blocks.

[0106] After compressing a data block, the physical offset address of the compressed data block in the original data storage space can be determined. There are various ways to determine the physical offset address of the compressed data block in the original data storage space. For example, current storage information of the original data storage space can be obtained, the storage address to be allocated can be identified from the current storage information, and the physical offset address of each compressed data block can be selected from the storage addresses to be allocated based on the data capacity of the compressed data block.

[0107] After determining the physical offset address of the compressed data block in the original data storage space, the logical offset address of each data block can be associated with the corresponding physical offset address of the compressed data block to obtain at least one data index of the data to be stored.

[0108] It should be noted that the physical offset address field is smaller than the corresponding logical offset address. Taking the persistence structure as [300,100][100,100][300,150][200,150] and the memory structure as [0,0][300,100][400,200][700,350][900,500] as an example, the mapping relationship between the physical offset address and the logical offset address can be as follows: Figure 5 As shown, in Figure 5 It can be found that the physical offset address corresponding to the logical offset address [0,300] can be [0,100], and the data block is compressed at this time; the physical offset address corresponding to the logical offset address [300,400] can be [100,200]. At this time, it can be found that this data block may not be compressed, and so on, thereby further reducing the size of the mapping field, that is, the mapping field of the physical offset address can be smaller than the mapping field of the logical offset address.

[0109] After generating at least one data index corresponding to the data to be stored, the data to be stored can be written in parallel to the second storage space of the second storage medium based on the data index. There are multiple ways to write the data to be stored in parallel to the second storage space of the second storage medium based on the data index. For example, metadata corresponding to the data to be stored can be determined based on a physical offset address, and index headers can be added to the metadata and the data to be stored respectively according to the data index. The data to be stored with the index header added is written to the target data storage space, and the metadata with the index header added is written to the original data storage space.

[0110] Among them, there are many ways to write the metadata of the added index header into the original data storage space. For example, the metadata of the added index header can be directly written into the original data storage space, or the metadata of the added index header can be transferred to the original data storage space of the second storage space through the first storage space, and so on.

[0111] Among them, taking the data to be stored as IO as an example, for small IO whose data capacity is less than the preset data capacity threshold, the index header can be directly added (pin), and the original data storage space of the second storage space can be directly written through the buffer. For large IO whose data capacity is greater than the preset data capacity threshold, the original data storage space and the metadata storage space can be written concurrently to write the metadata (meta IO) of the data to be stored in the original data storage space (SlowBlob). The content of meta IO records the location information of the large IO in the target data storage space (BigIOBlob). In the process of writing large IO, it is necessary to wait until both IOs (large IO and meta IO corresponding to large IO) are successful, and then the confirmation information of the successful writing of large IO can be returned. The process of IO writing can be as follows Figure 6 As shown, Blob can be understood as an object of data storage, that is, an object or identifier of data to be stored, SlowBlob can be understood as an object of the second storage medium where data is stored, which can refer to an object in the original data storage space in the second storage space, and BigIOBlob can be understood as an object of large blocks of data stored in the second storage medium, which can refer to an object in the target data storage space in the second storage space. For example, the data to be stored may include two small IOs S1 and S2, and two large IOs B1 and B2. S1 and S2 can be directly written into SlowBlob. In addition, the metadata of the two large IOs B1M and B2M can be written into SlowBlob, and the two large IOs B1 and B2 can be directly written into BigIOBlob.

[0112] Among them, if each data (IO) adds a persistence operation based on the data index in series, it is not acceptable for data storage. Therefore, the persistence operation based on the index can be processed in parallel, that is, the data to be stored can be written in parallel to the second storage space of the second storage medium based on the data index. In the case of parallel writing (i.e. persistence operation), after the data to be stored is written in parallel to the second storage space of the second storage medium based on the data index, index recovery can also be performed. The process of index recovery may include: when it is detected that there are multiple data to be stored written in parallel to the second storage space, the persistence state of each data to be stored is detected in the second storage space, based on the persistence state and the data index, at least one target data to be stored is screened out from the data to be stored, and the target data to be stored is deleted in the second storage data space, the data index of the target data to be stored is restored, and based on the restored data index, the target data to be stored is written in parallel to the second storage space of the second storage medium.

[0113] Among them, the persistence state is used to indicate the completion state or completion result of the persistence of the data to be stored in the second storage space, and may include one of persistence success or persistence failure. The so-called persistence success can be understood as the data to be stored has been successfully stored in the second storage space, and persistence failure can be understood as the data to be stored has not been successfully stored in the second storage space. Based on the persistence state and the data index, there can be multiple ways to filter out at least one target data to be stored in the data to be stored. For example, when it is detected that there are data to be stored that has failed to be persisted and data to be stored that has successfully been persisted, the data to be stored are sorted according to the data index, and based on the sorting result, the first candidate data to be stored is filtered out from the data to be stored that has failed to be persisted, and based on the second candidate data to be stored and the sorting result, the second candidate data to be stored is filtered out from the data to be stored that has successfully been persisted, and based on the sorting result, the data to be stored after the second candidate data to be stored in the data to be stored is used as the target data to be stored.

[0114] There are various ways to select the first candidate data to be stored from the data to be stored that failed persistence based on the sorting result. For example, based on the sorting result, the data to be stored that is ranked first among the data to be stored that failed persistence can be selected as the first candidate data to be stored. For example, if the data to be stored includes data A, B, C, and D to be stored, and the sorting result is ABCD, and data B and C to be stored are data to be stored that failed persistence, then the first candidate data to be stored is data B to be stored.

[0115] After the first candidate data to be stored is screened out from the data to be stored that failed to be persisted, the second candidate data to be stored can be screened out from the data to be stored that succeeded in persistence based on the first candidate data to be stored and the sorting result. There are many ways to screen out the second candidate data to be stored from the data to be stored that succeeded in persistence. For example, based on the sorting result, at least one adjacent data to be stored adjacent to the first candidate data to be stored can be screened out from the data to be stored that succeeded in persistence. According to the sorting result, the data to be stored with the highest sorting position can be screened out from the adjacent data to be stored as the second candidate data to be stored. For example, the data to be stored includes data A, B, C and D to be stored, and the sorting result is ABCD. For example, the data to be stored B and C are the data to be stored that failed to be persisted. At this time, the first candidate data to be stored is data B to be stored, and the data to be stored adjacent to data B to be stored that succeeded in persistence can be data A and D to be stored. According to the sorting result, data A to be stored can be used as the second candidate data to be stored.

[0116] After the second candidate data to be stored is screened out from the data to be stored that has been persisted successfully, the data to be stored after the second candidate data to be stored in the data to be stored can be used as the target data to be stored based on the sorting result. For example, if the data to be stored includes data A, B, C and D to be stored, and the second candidate data to be stored is data A to be stored, then the data B, C and D to be stored can be used as the target data to be stored. The target data to be stored here can be understood as the data to be stored that needs to be treated as data to be stored that has failed to be persisted. Although there may be data to be stored that has been persisted successfully among the data to be stored B, C and D, due to the existence of data to be stored that has failed to be persisted in the interval, these successfully persisted data may not be credible and need to be indexed and rewritten.

[0117] After selecting at least one target data to be stored from the data to be stored, the target data to be stored can be deleted from the second storage space. Then, the data index of the target data to be stored is restored, and based on the restored data index, the target data to be stored is written in parallel to the second storage space of the second storage medium. There are many ways to restore the data index of the target data to be stored. For example, based on the data capacity of the target data to be stored, at least one target data index of the target data to be stored can be generated, and the data index of the target data to be stored can be updated based on the target data index, and the updated data index is used as the restored data index.

[0118] The index recovery process can be as follows: Figure 7 As shown, the mapping relationship between the physical offset address and the logical offset address is as follows Figure 5 For example, if the persistence status of [200,350] and [350,500] in the persistence structure is persistence failure, then checkpoint (a data storage detection mechanism) can be used to detect that the indexes that need to be restored are [200,350] and [350,500]. The corresponding memory results can be [400,700] and [700,900]. Recovery (a recovery mechanism in data storage) is used to restore the data indexes that need to be restored, and the data is rewritten into the original data storage space and the target data storage space through the restored data indexes.

[0119] Optionally, the logical offset address in the data index includes the first data storage interval, and the physical offset address includes the second data storage interval. In some embodiments, there may be multiple ways to read the data stored in the second storage space. For example, when a second data read request is received, an original data read interval corresponding to the second storage space is identified in the second data read request, a target data storage interval corresponding to the original data read interval is determined based on the first data storage interval and the second data storage interval, and target data corresponding to the target data storage space is read from the second storage space to obtain target read data corresponding to the second data read request.

[0120] Among them, the original data reading interval can be understood as a range, which can indicate at least one data stored in the second storage space. Each data in the data index can correspond to a storage interval. The logical offset address in the data index includes the first data storage interval, and the physical offset address includes the second data storage interval. According to the first data storage interval and the second data storage interval, there can be many ways to determine the target data storage interval corresponding to the original data reading interval. For example, the reading start position and the reading end position can be identified in the original data reading interval. Based on the reading start position and the reading end position, at least one candidate data storage interval that matches the data reading interval is screened out in the first data storage space, and the second data storage interval corresponding to the candidate data interval is screened out in the second data storage interval to obtain the target data storage interval.

[0121] Among them, there can be multiple ways to screen out at least one candidate data storage interval that matches the data reading interval in the first data storage interval based on the reading starting point position and the reading ending point position. For example, the storage starting point position and the storage ending point position can be identified in the first data storage interval, and the boundary position of the data index corresponding to the second storage space can be determined. The storage starting point position is compared with the reading starting point position to obtain a first comparison result, and the storage ending point position is compared with the reading ending point to obtain a second comparison result. According to the first comparison result, the second comparison result and the boundary position, at least one candidate data storage interval is screened out in the first data storage interval.

[0122] There are various ways to select at least one candidate data storage interval from the first data storage interval based on the first comparison result, the second comparison result, and the boundary position. For example, based on the first comparison result and the boundary position, a target storage starting position that is less than or equal to the read starting position can be selected from the storage starting positions. Based on the second comparison result and the boundary value, a target read end position that is greater than or equal to the read end position can be selected from the storage starting positions. Based on the target storage starting position and the target read end position, at least one candidate data storage interval can be selected from the first data storage interval. Alternatively, at least one data storage space whose storage starting position is greater than the read starting position and whose storage end position is less than the read end position can be selected from the first data storage interval to obtain the first candidate data storage interval. At least one data storage interval whose storage starting position is less than the read starting position and whose storage end position is less than the read end position can be selected from the first data storage interval to obtain the second candidate data storage interval. Based on the first data storage interval, a storage starting position greater than the read starting position and whose storage end position is greater than the read end position can be selected to obtain the third candidate data storage interval. The first candidate data storage interval, the second candidate data storage interval, and the third candidate data storage interval can be used as candidate data storage intervals, and so on.

[0123] After selecting at least one candidate data storage interval that matches the data reading interval from the first data storage interval, a second data storage interval corresponding to the candidate data storage interval may be selected from the second data storage interval to obtain a target data storage interval.

[0124] The process of determining the target data storage interval corresponding to the original data reading interval can be regarded as the process of index query. The process of index query can be as follows: Figure 8 As shown, taking the input original data reading range (range) of [50,600] as an example, the upper-bound (perform greater than operation) and the lower-bound (perform greater than or equal to operation) on the left side can be performed respectively, so that the candidate data storage range [0,700] can be obtained. The logical offset address corresponding to the candidate data storage range here can be used to query the target data storage range [0,350] of the physical offset address corresponding to the logical offset address through the mapping relationship of the data index.

[0125] Among them, during the index query process, it can be found that there is an amplification problem in the index query process, such as Figure 9As shown, a large IO is 2MB when written. If only one index is recorded and only 4KB of data needs to be read, 2MB of data will be read in the index query. Therefore, in this solution, when generating the data index, at least one data index is generated according to the data capacity of the data to be stored, that is, by splitting the data index of a large IO into multiple data indexes, for example, generating a data index according to 64KB. At this time, when reading 4KB of data, it only needs to be enlarged to 64KB, which greatly reduces the extent of data amplification, thereby saving computing resources and network loans, and improving the storage efficiency of data storage.

[0126] After determining the target data storage interval corresponding to the original data read interval based on the first data storage interval and the second data storage interval, the target data corresponding to the target data storage space can be read out in the second storage space to obtain the target read data corresponding to the second data read request. There are many ways to read the target data corresponding to the target data storage space in the second storage space. For example, the target data corresponding to the target data storage interval can be read out in the second storage space, and the index header can be stripped out of the candidate data to obtain the candidate read data. When the data type of the candidate read data is the metadata corresponding to the data to be stored, the target read data is read out in the original data storage space of the second storage space based on the metadata. When the data type of the candidate data is the data to be stored, the candidate read data is used as the target read data.

[0127] In the process of reading the target data, it is necessary to trigger a second read based on the data type of the candidate data read out for the first time. If the candidate data read out is metadata (mate IO), it is necessary to trigger a second read and read out the target data in the original data storage space. If the type of the candidate data read out is data to be stored (i.e. normal data), the candidate data can be directly used as the target data. The process of reading the target data can be as follows: Figure 10 As shown, taking the target data storage interval of [0,350] as an example, when data is read in the original data storage space, the candidate data read out can be metadata. At this time, it is necessary to trigger a secondary read and read the data corresponding to [0,700] in the target data storage space through the metadata to obtain the target read data. When data is read in the target data storage space, the read data can be used as the target read data.

[0128] For example, if the data to be stored is IO, the first storage space is SSD space, and the second storage space is HDD space, this solution can automatically plan the storage path by sensing the load and identifying the IO size, so as to take into account both large and small IO processing scenarios. The overall process of data storage can be as follows: Figure 11As shown, a and Bm are retired IOs, c, e, and f are small IOs, and B and D are large-block IOs. Bm and Dm are the metadata for the large-block IOs. After obtaining at least one data blob to be stored, the data capacity of the blob is identified. A two-stage asynchronous write is used for small IOs, while a large-block write-through is used for large IOs. After determining the storage path, an index map is used to allocate data indexes and write the IOs into the cache. Small IOs in the cache are written to the SSD space (FastBlob). After writing, the small IOs are asynchronously written to the original data storage space (SlowBlob) in the HDD space using the corresponding buffer. Large IOs in the cache are written to the target data storage space (BigIOBlob) in the HDD space. Metadata corresponding to the storage location of the large IO in the BigIOBlob can also be written to the original data storage space (SlowBlob) in the HDD space. During the write process, the metadata can be transferred to the SlowBlob via FastBlob. For small IO, once it is successfully written to SlowBlob, this small IO can be eliminated and deleted from the cache. For large IO, once it is successfully written to BigIOBlob, this large (block) IO can be eliminated and deleted from the cache.

[0129] This solution maps a single blob into two logical blobs during the storage of large (block) I / O operations, named SlowBlob and BigIOBlob. The introduction of BigIOBlob allows for parallel processing of large and small I / O operations. Furthermore, the two types of I / O use different storage spaces, making it easier to queue large (block) I / O operations. Therefore, in applications with a high concentration of large I / O, network and SSD bandwidth can be saved, while also optimizing SSD lifespan.

[0130] From the above, it can be seen that after obtaining at least one data to be stored and data attribute information of the data to be stored, the embodiment of the present application identifies the data capacity of the data to be stored in the data attribute information. When the data capacity does not exceed the preset data capacity threshold, the data to be stored is transferred to the second storage space of the second storage medium through the first storage space of the first storage medium. The write speed of the first storage medium is greater than the write speed of the second storage medium. When the data capacity exceeds the preset data capacity threshold, the data to be stored is written to the second storage space of the second storage medium. The scheme can determine different data storage paths according to the size of the identified data to be stored. When the data capacity of the data to be stored exceeds the preset data capacity threshold, that is, when the data to be stored is large block data (IO), it can bypass the first storage space of the first storage medium (fast storage medium) and directly write to the second storage space of the second storage medium (slow storage medium), thereby optimizing the life of the first storage medium. Moreover, for large block data, the bandwidth of the first storage medium and the network bandwidth can also be saved, thereby improving the storage efficiency of data storage.

[0131] The method described in the above embodiment will be further described in detail below with examples.

[0132] In this embodiment, the data storage device is specifically integrated into an electronic device, the electronic device is a server, the first storage medium is an SSD, the first storage space is an SSD space, the second storage medium is an HDD, the second storage space is an HDD space, the data is IO, the data to be stored is IO to be stored, the target cache space is cache, the internal cache space is buffer, the original data storage space is SlowBlob, and the target data storage space is BigIOBlob as an example for explanation.

[0133] like Figure 12 As shown, a data storage method, the specific process is as follows:

[0134] 201. The server obtains at least one IO to be stored and data attribute information of the IO to be stored.

[0135] For example, the server can directly obtain at least one IO to be stored and data attribute information of the IO to be stored uploaded by the terminal or client, or it can obtain at least one IO to be stored uploaded by the terminal or client and identify the data attribute information of the IO to be stored in the IO to be stored, or obtain at least one IO to be stored on the network or data platform and identify the data attribute information in the IO to be stored, or, when the data of the IO to be stored is large or the data capacity is large, it can also receive a data storage request, which carries the storage address of the IO to be stored and the IO to be stored, and based on the storage address, obtain at least one IO to be stored and the data attribute information of the IO to be stored, and so on.

[0136] 202. The server identifies the data capacity of the IO to be stored in the data attribute information.

[0137] For example, the server can count the size of the IO to be stored in the data attribute information to obtain the data capacity of the IO to be stored, or it can directly identify the size of the IO to be stored in the data attribute information to obtain the data capacity of the IO to be stored, and so on.

[0138] 203. When the data capacity does not exceed the preset data capacity threshold, the server writes the storage IO to the SSD space through the cache.

[0139] For example, when the data capacity does not exceed the preset data capacity threshold, the server can write the IO to be stored into the cache, and then write the IO to be stored in the cache into the SSD space of the SSD. Alternatively, the cache can be used as a transitional storage space to transition the IO to be stored to the SSD space of the SSD through the transitional storage space, and so on.

[0140] 204. The server writes the IO to be stored cached in the cache to the HDD space of the HDD.

[0141] For example, the server can convert the IO to be stored into a large block of buffer IO through SSD, thereby obtaining the aggregated IO to be stored. The aggregated IO to be stored is asynchronously written to the buffer corresponding to the HDD space. When it is detected that all the IO to be stored after aggregation is stored in the buffer, the data identifier of the IO to be stored is identified in the data attribute information, and based on the data identifier, the target data index of each IO to be stored in the aggregated IO to be stored is generated. According to the target data index, the aggregated IO to be stored in the buffer is written to the HDD space through sync. When it is detected that all the IO to be stored after aggregation is stored in the buffer, the aggregated IO to be stored in the buffer is asynchronously written to the HDD space.

[0142] Optionally, in some embodiments, after the server aggregates the IOs to be stored to obtain aggregated IOs to be stored, it detects at least one stored IO in the buffer, obtains the current cached IO, and deletes the current cached IO from the cache.

[0143] Optionally, in some embodiments, when an abnormality is detected, the server obtains the data storage record corresponding to the HDD space, and based on the data storage record, identifies the data location of the stored data in the HDD space. According to the data location, the data that has not yet been persisted in the HDD space can be identified in the cache. At this time, it is necessary to stop writing the data that has not yet been persisted to the HDD space of the HDD.

[0144] Optionally, in some embodiments, when a first data read request for an IO to be read is received, the server identifies the data identifier of the IO to be stored in the first data read request, queries the IO to be read in the buffer and HDD space based on the data identifier, and determines the data storage status of the IO to be read based on the query result. When the data storage status is not asynchronously written to the buffer, the data storage location of the IO to be read is determined to be cache. When the data storage status is not asynchronously written to the HDD space, the data storage location of the IO to be read is determined to be SSD space. When the data storage status is asynchronously written to the HDD space, the data storage location of the IO to be read is determined to be HDD space. When the data storage location is cache, the IO to be read is read from the cache based on the data identifier of the IO to be read. When the data storage location is SSD space, the IO to be read can be read from the SSD space based on the data identifier of the IO to be read. When the data storage location is HDD space, the IO to be read can be read from the HDD space based on the data identifier of the IO to be read, and so on.

[0145] 205. When the data capacity exceeds a preset data capacity threshold, the server generates at least one data index corresponding to the IO to be stored according to the data capacity of the IO to be stored.

[0146] For example, the server divides the IO to be stored into at least one data block of a preset data capacity according to the data capacity of the IO to be stored, or it can also determine the number of indexes of the data index based on the data capacity of the IO to be stored and divide the IO to be stored into data blocks corresponding to the number of indexes.

[0147] The server can identify the logical offset address corresponding to each data block in the data attribute information, or it can also identify the data identifier of the IO to be stored in the data attribute information, obtain the preset logical offset address corresponding to the data identifier, and filter out the logical offset address corresponding to each data block in the preset logical offset address, and so on.

[0148] The server may directly compress the data in the data block to obtain a compressed data block, or may filter out at least one compressible data block from the data block to obtain a data block to be compressed, and compress the data to be compressed to obtain a compressed data block.

[0149] The server may obtain the current storage information of the SlowBlob, identify the storage address to be allocated in the current storage information, and filter out the physical offset address of each compressed data block in the storage address to be allocated according to the data capacity of the compressed data block.

[0150] The server associates the logical offset address of each data block with the physical offset address of the corresponding compressed data block to obtain at least one data index of the IO to be stored.

[0151] 206. The server writes the IO to be stored to the HDD space of the HDD in parallel based on the data index.

[0152] For example, the server can determine the metadata corresponding to the IO to be stored based on the physical offset address, add index headers to the metadata and the IO to be stored according to the data index, write the IO to be stored with the added index header into BigIOBlob, and write the metadata with the added index header into SlowBlob.

[0153] Optionally, in some embodiments, when the server detects that there are multiple IOs to be stored written to the HDD space in parallel, it detects the persistence status of each IO to be stored in the HDD space. When it detects that there are IOs to be stored that have failed to be persisted and IOs to be stored that have succeeded in persistence, it sorts the IOs to be stored according to the data index. Based on the sorting result, the IO to be stored with the highest sorting position among the IOs to be stored that have failed to be persisted is selected as the first candidate IO to be stored. Based on the sorting result, at least one adjacent IO to be stored adjacent to the first candidate IO to be stored is selected from the IOs to be stored that have succeeded in persistence. Based on the sorting result, the IO to be stored with the highest sorting position among the adjacent IOs to be stored is selected as the second candidate IO to be stored. Based on the sorting result, the IO to be stored after the second candidate IO to be stored in the IO to be stored is selected as the target IO to be stored. The target IO to be stored is deleted from the HDD space.

[0154] The server may generate at least one target data index for the target IO to be stored based on the data capacity of the target IO to be stored, update the data index of the target IO to be stored based on the target data index, and use the updated data index as the recovered data index. Based on the recovered data index, the target IO to be stored may be written in parallel to the HDD space of the HDD.

[0155] Optionally, in some embodiments, upon receiving the second data read request, the server identifies the original data read interval corresponding to the HDD space in the second data read request, identifies the read start position and the read end position in the original data read interval, identifies the storage start position and the storage end position in the first data storage interval, determines the boundary position of the data index corresponding to the HDD space, compares the storage start position with the read start position to obtain a first comparison result, and compares the storage end position with the read end position to obtain a second comparison result.

[0156] The server can filter out a target storage starting position that is less than or equal to the reading starting position from the storage starting position based on the first comparison result and the boundary position, and filter out a target reading end position that is greater than or equal to the reading end position from the storage starting position based on the second comparison result and the boundary value. Based on the target storage starting position and the target reading end position, the server can filter out at least one candidate data storage interval from the first data storage interval. Alternatively, the server can filter out at least one data storage space in the first data storage interval whose storage starting position is greater than the reading starting position and whose storage end position is less than the reading end position to obtain a first candidate data storage interval, filter out at least one data storage interval in the first data storage interval whose storage starting position is less than the reading starting position and whose storage end position is less than the reading terminal position to obtain a second candidate data storage interval, filter out in the first data storage interval whose storage starting position is greater than the reading starting position and whose storage end position is greater than the reading end position to obtain a third candidate data storage interval, and use the first candidate data storage interval, the second candidate data storage interval, and the third candidate data storage interval as candidate data storage intervals, and so on.

[0157] The server selects a second data storage interval corresponding to the candidate data storage interval from the second data storage interval to obtain a target data storage interval. The server reads the target data corresponding to the target data storage interval from the HDD space, and strips the index header from the candidate data to obtain a candidate read block. When the data type of the candidate read data is metadata corresponding to the to-be-stored IO, the server reads the target read data from the SlowBlob in the HDD space based on the metadata. When the data type of the candidate data is the to-be-stored IO, the server uses the candidate read data as the target read data.

[0158] From the above, it can be seen that after the server of the embodiment of the present application obtains at least one IO to be stored and data attribute information of the IO to be stored, it identifies the data capacity of the IO to be stored in the data attribute information. When the data capacity does not exceed the preset data capacity threshold, the IO to be stored is transferred to the HDD space of the HDD through the SSD space of the SSD. The write speed of the SSD is greater than the write speed of the HDD. When the data capacity exceeds the preset data capacity threshold, the IO to be stored is written to the HDD space of the HDD. This scheme can determine different data storage paths according to the size of the identified IO to be stored. When the data capacity of the IO to be stored exceeds the preset data capacity threshold, that is, when the IO to be stored is a large block of data (IO), it can bypass the SSD space of the SSD (fast storage medium) and be directly written to the HDD space of the HDD (slow storage medium), thereby optimizing the life of the SSD. Moreover, for large block data, the bandwidth of the SSD and the network bandwidth can also be saved, thereby improving the storage efficiency of data storage.

[0159] In order to better implement the above method, an embodiment of the present application also provides a data storage device, which can be integrated into an electronic device, such as a server or terminal, and the terminal can include a tablet computer, a laptop computer and / or a personal computer, etc.

[0160] For example, Figure 13 As shown, the data storage device may include an acquisition unit 301, an identification unit 302, a first storage unit 303, and a second storage unit 304, as follows:

[0161] (1) Acquisition unit 301;

[0162] The acquiring unit 301 is configured to acquire at least one piece of data to be stored and data attribute information of the data to be stored.

[0163] For example, the acquisition unit 301 can be specifically used to acquire at least one data to be stored and data attribute information of the data to be stored uploaded by the terminal or client, or to acquire at least one data to be stored uploaded by the terminal or client, and identify the data attribute information of the data to be stored in the data to be stored, or to acquire at least one data to be stored on the network or data platform, and identify the data attribute information in the data to be stored, or, when the amount of data to be stored is large or the data capacity is large, a data storage request can also be received, which carries the data to be stored and the storage address of the data to be stored, and based on the storage address, at least one data to be stored and the data attribute information of the data to be stored are acquired, and so on.

[0164] (2) Identification unit 302;

[0165] The identification unit 302 is configured to identify the data capacity of the data to be stored in the data attribute information, where the data capacity indicates the data size of the data to be stored.

[0166] For example, the identification unit 302 can be specifically used to count the size of the data to be stored in the data attribute information to obtain the data capacity of the data to be stored, or to identify the size of the data to be stored in the data attribute information to obtain the data capacity of the data to be stored.

[0167] (3) a first storage unit 303;

[0168] The first storage unit 303 is used to transfer the data to be stored from the first storage space of the first storage medium to the second storage space of the second storage medium when the data capacity does not exceed the preset data capacity threshold, and the writing speed of the first storage medium is greater than the writing speed of the second storage medium.

[0169] For example, the first storage unit 303 can be specifically used to write the data to be stored into the first storage space of the first storage medium through the target cache space when the data capacity does not exceed the preset data capacity threshold, and write the data to be stored cached in the target cache space into the second storage space of the second storage medium; when all the data to be stored are stored in the second storage space, the data to be stored is deleted in the first storage space.

[0170] (4) second storage space 304;

[0171] The second storage unit 304 is configured to write the data to be stored into the second storage space of the second storage medium when the data capacity exceeds a preset data capacity threshold.

[0172] For example, the second storage unit 304 can be specifically used to generate at least one data index corresponding to the data to be stored based on the data capacity of the data to be stored when the data capacity exceeds a preset data capacity threshold, determine the metadata corresponding to the data to be stored based on the physical offset address in the data index, add index headers to the metadata and the data to be stored respectively according to the data index, write the data to be stored with the added index header into the target data storage space, and write the metadata with the added index header into the original data storage space.

[0173] Optionally, in some embodiments, the data storage device may further include a first reading unit 305, such as Figure 14 As shown, the specific details can be as follows:

[0174] The first reading unit 505 is configured to read the data to be read when receiving a first data reading request for the data to be read.

[0175] For example, the first reading unit 305 can be specifically used to determine the data storage status of the data to be read according to the first data reading request when receiving a first data reading request for the data to be read, and based on the data storage status, identify the data storage location of the data to be read in the target cache space, the internal cache space, the first storage space and the second storage space, and read the data to be read according to the data storage location.

[0176] Optionally, in some embodiments, the data storage device may further include a second reading unit 306, such as Figure 15 As shown, the specific details can be as follows:

[0177] The second reading unit 306 is configured to read target read data corresponding to the second data read request from the second storage space when receiving the second data read request.

[0178] For example, the second reading unit 306 can be specifically used to identify the original data reading interval corresponding to the second storage space in the second data reading request when a second data reading request is received, determine the target data storage interval corresponding to the original data reading interval based on the first data storage interval and the second data storage interval, and read the target data corresponding to the target data storage space in the second storage space to obtain the target reading data corresponding to the second data reading request.

[0179] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can be found in the previous method embodiments and will not be repeated here.

[0180] As can be seen from the above, in the embodiment of the present application, after the acquisition unit 301 acquires at least one data to be stored and data attribute information of the data to be stored, the identification unit 302 identifies the data capacity of the data to be stored in the data attribute information. When the data capacity does not exceed the preset data capacity threshold, the first storage unit 303 transfers the data to be stored through the first storage space of the first storage medium to the second storage space of the second storage medium. The write speed of the first storage medium is greater than the write speed of the second storage medium. When the data capacity exceeds the preset data capacity threshold, the second storage unit 304 writes the data to be stored into the second storage space of the second storage medium. This scheme can determine different data storage paths according to the size of the identified data to be stored. When the data capacity of the data to be stored exceeds the preset data capacity threshold, that is, when the data to be stored is large block data (IO), it can bypass the first storage space of the first storage medium (fast storage medium) and directly write into the second storage space of the second storage medium (slow storage medium), thereby optimizing the life of the first storage medium. Moreover, for large block data, the bandwidth of the first storage medium and the network bandwidth can also be saved, thereby improving the storage efficiency of data storage.

[0181] The present application also provides an electronic device, such as Figure 16 , which shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0182] The electronic device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will understand that Figure 16 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0183] Processor 401 is the control center of the electronic device. It connects all parts of the electronic device using various interfaces and circuits. It performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in memory 402 and accessing data stored in memory 402. Optionally, processor 401 may include one or more processing cores. Preferably, processor 401 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0184] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0185] The electronic device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0186] The electronic device may further include an input unit 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0187] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402 to implement various functions as follows:

[0188] Obtain at least one data to be stored and data attribute information of the data to be stored, identify the data capacity of the data to be stored in the data attribute information, and the data capacity indicates the data size of the data to be stored. When the data capacity does not exceed a preset data capacity threshold, transfer the data to be stored to the second storage space of the second storage medium through the first storage space of the first storage medium, and the write speed of the first storage medium is greater than the write speed of the second storage medium. When the data capacity exceeds the preset data capacity threshold, write the data to be stored into the second storage space of the second storage medium.

[0189] For example, the electronic device can obtain at least one data to be stored and data attribute information of the data to be stored. The data capacity of the data to be stored is identified in the data attribute information, and the data capacity indicates the data size of the data to be stored. When the data capacity does not exceed the preset data capacity threshold, the data to be stored is written to the first storage space of the first storage medium through the target cache space, and the data to be stored cached in the target cache space is written to the second storage space of the second storage medium. When all the data to be stored is stored in the second storage space, the data to be stored is deleted in the first storage space. When the data capacity exceeds the preset data capacity threshold, at least one data index corresponding to the data to be stored is generated according to the data capacity of the data to be stored, and the metadata corresponding to the data to be stored is determined based on the physical offset address in the data index. According to the data index, an index header is added to the metadata and the data to be stored respectively, and the data to be stored with the added index header is written to the target data storage space, and the metadata with the added index header is written to the original data storage space. When a first data read request is received for data to be read, the data storage status of the data to be read is determined based on the first data read request. Based on the data storage status, the data storage location of the data to be read is identified in the target cache space, the internal cache space, the first storage space, and the second storage space. The data to be read is read based on the data storage location. When a second data read request is received, the original data read interval corresponding to the second storage space is identified in the second data read request. Based on the first data storage interval and the second data storage interval, a target data storage interval corresponding to the original data read interval is determined. The target data corresponding to the target data storage space is read from the second storage space to obtain the target read data corresponding to the second data read request.

[0190] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.

[0191] From the above, it can be seen that after obtaining at least one data to be stored and data attribute information of the data to be stored, the embodiment of the present application identifies the data capacity of the data to be stored in the data attribute information. When the data capacity does not exceed the preset data capacity threshold, the data to be stored is transferred to the second storage space of the second storage medium through the first storage space of the first storage medium. The write speed of the first storage medium is greater than the write speed of the second storage medium. When the data capacity exceeds the preset data capacity threshold, the data to be stored is written to the second storage space of the second storage medium. The scheme can determine different data storage paths according to the size of the identified data to be stored. When the data capacity of the data to be stored exceeds the preset data capacity threshold, that is, when the data to be stored is large block data (IO), it can bypass the first storage space of the first storage medium (fast storage medium) and directly write to the second storage space of the second storage medium (slow storage medium), thereby optimizing the life of the first storage medium. Moreover, for large block data, the bandwidth of the first storage medium and the network bandwidth can also be saved, thereby improving the storage efficiency of data storage.

[0192] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0193] To this end, an embodiment of the present application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the data storage methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0194] Obtain at least one data to be stored and data attribute information of the data to be stored, identify the data capacity of the data to be stored in the data attribute information, and the data capacity indicates the data size of the data to be stored. When the data capacity does not exceed a preset data capacity threshold, transfer the data to be stored to the second storage space of the second storage medium through the first storage space of the first storage medium, and the write speed of the first storage medium is greater than the write speed of the second storage medium. When the data capacity exceeds the preset data capacity threshold, write the data to be stored into the second storage space of the second storage medium.

[0195] For example, at least one data to be stored and data attribute information of the data to be stored are obtained. The data attribute information identifies the data capacity of the data to be stored, where the data capacity indicates the data size of the data to be stored. When the data capacity does not exceed a preset data capacity threshold, the data to be stored is written to the first storage space of the first storage medium via the target cache space, and the data to be stored cached in the target cache space is written to the second storage space of the second storage medium. When all the data to be stored is stored in the second storage space, the data to be stored is deleted from the first storage space. When the data capacity exceeds a preset data capacity threshold, at least one data index corresponding to the data to be stored is generated based on the data capacity of the data to be stored, metadata corresponding to the data to be stored is determined based on the physical offset address in the data index, and index headers are added to the metadata and the data to be stored based on the data index. The data to be stored with the index header added is written to the target data storage space, and the metadata with the index header added is written to the original data storage space. When a first data read request for data to be read is received, the data storage status of the data to be read is determined based on the first data read request. Based on the data storage status, the data storage location of the data to be read is identified in the target cache space, the internal cache space, the first storage space, and the second storage space, and the data to be read is read based on the data storage location. When a second data read request is received, the original data read interval corresponding to the second storage space is identified in the second data read request, and the target data storage interval corresponding to the original data read interval is determined based on the first data storage interval and the second data storage interval. The target data corresponding to the target data storage space is read from the second storage space to obtain the target read data corresponding to the second data read request.

[0196] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0197] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0198] Since the instructions stored in the computer-readable storage medium can execute the steps in any data storage method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data storage method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0199] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the aforementioned data storage or data writing / reading aspects.

[0200] The above is a detailed introduction to a data storage method and related equipment provided in an embodiment of the present application. The related equipment may include a data storage device, an electronic device, a computer program product and a computer-readable storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data storage method, characterized in that: include: Acquire at least one piece of data to be stored and data attribute information of the data to be stored; identifying a data capacity of the data to be stored in the data attribute information, wherein the data capacity indicates a data size of the data to be stored; When the data capacity does not exceed a preset data capacity threshold, transferring the data to be stored from the first storage space of the first storage medium to the second storage space of the second storage medium, the writing speed of the first storage medium being greater than the writing speed of the second storage medium; When the data capacity exceeds a preset data capacity threshold, the data to be stored is written into the second storage space of the second storage medium.

2. The data storage method according to claim 1, wherein: The transferring the data to be stored from the first storage space of the first storage medium to the second storage space of the second storage medium includes: Writing the data to be stored into the first storage space of the first storage medium through the target cache space; Writing the data to be stored cached in the target cache space into the second storage space of the second storage medium; When all the data to be stored are stored in the second storage space, the data to be stored is deleted from the first storage space.

3. The data storage method according to claim 2, wherein: Writing the data to be stored cached in the target cache space into the second storage space of the second storage medium includes: Aggregating the data to be stored to obtain aggregated data to be stored; Asynchronously writing the aggregated data to be stored into the internal cache space corresponding to the second storage space; When it is detected that all the aggregated data to be stored are stored in the internal cache space, the aggregated data to be stored in the internal cache space are asynchronously written into the second storage space.

4. The data storage method according to claim 3, wherein: After aggregating the data to be stored to obtain aggregated data to be stored, the method further includes: Detecting at least one stored data in the internal cache space to obtain current cache data; The currently cached data is deleted in the target cache space.

5. The data storage method according to claim 2, wherein: After writing the data to be stored into the first storage space of the first storage medium through the target cache space, the method further includes: When an abnormality is detected, obtaining a data storage record corresponding to the second storage space; identifying a data location of stored data in the second storage space based on the data storage record; According to the data location, writing the data in the target cache space into the second storage space is stopped.

6. The data storage method according to claim 3, characterized in that: Also includes: When receiving a first data read request for data to be read, determining a data storage state of the data to be read according to the first data read request; Based on the data storage status, identifying a data storage location of the to-be-read data in the target cache space, the internal cache space, the first storage space, and the second storage space; The to-be-read data is read according to the data storage location.

7. The data storage method according to claim 6, characterized in that: The step of identifying a data storage location of the to-be-read data in the target cache space, the internal cache space, the first storage space, and the second storage space based on the data storage state includes: When the data storage state is that the data is not asynchronously written into the internal cache space, determining that the data storage location of the to-be-read data is the target cache space; When the data storage status is that the data is not asynchronously written into the second storage space, determining that the data storage location of the to-be-read data is the first storage space; When the data storage status is that the data has been asynchronously written into the second storage space, it is determined that the data storage location of the data to be read is the second storage space.

8. The data storage method according to claim 1, wherein: Writing the data to be stored into the second storage space of the second storage medium includes: generating at least one data index corresponding to the data to be stored according to the data capacity of the data to be stored; Based on the data index, the data to be stored is written in parallel into the second storage space of the second storage medium.

9. The data storage method according to claim 8, characterized in that: The second data storage space includes a target data storage space, and generating at least one data index corresponding to the data to be stored according to the data capacity of the data to be stored includes: Dividing the data to be stored into at least one data block according to the data capacity of the data to be stored, and generating a logical offset address corresponding to each data block based on the data attribute information; compressing the data block to obtain a compressed data block, and determining a physical offset address of the compressed data block in the target data storage space; The logical offset address of each data block is associated with the physical offset address of the corresponding compressed data block to obtain at least one data index of the data to be stored.

10. The data storage method according to claim 9, characterized in that: The second storage space also includes original data storage space, and the step of writing the data to be stored in parallel into the second storage space of the second storage medium based on the data index includes: Determining metadata corresponding to the data to be stored based on the physical offset address, where the metadata indicates location information of the data to be stored in the target data storage space; According to the data index, adding index headers to the metadata and the data to be stored respectively; The data to be stored with the index header added is written into the target data storage space, and the metadata with the index header added is written into the original data storage space.

11. The data storage method according to claim 8, characterized in that: After writing the data to be stored in parallel into the second storage space of the second storage medium based on the data index, the method further includes: When there are multiple data to be stored and written into the second storage space in parallel, detecting the persistence status of each data to be stored in the second storage space; Based on the persistent state and the data index, screening out at least one target data to be stored from the data to be stored, and deleting the target data to be stored in the second storage space; The data index of the target data to be stored is restored, and based on the restored data index, the target data to be stored is written in parallel into the second storage space of the second storage medium.

12. The data storage method according to claim 11, characterized in that: The step of selecting at least one target data to be stored from the data to be stored based on the persistent state and the data index includes: When it is detected that there is data to be stored that has failed to be persisted and data to be stored that has succeeded in being persisted, the data to be stored are sorted according to the data index; Based on the sorting result, first candidate data to be stored is screened out from the data to be stored that failed to be persisted; According to the first candidate data to be stored and the sorting result, second candidate data to be stored is screened out from the successfully persisted data to be stored; Based on the sorting result, the data to be stored following the second candidate data to be stored in the data to be stored is used as the target data to be stored.

13. The data storage method according to claim 9, wherein: The logical offset address includes a first data storage interval, the physical offset address includes a second data storage interval, and after writing the to-be-stored data in parallel into the second storage space of the second storage medium based on the data index, the method further includes: When receiving a second data read request, identifying an original data read interval corresponding to the second storage space in the second data read request; Determine a target data storage interval corresponding to the original data reading interval according to the first data storage interval and the second data storage interval; Target data corresponding to the target data storage space is read out in the second storage space to obtain target read data corresponding to the second data read request.

14. The data storage method according to claim 13, wherein: The determining, based on the first data storage interval and the second data storage interval, a target data storage interval corresponding to the original data reading interval includes: Identifying a reading start point and a reading end point in the raw data reading interval; Based on the read start position and the read end position, screening out at least one candidate data storage interval that matches the data read interval in the first data storage interval; A second data storage interval corresponding to the candidate data storage interval is screened out from the second data storage interval to obtain a target data storage interval.

15. The data storage method according to claim 14, characterized in that: The selecting, based on the read start position and the read end position, at least one candidate data storage interval that matches the data read interval from the first data storage interval includes: Identifying a storage start point and a storage end point in the first data storage interval, and determining a boundary position of a data index corresponding to the second storage space; Comparing the storage starting position with the reading starting position to obtain a first comparison result, and comparing the storage end position with the reading end position to obtain a second comparison result; At least one candidate data storage interval is selected from the first data storage interval according to the first comparison result, the second comparison result, and the boundary position.

16. The data storage method according to claim 13, wherein: The step of reading candidate data corresponding to the target data storage interval in the second storage space to obtain target read data corresponding to the second data read request includes: Reading candidate data corresponding to the target data storage space in the second storage space, and stripping an index header from the candidate data to obtain candidate read data; When the data type of the candidate read data is metadata corresponding to the data to be stored, reading the target read data in the original data storage space of the second storage space based on the metadata; When the data type of the candidate read data is the data to be stored, the candidate read data is used as the target read data.

17. A data storage device, characterized in that: include: an acquiring unit, configured to acquire at least one piece of data to be stored and data attribute information of the data to be stored; an identification unit, configured to identify a data capacity of the data to be stored in the data attribute information, wherein the data capacity indicates a data size of the data to be stored; a first storage unit, configured to transfer the data to be stored from the first storage space of the first storage medium to the second storage space of the second storage medium when the data capacity does not exceed a preset data capacity threshold, wherein a write speed of the first storage medium is greater than a write speed of the second storage medium; The second storage unit is configured to write the data to be stored into the second storage space of the second storage medium when the data capacity exceeds a preset data capacity threshold.

18. An electronic device, characterized in that: The device comprises a processor and a memory, wherein the memory stores an application program, and the processor is configured to run the application program in the memory to execute the steps of the data storage method according to any one of claims 1 to 16.

19. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data storage method according to any one of claims 1 to 16 are implemented.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the data storage method according to any one of claims 1 to 16.

Images