Data processing method and device based on heterogeneous storage, equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method and device based on heterogeneous storage, equipment and storage medium. The method comprises: in response to a file write instruction issued by a client device to a virtual disk, storing content information of a file to be written in the virtual disk into a cache server, generating metadata of the file to be written, and storing the metadata of the file to be written into a metadata server; the metadata represents the content format and file attributes of the file; the client device is provided with a virtual disk; the content information of the file to be written is stored into a preset heterogeneous storage device to obtain storage location information of the file to be written; the preset heterogeneous storage device represents a device different from the storage medium of the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the position of the file in the preset heterogeneous storage device; and the storage location information of the file to be written is updated to the metadata server.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and storage medium based on heterogeneous storage. Background Technology

[0002] With the development of distributed heterogeneous storage and file system technologies, a variety of heterogeneous storage devices have emerged on the market, such as object storage, tape libraries, and optical disc libraries.

[0003] Different heterogeneous storage devices require the development of independent drivers and business logic. When users use remote heterogeneous storage devices to store and access data, they need to operate through their respective drivers and business logic, which results in low data processing efficiency and affects user experience. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium based on heterogeneous storage, which enables localized file operations on remote heterogeneous storage devices and improves data processing efficiency.

[0005] In a first aspect, embodiments of this application provide a data processing method based on heterogeneous storage, including:

[0006] In response to a file write command issued by a client device to a virtual disk, the system stores the content information of the file to be written in the virtual disk in a cache server, generates metadata of the file to be written, and stores the metadata of the file to be written in a metadata server; wherein, the content information represents the content expressed by the file, and the metadata represents the content format and file attributes of the file; the client device is equipped with a virtual disk;

[0007] The content information of the file to be written is stored in a preset heterogeneous storage device to obtain the storage location information of the file to be written; wherein, the preset heterogeneous storage device represents a device with a different storage medium than the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device.

[0008] The storage location information of the file to be written is updated in the metadata server.

[0009] Secondly, embodiments of this application provide a data processing apparatus based on heterogeneous storage, comprising:

[0010] The first response unit is used to respond to a file write command issued by the client device to the virtual disk, store the content information of the file to be written in the virtual disk in the cache server, generate the metadata of the file to be written, and store the metadata of the file to be written in the metadata server; wherein, the content information represents the content expressed by the file, and the metadata represents the content format and file attributes of the file; the client device is equipped with a virtual disk;

[0011] A location determination unit is used to store the content information of the file to be written into a preset heterogeneous storage device to obtain the storage location information of the file to be written; wherein, the preset heterogeneous storage device represents a device with a different storage medium than the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device.

[0012] The data storage unit is used to update the storage location information of the file to be written to the metadata server.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0014] The memory stores computer-executed instructions;

[0015] The processor executes computer execution instructions stored in the memory, causing the processor to perform the possible implementation of the first aspect described above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the possible implementations of the first aspect described above.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the possible implementations of the first aspect described above.

[0018] This application provides a data processing method, apparatus, device, and storage medium based on heterogeneous storage. A virtual disk is set up in the client device. The user opens the virtual disk and issues a file write command through it, for example, the user creates a file using the same operating method as a local disk. Responding to the user's file write command, the content information of the file to be written is obtained and stored in a cache server. The metadata of the file to be written is also obtained and stored in a metadata server. A protocol connection exists between the electronic device and the remote heterogeneous storage device. The electronic device can store the content information of the file to be written in the remote heterogeneous storage device, allowing the user to perform localized file operations on the remote heterogeneous storage device as if writing a local file. The user is unaware of the heterogeneous storage device on the electronic device side, achieving consistency in user operations and shielding the differences in the underlying media. The storage location information of the file to be written is determined, and the metadata of the file to be written and its storage location information are associated and stored in the metadata server, achieving persistent storage of metadata, facilitating file management and access, and improving data processing efficiency. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A flowchart illustrating a data processing method based on heterogeneous storage provided in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram illustrating the mounting relationship between a virtual disk and a heterogeneous storage device provided in an embodiment of this application.

[0022] Figure 3 A flowchart illustrating a data processing method based on heterogeneous storage provided in an embodiment of this application;

[0023] Figure 4 A flowchart illustrating a data processing method based on heterogeneous storage provided in an embodiment of this application;

[0024] Figure 5 A schematic diagram of the data reading process provided in the embodiments of this application;

[0025] Figure 6 A schematic diagram of the structure of a data processing device based on heterogeneous storage provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0027] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] This application belongs to the field of computer technology related to distributed heterogeneous storage and file systems, specifically involving the standardized access, unified metadata management, and transparent cross-network access of heterogeneous storage devices through a virtual file system layer. Heterogeneous storage devices can include block storage, object storage, tape libraries, optical disc libraries, etc., and storage media can be categorized into three states: online media, near-line media, and offline media. Online media can refer to an online cache pool, near-line media can refer to a tape library or optical disc library, and offline media can refer to retrieved tapes, etc.

[0030] Heterogeneous storage devices require the development of independent drivers and business logic, necessitating users to learn multiple APIs (Application Programming Interfaces). This demands a high level of expertise from users and limits the applicability of heterogeneous storage devices. Furthermore, accessing heterogeneous storage devices across networks requires explicit handling of protocol differences, such as block uploads and sequential read / write operations, making them unsuitable for local file operations and impacting user experience. For nearline and offline media, file metadata cannot be persisted, creating a black hole in file management, leading to fragmented file system views, affecting file management and access, and consequently impacting file data processing efficiency and accuracy.

[0031] This application provides a data processing method, apparatus, device, and storage medium based on heterogeneous storage, which aims to solve the above-mentioned technical problems in the prior art.

[0032] It should be noted that the data in this embodiment is not specific to any particular user and does not reflect the personal information of any particular user. It should also be noted that the data in this embodiment comes from a publicly available dataset.

[0033] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Figure 1 This is a flowchart illustrating a data processing method based on heterogeneous storage, provided as an embodiment of this application. This method can be executed by a data processing device based on heterogeneous storage. Figure 1 As shown, the method includes:

[0036] S101. Responding to the file write command sent by the client device to the virtual disk, the content information of the file to be written in the virtual disk is stored in the cache server, the metadata of the file to be written is generated, and the metadata of the file to be written is stored in the metadata server; wherein, the content information represents the content expressed by the file, the metadata represents the content format and file attributes of the file, and the client device is configured with a virtual disk.

[0037] For example, the client device can be a terminal device used by the user, such as the user's personal computer. The user can allocate one or more virtual disk spaces on the client device, acting as virtual disks. That is, the client device is an electronic device configured with virtual disks. These virtual disks can be created using FUSE (File system in Userspace), which allows for the implementation of custom file systems without modifying the kernel. On the client device, these can be presented as / mnt / F or F: drive; that is, the virtual disk can be recognized as a standard block device. Users can operate the virtual disk as if it were a physical disk. The FUSE driver layer can receive file operations, such as open, read, and write operations. The backend storage adapter can map these operations to corresponding heterogeneous storage devices, making it seamless for the user.

[0038] In other words, users can perform file read and write operations through a virtual disk. When a user wants to write a file, they can create a new file on the virtual disk, write content into the file, and issue a file write command. The FUSE function within the virtual disk can receive the user's file write command in real time. After responding to the file write command, the content information of the file to be written is obtained. A high-speed cache server is pre-configured, containing a cache pool, which is an online storage medium. The content information of the file to be written can be stored in the cache server, i.e., stored in the cache pool. That is, when a user writes a file to the virtual disk, the virtual disk client pushes the file to the cache server. After storing the content information of the file to be written in the cache server, the metadata of the file to be written can be obtained. A metadata server is pre-configured, which may contain a preset database to store metadata. The metadata of the file to be written can be stored in the metadata server, i.e., stored in the preset database. Content information represents the content expressed by the file; that is, the content written by the user can be obtained. Metadata can represent the file's content format and file attributes, such as file size, length, offset, file handle, creation time, and type.

[0039] In this embodiment, responding to a file write command sent by a client device to a virtual disk and storing the content information of the file to be written in the virtual disk in a cache server includes: responding to a file write command sent by a client device to a virtual disk and obtaining the user's permission information; wherein, the permission information represents the user's operation permissions on the file; if it is determined from the user's permission information that the user has the permission to perform file write operations, then the content information of the file to be written in the virtual disk is stored in the cache server.

[0040] Specifically, permission information is pre-configured and stored for each user. This permission information refers to a user's file operation permissions, which may include allowed file operations, allowed heterogeneous storage devices, and allowed file sizes for reading and writing. File operations may include read and write operations. For example, each user can have their own client account and can perform write operations through the virtual disk in their client. Different users' clients can have different permissions.

[0041] The FUSE driver layer can receive user file write commands. Upon receiving a file write command, it retrieves the user's corresponding permission information. Based on the user's permission information, it determines whether the user has permission to perform a file write operation. If so, the file can be written to the virtual disk, the content information and metadata of the file to be written can be obtained, and subsequent steps S102 and S103 can be executed. If not, steps S102 and S103 are not executed, and a permission reminder message can be sent to the client device's visual interface to remind the user that they do not have file write permissions.

[0042] The advantage of this setup is that by defining user permissions, file tampering or leakage can be prevented, thus improving file data security.

[0043] In this embodiment, if it is determined that the user has the permission to write files based on the user's permission information, the content information of the file to be written in the virtual disk is stored in the cache server. This includes: if it is determined that the user has the permission to write files based on the user's permission information, obtaining the size information of the file to be written in the virtual disk and the remaining space information of the virtual disk; wherein, the remaining space information represents the remaining space in the bucket size corresponding to the virtual disk pre-configured for the user; if the size information of the file to be written is less than or equal to the remaining space information, the content information of the file to be written is stored in the cache server.

[0044] Specifically, after determining that a user has the permission to perform file write operations based on the user's permission information, it only means that the user can perform write operations, but it does not mean that the user can store all the content currently being written into the corresponding heterogeneous storage device.

[0045] Each virtual disk can correspond to a Bucket, meaning a Bucket is mounted as a virtual disk. Different Buckets are isolated from each other, thus isolating data for different users and implementing different access permissions. In other words, users can create buckets, limit bucket size, mount buckets on the client, and thus access the client's virtual disk; the contents of different buckets are not visible to each other. Data transfer between virtual disks and heterogeneous storage devices can be performed through Buckets. For example, the native protocols of heterogeneous storage devices can be converted into a unified interface conforming to preset standards and mounted across networks through a high-speed gateway to achieve remote localized file operations on various heterogeneous media. Localization means mounting a virtual disk locally, like a local hard drive; when reading and writing to the virtual disk, the system automatically reads and writes to the remote heterogeneous storage device, such as a tape library. Figure 2 This diagram illustrates the mounting relationship between a virtual disk and a heterogeneous storage device. Figure 2The heterogeneous storage device in this context can be a tape library, and F: refers to a virtual disk, in which FUSE is deployed.

[0046] A bucket quota can be set for each virtual disk bucket. The bucket quota is the bucket capacity, i.e., the maximum size of a file that a user can write to the heterogeneous storage device. For example, a bucket quota of 60T can be set for a user. When a user writes a file through the virtual disk, the remaining space in the bucket corresponding to the virtual disk can be determined in real time, i.e., the remaining space information of the virtual disk is determined. The size information of the file to be written is compared with the remaining space information. If the size information of the file to be written is less than or equal to the remaining space information, the content information and metadata of the file to be written can be obtained, and subsequent steps S102 and S103 can be executed; if the size information of the file to be written is greater than the remaining space information, not all the content information can be transferred to the heterogeneous storage device, and a pop-up window can be displayed on the visual interface of the electronic device, prompting the user to reduce the file size and rewrite.

[0047] The benefits of this setting are that it limits the number of files a user can write, preventing users from abusing storage media, and it determines in real time whether there is enough space available for the user to write files, thus improving the user experience.

[0048] S102. Store the content information of the file to be written into a preset heterogeneous storage device to obtain the storage location information of the file to be written; wherein, the preset heterogeneous storage device represents a device with a different storage medium than the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device.

[0049] For example, one or more heterogeneous storage devices are mounted remotely on an electronic device. These heterogeneous storage devices can be devices with different storage media than the electronic device. The storage media in different heterogeneous storage devices can be different; for example, heterogeneous storage devices can be block storage, object storage, tape libraries, optical disc libraries, etc. The storage media can include three states: online, near-line, and offline. In this embodiment, the remote heterogeneous storage device can be a near-line device. If a storage medium such as a tape or optical disc is plugged into the heterogeneous storage device, the storage medium is in a near-line state; if the storage medium is unplugged, the storage medium is in an offline state. Data transmission between the electronic device and the heterogeneous storage device can be performed through a preset protocol interface. For example, the native protocol of the heterogeneous storage device can be converted into a unified interface conforming to a preset standard.

[0050] After receiving the content information of the file to be written, the electronic device can transmit the content information to a heterogeneous storage device for storage through a preset interface. A virtual disk can be connected to one or more heterogeneous storage devices, and the specific heterogeneous storage device to which a file to be written to the virtual disk will be stored can be preset. After receiving the content information, the electronic device can directly send the content information to the corresponding heterogeneous storage device and determine the storage location information of the file to be written within that device. The storage location information indicates the preset heterogeneous storage device where the file resides and the file's location within that device; that is, it determines which heterogeneous storage device the file to be written exists in and its storage path within that device.

[0051] In this embodiment, storing the content information of the file to be written in the virtual disk to the cache server includes: obtaining the size information of the file to be written in the virtual disk; if the size information is greater than a preset first threshold, then dividing the content information of the file to be written into blocks to obtain at least two file blocks; wherein, the at least two file blocks are combined to form the content information of the file to be written; and uploading the at least two file blocks in parallel to the cache server for storage.

[0052] Specifically, to obtain the content information of the file to be written, the content information can be first written to a cache server, and then written to a preset heterogeneous storage device. When writing the content information to the cache server, the content information can be written directly as a complete file block, or the size of the content information can be determined and written to the cache server in blocks according to the size of the content information.

[0053] The size of the content information is determined as the size of the file to be written. For example, the size of the file to be written could be 100KB. A first threshold is preset. The size of the file to be written is compared with the preset first threshold. If the size is less than or equal to the preset first threshold, the entire content information can be directly stored in the preset heterogeneous storage device. If the size is greater than the preset first threshold, the content information of the file to be written can be divided into blocks according to the preset first threshold, resulting in at least two file blocks. For example, if the content information of the file to be written is 128MB and the first threshold is 4MB, it can be divided into blocks of 4MB each.

[0054] The content of the file to be written can be divided according to a first threshold, ensuring that the size of each file block is within the first threshold. When dividing to the last file block, if the last file block is smaller than the first threshold, the division can stop; that is, the size of the last file block can be smaller than the first threshold. The divided file blocks can be combined to form the complete content of the file to be written. Multiple divided file blocks are then uploaded in parallel to a preset heterogeneous storage device for storage, such as object storage or a tape library.

[0055] The advantage of this setup is that it allows for the processing of large files in chunks and improves file upload efficiency through parallel uploading.

[0056] In this embodiment, storing the content information of the files to be written in the virtual disk to the cache server includes: if there are at least two files to be written, and the size information of each file to be written is less than a preset second threshold, then the content information of the at least two files to be written is aggregated to obtain an aggregate block; wherein, the aggregate block represents the complete content information of the at least two files to be written; and the aggregate block is uploaded to the cache server for storage.

[0057] Specifically, in addition to processing large files in blocks, small files can also be aggregated during write operations to perform batch write operations. For example, 1000 4KB write operations can be combined into a single 4000KB write operation.

[0058] A second threshold can be preset. After determining that there are multiple files to be written, the size of the content information of each file can be determined, i.e., the size information of each file to be written is obtained. Each size information is compared with the second threshold. If the size information of each file to be written is less than the second threshold, the content information of these files to be written can be aggregated to obtain an aggregate block. That is, the aggregate block can represent the complete content information of these files to be written. Uploading the aggregate block to a heterogeneous storage device for storage means that the complete content information of these files to be written is written to the heterogeneous storage device in batches.

[0059] If the size of one or more files to be written is equal to or greater than the second threshold, then that file can be identified as a separate file to be uploaded, and all other files will be aggregated. It is worth noting that a size threshold for the aggregation block can be preset; that is, the size of the aggregation block cannot exceed this threshold. If it does, the excess portion can be treated as a new aggregation block.

[0060] The advantage of this setup is that it aggregates small files, reduces metadata overhead and IOPS (Input / Output Operations Per Second) overhead, and improves file upload efficiency.

[0061] In this embodiment, an intelligent proxy gateway is deployed in the electronic device; the method further includes: when storing the content information of the file to be written to a preset heterogeneous storage device, the content information of the file to be written is encrypted through the intelligent proxy gateway to obtain a first hash value of the file to be written, and the first hash value of the file to be written is stored; wherein, the first hash value is used to characterize the content information of the file.

[0062] Specifically, an intelligent proxy gateway can be deployed in the electronic device. This gateway can be positioned between the electronic device and a remote heterogeneous storage device, providing functions such as protocol conversion, caching acceleration, and data reading, and supporting multi-protocol access. In this embodiment, the intelligent proxy gateway can be deployed between the FUSE driver and the remote heterogeneous storage device. The intelligent proxy gateway can first write the file content information to a high-speed cache server, and then write it to the heterogeneous storage device, instead of having FUSE write it directly to the heterogeneous storage device. In other words, after FUSE obtains the content information of the file to be written, it needs to pass the content information to the heterogeneous storage device through the intelligent proxy gateway.

[0063] The FUSE driver can forward content information to the smart proxy gateway. Using a preset encryption algorithm, the smart proxy gateway encrypts the content information of the file to be written, obtaining a first hash value for the file, which is then stored in the smart proxy gateway. Each time a user writes a file, the smart proxy gateway generates a new first hash value; different first hash values ​​correspond to different files. In this embodiment, the preset encryption algorithm is not specifically limited.

[0064] The advantage of this setup is that each written file corresponds to a first hash value, which facilitates subsequent integrity verification of read files and improves data security.

[0065] S103. Update the storage location information of the file to be written to the metadata server.

[0066] For example, the metadata server can be a database located within an electronic device or in a device independent of the electronic device. After obtaining the metadata of the file to be written, the metadata can be sent to the metadata server. In this embodiment, the type of metadata server is not specifically limited; for example, the metadata server can be MongoDB, that is, the metadata can be stored in MongoDB.

[0067] After obtaining the storage location information of the file to be written, this information can also be sent to a metadata server for storage. The metadata and storage location information of the file to be written can be associated and stored in the metadata server, making it easier to find the corresponding file later.

[0068] This application provides a data processing method based on heterogeneous storage. A virtual disk is set up in the client device. The user opens the virtual disk and issues a file write command through it, for example, the user creates a file using the same operating method as a local disk. In response to the user's file write command, the content information of the file to be written is obtained and stored in a high-speed cache server. The metadata of the file to be written is also obtained and stored in a metadata server. A protocol connection exists between the electronic device and the remote heterogeneous storage device. The electronic device can store the content information of the file to be written in the remote heterogeneous storage device, allowing the user to perform localized file operations on the remote heterogeneous storage device as if writing a local file. The user is unaware of the heterogeneous storage device on the electronic device side, achieving consistency in user operations and shielding the differences in the underlying media. The storage location information of the file to be written is determined, and the metadata of the file to be written and its storage location information are associated and stored in the metadata server, achieving persistent storage of metadata, facilitating file management and access, and improving data processing efficiency.

[0069] Figure 3 This application provides a flowchart illustrating a data processing method based on heterogeneous storage, as shown in the embodiments below. Figure 3 As shown, in this embodiment... Figure 1 Based on the embodiments, a data processing method based on heterogeneous storage is described in detail, the method including:

[0070] S301. Responding to the file write command sent by the client device to the virtual disk, the content information of the file to be written in the virtual disk is stored in the cache server, the metadata of the file to be written is generated, and the metadata of the file to be written is stored in the metadata server; wherein, the content information represents the content expressed by the file, and the metadata represents the content format and file attributes of the file; the client device is equipped with a virtual disk.

[0071] S302. Locate the initial storage location information of the file to be written from the metadata server.

[0072] For example, the metadata server can be in the form of MongoDB. Every time a user writes a file, the metadata of that file can be stored in the metadata server. That is, the metadata server can store all the metadata.

[0073] Once a file is written to a heterogeneous storage device, its storage location information can be obtained and stored in the metadata server. In other words, the metadata server can also include the file's storage location information. For each file, the metadata server can associate and store the file's metadata and storage location information together for easy retrieval later.

[0074] When a user performs a write operation, FUSE can be used to retrieve the metadata of the file to be written and determine whether the corresponding storage location information already exists in the metadata server. In other words, it determines whether the file to be written is a newly created file. The storage location information already stored in the metadata server is used as the initial storage location information; that is, the initial storage location information of the file to be written is retrieved from the metadata server.

[0075] The system checks the metadata server to see if the initial storage location information of the file to be written exists. For example, it can determine if the path of the file to be written exists in the metadata server. If it exists, the file to be written is not a newly created file; if it does not exist, the file to be written is determined to be a newly created file.

[0076] In this embodiment, the metadata of the file to be written can also be retrieved from the metadata server. For example, the content information and metadata of the file to be written can be forwarded to the smart proxy gateway through FUSE in the virtual disk; and the metadata of the file to be written can be retrieved from the metadata server through the smart proxy gateway.

[0077] Specifically, FUSE obtains the content and metadata of the file to be written, and forwards this information to the smart proxy gateway. The smart proxy gateway is deployed between FUSE and the heterogeneous storage device, enabling protocol access between electronic devices and heterogeneous storage devices.

[0078] The intelligent proxy gateway can also enable data transmission between electronic devices and metadata servers. The intelligent proxy gateway obtains the metadata of the file to be written and checks whether that metadata exists in the metadata server.

[0079] The advantage of this configuration is that the FUSE driver can forward the obtained data to the smart proxy gateway, which can then query metadata from the metadata server to determine whether to write the file repeatedly, thereby improving the efficiency of data processing.

[0080] S303. If the metadata server does not have the initial storage location information of the file to be written, or if the metadata server has the initial storage location information and meets the preset writing conditions, then through the cache server, based on the preset interface, the content information of the file to be written is stored in the preset heterogeneous storage device to obtain the storage location information of the file to be written.

[0081] For example, if it is determined that the metadata for the file to be written does not exist in the metadata server, it indicates that the file to be written is a newly created file. The content information of the file to be written can be stored in a heterogeneous storage device to obtain the storage location information of the file to be written, thus avoiding duplicate file storage. For example, content information from a cache server can be written to a heterogeneous storage device based on a preset interface. Alternatively, the metadata and storage location information of the file to be written can be stored in the metadata server. For example, a new record can be created in the metadata server; this record contains the metadata and storage location information of the file to be written. If the file is subsequently written again, this storage location information can be used as the initial storage location information.

[0082] If initial storage location information exists in the metadata server, it can be determined whether this initial storage location information meets the preset write conditions. If so, the content information of the file to be written in the cache server can be stored in the preset heterogeneous storage device based on the preset interface. In this embodiment, the preset write conditions are not specifically limited. For example, the preset write conditions can be specific storage location information. If the initial storage location information is not specific storage location information, then the preset write conditions are met.

[0083] In this embodiment, metadata can be persisted by storing it on a metadata server. Metadata persistence means that once a file is written to a virtual disk, it can always be retrieved within the file system. However, in reality, this file may have already been written to a tape, which is now offline, i.e., removed from the tape library, and the local cache has been cleared. When reading the file, the tape needs to be put back into the tape library.

[0084] In this embodiment, the method further includes: after storing the content information of the file to be written into a preset heterogeneous storage device, determining the current storage location information of the content information of the file to be written in the preset heterogeneous storage device; if the metadata server has the initial storage location information of the file to be written, then storing the current storage location information and the initial storage location information together in the metadata server, or replacing the initial storage location information with the current storage location information.

[0085] Specifically, if the metadata server contains initial storage location information and the preset write conditions are met, then the content information needs to be written to the heterogeneous storage device to obtain the current storage location information. In other words, the file to be written corresponds to two storage location information: the initial storage location information and the current storage location information.

[0086] The metadata server originally stores initial storage location information. Either both storage location information and the initial storage location information can be stored in the metadata server, or the initial storage location information can be replaced, recording only the current storage location information, i.e., the latest storage location information. In this embodiment, a storage strategy for the storage location information can be pre-set according to actual needs to determine whether to replace the initial storage location information.

[0087] The advantage of this setup is that if the metadata server already stores the storage location information of historical versions, the information at the storage location of historical versions can be retained or updated, improving the flexibility of data processing, facilitating data traceability, and enhancing the user experience.

[0088] S304. Update the storage location information of the file to be written to the metadata server.

[0089] This application provides a data processing method based on heterogeneous storage. A virtual disk is set up in the client device. The user opens the virtual disk and issues a file write command through it, for example, the user creates a file using the same operating method as a local disk. In response to the user's file write command, the content information of the file to be written is obtained and stored in a high-speed cache server. The metadata of the file to be written is also obtained and stored in a metadata server. A protocol connection exists between the electronic device and the remote heterogeneous storage device. The electronic device can store the content information of the file to be written in the remote heterogeneous storage device, allowing the user to perform localized file operations on the remote heterogeneous storage device as if writing a local file. The user is unaware of the heterogeneous storage device on the electronic device side, achieving consistency in user operations and shielding the differences in the underlying media. The storage location information of the file to be written is determined, and the metadata of the file to be written and its storage location information are associated and stored in the metadata server, achieving persistent storage of metadata, facilitating file management and access, and improving data processing efficiency.

[0090] Figure 4 This application provides a flowchart illustrating a data processing method based on heterogeneous storage, as shown in the embodiments below. Figure 4 As shown, in this embodiment... Figure 1 Based on the embodiments, a data processing method based on heterogeneous storage is described in detail, the method including:

[0091] S401. Responding to the file write command sent by the client device to the virtual disk, the content information of the file to be written in the virtual disk is stored in the cache server, the metadata of the file to be written is generated, and the metadata of the file to be written is stored in the metadata server; wherein, the content information represents the content expressed by the file, and the metadata represents the content format and file attributes of the file; the client device is equipped with a virtual disk.

[0092] For example, the cache pool of a cache server can be a storage medium in the form of a hard disk. For instance, the cache pool can be a standard cache volume deployed in an electronic device, or a hard disk installed on the electronic device. After obtaining the content information of the file to be written, the content information of the file to be written can be stored in the cache server. The cache server is an online storage medium.

[0093] In this embodiment, the method further includes: for files in the cache server, obtaining file information of the files according to a preset time period; wherein the file information includes at least one of the following: file read frequency, last access time, and file size; if the file information does not meet the preset retention conditions, the file is deleted from the cache server.

[0094] Specifically, since cache servers cannot persistently store all content information, this embodiment achieves intelligent data flow within the cache server through a three-layer linkage mechanism without the user's awareness. This three-layer linkage mechanism includes hot data acceleration, cold data settling, and offline media wake-up. Hot data acceleration refers to caching frequently accessed files in the cache server, reducing read latency from minutes to microseconds. Cold data settling refers to removing infrequently accessed files from the cache server. Offline media wake-up refers to automatically triggering background migration when offline media reloading is detected, such as tape reloading, prioritizing the restoration of recently queried files to the cache server. In other words, the data in the cache server can be continuously updated.

[0095] A pre-set time period can be used to periodically update the cache server. For example, at each pre-set time period, file information for each file in the cache server can be retrieved. This file information can include various categories such as file read frequency, last access time, file size, and disk space usage.

[0096] For each file in the cache server, it is determined whether the file information meets the preset retention conditions. If it does, the file content is retained in the cache server; otherwise, it is deleted from the cache server. For example, the preset retention condition could be that the file read frequency must be greater than a preset frequency threshold. Files with a read frequency less than or equal to the frequency threshold can be deleted from the cache server.

[0097] The advantage of this setup is that it regularly cleans up the files in the cache server, avoiding redundant data from occupying space. This ensures that the files stored in the cache server are relatively important, making it easier to read data directly from the cache server and thus improving the efficiency of data reading.

[0098] In this embodiment, if the file information does not meet the preset retention conditions, the file is deleted from the cache server, including: determining the weight information of the file based on the file information; wherein the weight information represents the importance of the file; if the weight information is less than a preset weight threshold, the file is deleted from the cache server.

[0099] Specifically, for a file in the cache server, the file information is retrieved, and based on this information, the file's weight information is determined. Weight information characterizes the importance of a file; for example, the more frequently a file is read, the higher its weight; the earlier the last access time, the lower its weight.

[0100] For each information category in the file information, a weight can be preset. Based on the preset weight, the file information is weighted and summed, and the result is determined as the weight information. A weight threshold is preset, and the file's weight information is compared with the weight threshold. If the weight information is less than the preset weight threshold, the file is deleted from the cache server; if the weight information is equal to or greater than the preset weight threshold, the file is retained in the cache server.

[0101] The advantage of this setup is that by calculating weight information, the importance of files can be quantified, which facilitates the management of files in the cache server and improves the efficiency of subsequent file reading.

[0102] S402. Store the content information of the file to be written into a preset heterogeneous storage device to obtain the storage location information of the file to be written; wherein, the preset heterogeneous storage device represents a device with a different storage medium than the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device.

[0103] S403. Update the storage location information of the file to be written to the metadata server.

[0104] S404. Respond to the file read command sent by the client device to the virtual disk and obtain the metadata of the file to be read; if the content information of the file to be read is not found in the cache server according to the metadata of the file to be read, then search for the storage location information corresponding to the metadata of the file to be read in the metadata server; and obtain the content information of the file to be read from the preset heterogeneous storage device according to the storage location information corresponding to the metadata of the file to be read.

[0105] For example, users can perform both write and read operations on the virtual disk. For instance, double-clicking a file on the virtual disk issues a file read command. The FUSE function on the virtual disk responds to this command, identifying the file the user wants to read. Each file stores its own metadata, and the metadata of the file to be read is retrieved. In this embodiment, a file read operation does not necessarily follow a write operation.

[0106] FUSE forwards the metadata of the file to be read to the smart proxy gateway. The smart proxy gateway then uses this metadata to search the cache server for the file. For example, the metadata might include the file's storage path on the virtual disk. Each file on the cache server has its own corresponding metadata, such as the file's storage path on the virtual disk. Based on the storage path of the file to be read, the gateway searches the cache server for the file's content information.

[0107] If the content information of the file to be read is found in the cache server, it can be directly returned to the visual interface of the electronic device for the user to browse. For the user, this means double-clicking a document to directly open its content on the page.

[0108] If the content information of the file to be read is not found in the cache server, the metadata of the file to be read can be retrieved from the metadata server. Based on the correlation between metadata and storage location information in the metadata server, the storage location information corresponding to the metadata of the file to be read is determined. Based on the storage location information corresponding to the metadata of the file to be read, the heterogeneous storage device where the file to be read is located is determined, and the content information of the file to be read is retrieved from that heterogeneous storage device based on a preset interface, and returned to the visual interface of the electronic device. For the user, this means that the user double-clicks a document, directly opening the document's content on the page.

[0109] The advantage of this setup is that when accessing data, the system prioritizes searching the cache server. If the data is not found in the cache server, it retrieves it from a remote heterogeneous storage device based on the metadata and storage location information in the metadata server. The user is unaware of the query process, thus improving the user experience.

[0110] In this embodiment, the method further includes storing the content information of the file to be read in a high-speed cache server.

[0111] Specifically, if a user wants to access a file to be read, it indicates that the file is currently of high importance to the user. However, the cache server does not have the content information of this file. Therefore, after obtaining the content information of the file to be read from the heterogeneous storage device, the content information can be stored in the cache server for easy retrieval later. In this embodiment, migrating the content information from the heterogeneous storage device back to the cache pool does not require a smart proxy gateway. The smart proxy gateway transmits metadata and content information between the client and each server. The metadata is stored in the metadata server, and the content information is stored in the cache server. The heterogeneous storage device is connected to the server side, and the content information can be directly written to the cache server during data migration.

[0112] The advantage of this setup is that for files in near-line media such as disk libraries and optical disc libraries, automatic loading can be triggered when a user accesses the file, automatically migrating the content information back to the online cache server; for files in offline media, automatic migration can be triggered after the media is reinserted, that is, the content information is automatically migrated back to the online cache server, facilitating subsequent access and improving data processing efficiency.

[0113] In this embodiment, the preset heterogeneous storage device includes a pluggable storage medium; the method further includes: if it is determined, based on the storage location information corresponding to the metadata of the file to be read, that the storage location information in the preset heterogeneous storage device cannot be accessed, a prompt message is issued; wherein, the prompt message is used to prompt the user to load the offline storage device.

[0114] Specifically, this embodiment can pre-configure various heterogeneous storage devices, such as tape libraries, Blu-ray disc libraries, etc. Different heterogeneous storage devices can correspond to different storage media. In this embodiment, heterogeneous devices generally refer to near-line devices. Some heterogeneous storage devices contain pluggable storage media. Offline refers to removing tapes or optical discs from near-line devices. For example, a tape library may contain pluggable tapes, and the removed tapes or optical discs are considered offline storage media.

[0115] After obtaining the storage location information, it can be read from the corresponding heterogeneous storage device. If the corresponding storage medium exists in the heterogeneous storage device, the file content information can be read; if the corresponding storage medium does not exist in the heterogeneous storage device, the file content information cannot be read. In other words, if the storage location information in the heterogeneous storage device cannot be accessed, it indicates that the storage medium for that location has been removed, and a prompt message can be issued on the electronic device to remind the user to load the offline storage medium. For example, it can remind the user to insert a magnetic tape or optical disc. In this embodiment, a robotic arm can also automatically insert the magnetic tape or optical disc.

[0116] The advantage of this setup is that it can remind users to load offline media in a timely manner, thereby enabling file reading and facilitating interaction with the user during the data reading process, improving data reading efficiency and enhancing the user experience.

[0117] In this embodiment, an intelligent proxy gateway is deployed in the electronic device; the method further includes: after obtaining the content information of the file to be read, encrypting the content information of the file to be read through the intelligent proxy gateway to obtain a second hash value of the file to be read; wherein, the second hash value is used to represent the content information of the file; if a first hash value that matches the second hash value is found, it is determined that the content information of the file to be read has been verified; wherein, the first hash value is a hash value representing the content information of the file generated when writing the file.

[0118] Specifically, when a file is written, the smart proxy gateway calculates its first hash value using a preset encryption algorithm. When a user needs to read the file, the smart proxy gateway obtains the file's content and encrypts it using the same preset encryption algorithm to obtain the second hash value. The encryption algorithms used to generate the first and second hash values ​​can be the same.

[0119] The intelligent proxy gateway compares the generated second hash value with all previously determined first hash values. If a first hash value matching the second hash value is found, it is determined that the content information of the file to be read is complete, and the content information verification of the file to be read is completed. If no first hash value matching the second hash value is found, it is determined that the content information verification of the file to be read has failed, and the content information of the file to be read may have been tampered with or lost. The user can be prompted for confirmation.

[0120] The advantage of this setup is that it allows for integrity verification of file content using the first and second hash values, thereby improving the security of data processing.

[0121] Figure 5 This is a schematic diagram of the data reading process. As shown in Figure 5, "read" represents a read operation, the client application can refer to a virtual disk on an electronic device, and the cache pool is the cache server in this embodiment, which can be in the form of an SSD (Solid State Drive), serving as an online storage medium. Figure 5 The process is as follows:

[0122] 1. The user performs a read operation on the client application.

[0123] 2. When the FUSE driver layer responds to a read operation, it determines the metadata of the file to be read and forwards the metadata to the smart proxy gateway.

[0124] 3. The intelligent proxy gateway checks whether the file to be read has been cached in the cache pool.

[0125] 4. If the file is found in the cache pool, return the file to be read to the smart proxy gateway; if the file is not found, return the result of the miss to the smart proxy gateway.

[0126] 5. The intelligent proxy gateway receives the results of a match failure and, based on the metadata, retrieves the storage location information corresponding to the metadata from MongoDB. MongoDB is the metadata server in this embodiment.

[0127] 6. MongoDB returns storage location information to the intelligent proxy gateway.

[0128] 7. The intelligent gateway agent retrieves the content information of the file to be read from the heterogeneous storage device based on the storage location information. It is worth noting that the cache pool is an online medium, while the heterogeneous storage device can be either near-line or offline. That is, content information can be retrieved from either near-line or offline heterogeneous storage devices.

[0129] 8. Heterogeneous storage devices write content information back to the cache pool.

[0130] 9. Display content information on the client's interface through a smart proxy gateway.

[0131] This application provides a data processing method based on heterogeneous storage. An electronic device is equipped with a virtual disk, and a file-specific device (FUSE) is deployed within the virtual disk. A user opens the virtual disk and issues a file write command, for example, using the same operating method as a local disk, by creating a new document to write the file. The FUSE in the virtual disk responds to the user's file write command, obtaining the content information and metadata of the file to be written. A protocol connection exists between the electronic device and a remote heterogeneous storage device. The electronic device can store the content information of the file to be written on the remote heterogeneous storage device, allowing the user to perform localized file operations on the remote heterogeneous storage device as if writing a local file. The user is unaware of the heterogeneous storage device on the electronic device side, achieving consistency in user operations and shielding the differences in the underlying media. The storage location information of the file to be written is determined, and the metadata of the file to be written and its storage location information are associated and stored in a metadata server, achieving persistent storage of metadata, facilitating file management and access, and improving data processing efficiency.

[0132] Figure 6 This is a schematic diagram of a data processing device based on heterogeneous storage, provided as an embodiment of this application. Figure 6 As shown, the data processing device 60 based on heterogeneous storage provided in this embodiment includes:

[0133] The first response unit 601 is used to respond to a file write command sent by the client device to the virtual disk, store the content information of the file to be written in the virtual disk in a cache server, generate the metadata of the file to be written, and store the metadata of the file to be written in a metadata server; wherein, the content information represents the content expressed by the file, and the metadata represents the content format and file attributes of the file; the client device is equipped with a virtual disk;

[0134] The location determination unit 602 is used to store the content information of the file to be written into a preset heterogeneous storage device to obtain the storage location information of the file to be written; wherein, the preset heterogeneous storage device represents a device with a different storage medium than the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device.

[0135] The data storage unit 603 is used to update the storage location information of the file to be written to the metadata server.

[0136] In one possible implementation, the position determination unit 602 includes:

[0137] The metadata lookup module is used to find the initial storage location information of the file to be written from the metadata server;

[0138] The content writing module is used to store the content information of the file to be written to a preset heterogeneous storage device through the cache server based on a preset interface if the initial storage location information of the file to be written does not exist in the metadata server, or if the initial storage location information exists in the metadata server and the preset writing conditions are met.

[0139] One possible implementation also includes:

[0140] The current location determination unit is used to determine the current storage location information of the content information of the file to be written in the preset heterogeneous storage device after storing the content information of the file to be written in the preset heterogeneous storage device.

[0141] A location storage unit is configured to, if the metadata server contains initial storage location information for the file to be written, store the current storage location information and the initial storage location information together in the metadata server, or replace the initial storage location information with the current storage location information.

[0142] In one possible implementation, the first response unit 601 includes:

[0143] The block segmentation module is used to obtain the size information of the file to be written in the virtual disk. If the size information is greater than a preset first threshold, the content information of the file to be written is segmented to obtain at least two file blocks. The at least two file blocks are combined to form the content information of the file to be written. The at least two file blocks are uploaded in parallel to the cache server for storage.

[0144] In one possible implementation, the first response unit 601 includes:

[0145] An aggregation module is used to aggregate the content information of at least two files to be written if there are at least two files to be written, and the size information of each file to be written is less than a preset second threshold, to obtain an aggregation block; wherein, the aggregation block represents the complete content information of at least two files to be written; and the aggregation block is uploaded to a cache server for storage.

[0146] In one possible implementation, the first response unit 601 includes:

[0147] The permission acquisition module is used to respond to file write commands sent by the client device to the virtual disk and acquire the user's permission information; wherein, the permission information represents the user's operation permissions on the file;

[0148] The permission determination module is used to store the content information of the file to be written in the virtual disk to the cache server if it is determined from the user's permission information that the user has the permission to perform file write operations.

[0149] In one possible implementation, the permission determination module is specifically used for:

[0150] If, based on the user's permission information, it is determined that the user has permission to perform file write operations, then the size information of the file to be written in the virtual disk and the remaining space information of the virtual disk are obtained; wherein, the remaining space information represents the remaining space in the bucket size corresponding to the virtual disk pre-configured for the user;

[0151] If the size of the file to be written is less than or equal to the remaining space information, then the content of the file to be written is stored in the cache server.

[0152] One possible implementation also includes:

[0153] The cache cleanup unit is used to obtain file information of a file in the cache server according to a preset time period; wherein the file information includes at least one of file read frequency, last access time, and file size; if the file information does not meet the preset retention conditions, the file is deleted from the cache server.

[0154] In one possible implementation, the cache cleanup unit is specifically used for:

[0155] Based on the file information, the weight information of the file is determined; wherein, the weight information represents the importance of the file;

[0156] If the weight information is less than a preset weight threshold, the file will be deleted from the cache server.

[0157] One possible implementation also includes:

[0158] The second response unit is used to respond to the file read command sent by the client device to the virtual disk and obtain the metadata of the file to be read;

[0159] The cache lookup unit is used to look up the storage location information corresponding to the metadata of the file to be read from the metadata server if the content information of the file to be read is not found in the cache server based on the metadata of the file to be read.

[0160] The file reading unit is used to obtain the content information of the file to be read from a preset heterogeneous storage device based on the storage location information corresponding to the metadata of the file to be read.

[0161] One possible implementation also includes:

[0162] The content caching unit is used to store the content information of the file to be read in the cache server.

[0163] In one possible implementation, the pre-defined heterogeneous storage device includes a pluggable storage medium; the device also includes:

[0164] The prompting unit is used to issue a prompting message if it is determined, based on the storage location information corresponding to the metadata of the file to be read, that the storage location information in the preset heterogeneous storage device cannot be accessed; wherein the prompting message is used to prompt the user to load the storage medium.

[0165] The data processing device based on heterogeneous storage provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0166] This application provides a heterogeneous storage virtual file system, which can be used to execute the heterogeneous storage-based data processing method described in this application. The system enables standardized access, unified metadata management, and transparent cross-network access for heterogeneous storage devices such as object storage, tape libraries, and optical disc libraries. It maintains file queryability even when the media is physically offline, and automatically triggers data migration after the accessed near-line or offline media is remounted to the heterogeneous storage device. In the heterogeneous storage virtual file system, the native protocols of heterogeneous storage devices such as block storage, object storage, tape libraries, and optical disc libraries can be converted into a unified interface and mounted across networks via a high-speed gateway. This enables remote local file operations on various heterogeneous media, allowing the system to solve problems related to multi-media compatibility, near-line and offline data accessibility, and consistent user experience.

[0167] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0168] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0169] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0170] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0171] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0172] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0173] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0174] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0175] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0176] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0177] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0180] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0182] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method based on heterogeneous storage, characterized in that, The method comprises the following steps: In response to a file write instruction issued by a client device to a virtual disk, storing content information of a file to be written in the virtual disk into a cache server, generating metadata of the file to be written, and storing the metadata of the file to be written into a metadata server; wherein the content information represents the content expressed by the file, and the metadata represents the content format and file attributes of the file; the client device is provided with a virtual disk; Storing the content information of the file to be written into a preset heterogeneous storage device to obtain storage location information of the file to be written; wherein the preset heterogeneous storage device represents a device different from the storage medium of the client device, and the storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device; Updating the storage location information of the file to be written into the metadata server.

2. The method of claim 1, wherein, Storing the content information of the file to be written into a preset heterogeneous storage device comprises the following steps: Finding initial storage location information of the file to be written from the metadata server; If the initial storage location information of the file to be written does not exist in the metadata server, or if the initial storage location information exists in the metadata server and a preset write condition is met, storing the content information of the file to be written into a preset heterogeneous storage device based on a preset interface through the cache server.

3. The method of claim 2, wherein, After storing the content information of the file to be written into a preset heterogeneous storage device, the method further comprises the following steps: Determining the current storage location information of the content information of the file to be written in the preset heterogeneous storage device; If the initial storage location information of the file to be written exists in the metadata server, storing the current storage location information and the initial storage location information together in the metadata server, or replacing the initial storage location information with the current storage location information.

4. The method of claim 1, wherein, Storing the content information of the file to be written in the virtual disk into the cache server comprises the following steps: Obtaining size information of the file to be written in the virtual disk, and if the size information is greater than a preset first threshold, performing block processing on the content information of the file to be written to obtain at least two file blocks; wherein the at least two file blocks are combined as the content information of the file to be written; Uploading the at least two file blocks to the cache server in parallel for storage.

5. The method of claim 1, wherein, Storing the content information of the file to be written in the virtual disk into the cache server comprises the following steps: If there are at least two files to be written, and the size information of each file to be written is less than a preset second threshold, performing aggregation processing on the content information of the at least two files to be written to obtain an aggregation block; wherein the aggregation block represents the total content information of the at least two files to be written; Uploading the aggregation block to the cache server for storage.

6. The method of claim 1, wherein, In response to a file write instruction issued by a client device to a virtual disk, storing content information of a file to be written in the virtual disk into a cache server, comprising: In response to a file write instruction issued by a client device to a virtual disk, obtaining permission information of a user issuing the file write instruction; wherein the permission information represents the operation permission of the user to the file; If it is determined that the user has the permission of file write operation according to the permission information of the user, then storing the content information of the file to be written in the virtual disk into the cache server.

7. The method of claim 6, wherein, If it is determined that the user has the permission of file write operation according to the permission information of the user, then storing the content information of the file to be written in the virtual disk into the cache server, comprising: If it is determined that the user has the permission of file write operation according to the permission information of the user, then obtaining size information of the file to be written in the virtual disk and remaining space information of the virtual disk; wherein the remaining space information represents the remaining space in the bucket size corresponding to the virtual disk configured for the user in advance; If the size information of the file to be written is less than or equal to the remaining space information, then storing the content information of the file to be written into the cache server.

8. The method of claim 1, wherein, Further comprising: For the file in the cache server, obtaining file information of the file according to a preset time period; wherein the file information comprises at least one of file read frequency, last access time, and file size; If the file information does not satisfy a preset retention condition, then deleting the file from the cache server.

9. The method of claim 8, wherein, If the file information does not satisfy a preset retention condition, then deleting the file from the cache server, comprising: According to the file information, determining weight information of the file; wherein the weight information represents the importance of the file; If the weight information is less than a preset weight threshold, then deleting the file from the cache server.

10. The method of claim 1, wherein, Further comprising: In response to a file read instruction issued by a client device to a virtual disk, obtaining metadata of a file to be read; If the content information of the file to be read is not found from the cache server according to the metadata of the file to be read, then finding storage location information corresponding to the metadata of the file to be read from a metadata server; According to the storage location information corresponding to the metadata of the file to be read, obtaining the content information of the file to be read from a preset heterogeneous storage device.

11. The method of claim 10, wherein, Further comprising: Storing the content information of the file to be read into the cache server.

12. The method of claim 10, wherein, The preset heterogeneous storage device comprises a pluggable storage medium; the method further comprises: If it is determined that the storage location information in the preset heterogeneous storage device cannot be accessed according to the storage location information corresponding to the metadata of the file to be read, then issuing a prompt information; wherein the prompt information is used to prompt the user to load the storage medium.

13. A data processing apparatus based on heterogeneous storage, characterized in that, Comprising: The first response unit is configured to store content information of a file to be written in the virtual disk into the cache server, generate metadata of the file to be written, and store the metadata of the file to be written into the metadata server in response to a file write instruction issued by the client device to the virtual disk; wherein the content information represents content expressed by the file, and the metadata represents content format and file attributes of the file; and the client device is provided with the virtual disk. The position determination unit is configured to store the content information of the file to be written into a preset heterogeneous storage device to obtain storage position information of the file to be written; wherein the preset heterogeneous storage device represents a device different from a storage medium of the client device, and the storage position information represents a preset heterogeneous storage device where the file is located and a position of the file in the preset heterogeneous storage device. The data storage unit is configured to update the storage position information of the file to be written into the metadata server.

14. An electronic device, comprising: The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-12. The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-12. The computer program is executed by the processor to implement the method in any one of claims 1-12. ​ 15. A computer readable storage medium characterized by: ​ 16. A computer program product, characterised in that, ​

Images