Data processing method and device based on heterogeneous storage, equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device based on heterogeneous storage, equipment and a storage medium. The method comprises the following steps: in response to a file writing instruction sent to a virtual disk by client equipment, storing content information of a to-be-written file in the virtual disk into a cache server, generating metadata of the to-be-written file, and storing the metadata of the to-be-written file into a metadata server; the metadata represents the content format and the file attribute of the file; a virtual disk is arranged in the client device; storing the content information of the to-be-written file into a preset heterogeneous storage device to obtain storage position information of the to-be-written file; the preset heterogeneous storage device represents a device different from a storage medium of the electronic device, and the storage position information represents the preset heterogeneous storage device where the file is located and the position of the file in the preset heterogeneous storage device; and updating the storage position information of the file to be written into the metadata server.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a data processing method and device based on heterogeneous storage, equipment and a storage medium. BACKGROUND

[0002] With the development of distributed heterogeneous storage and file system technology, various heterogeneous storage devices appear on the market, such as object storage, tape library, optical disc library and the like.

[0003] Different heterogeneous storage devices need to develop independent drivers and business logic. When a user uses a remote heterogeneous storage device for data storage and access, the user needs to operate through the respective corresponding drivers and business logic, and the data processing efficiency is low, which affects the user experience. SUMMARY

[0004] The embodiments of the present application provide a data processing method and device based on heterogeneous storage, equipment and a storage medium, to realize local file operation on a remote heterogeneous storage device and improve the data processing efficiency.

[0005] In a first aspect, the embodiments of the present application provide a data processing method based on heterogeneous storage, comprising:

[0006] In response to a file write instruction issued by a client device to a virtual disk, content information of a file to be written in the virtual disk is stored in a cache server, metadata of the file to be written is generated, and the metadata of the file to be written is stored in a metadata server. Wherein, the content information represents the content expressed by the file, and the metadata represents the content format and file attribute of the file. The client device is provided with a virtual disk;

[0007] The content information of the file to be written is stored in a preset heterogeneous storage device to obtain storage location information of the file to be written. Wherein, the preset heterogeneous storage device represents a device different from the storage medium of the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device;

[0008] The storage location information of the file to be written is updated to the metadata server.

[0009] In a second aspect, the embodiments of the present application provide a data processing device based on heterogeneous storage, comprising:

[0010] The first response unit is configured to store content information of a file to be written in the virtual disk into the cache server, generate metadata of the file to be written, and store the metadata of the file to be written into the metadata server in response to a file write instruction issued by the client device to the virtual disk; wherein the content information represents content expressed by the file, and the metadata represents content format and file attributes of the file; and the client device is provided with the virtual disk.

[0011] The position determination unit is configured to store the content information of the file to be written into a preset heterogeneous storage device to obtain storage position information of the file to be written; wherein the preset heterogeneous storage device represents a device different from a storage medium of the electronic device, and the storage position information represents a preset heterogeneous storage device where the file is located and a position of the file in the preset heterogeneous storage device.

[0012] The data storage unit is configured to update the storage position information of the file to be written into the metadata server.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor;

[0014] The memory stores computer execution instructions.

[0015] The processor executes the computer execution instructions stored in the memory, so that the processor executes the possible implementation manners of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the possible implementation manners of the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the possible implementation manners of the first aspect.

[0018] The embodiment of the present application provides a data processing method, device and equipment based on heterogeneous storage and a storage medium, a virtual disk is arranged in a client device, a user opens the virtual disk, and issues a file write instruction through the virtual disk, for example, the user creates a file by adopting an operation mode consistent with a local disk. In response to the file write instruction issued by the user, content information of a file to be written is acquired, the content information is stored into a cache server, and metadata of the file to be written is obtained, so that the metadata is stored into a metadata server. There is a protocol connection between an electronic device and a remote heterogeneous storage device, and the electronic device can store the content information of the file to be written into the remote heterogeneous storage device, so that the user performs a localized file operation on the remote heterogeneous storage device as if writing a local file, the user has no perception of the heterogeneous storage device on the side of the electronic device, consistency of user operation is realized, and differences of underlying media are shielded. Storage location information of the file to be written is determined, the metadata of the file to be written and the storage location information of the file to be written are stored in association into the metadata server, persistent storage of the metadata is realized, file management and access are facilitated, and the efficiency of data processing is improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0020] Figure 1 A flowchart of a data processing method based on heterogeneous storage provided by the embodiment of the present application is shown in the figure.

[0021] Figure 2 A mounting relationship between a virtual disk and a heterogeneous storage device provided by the embodiment of the present application is shown in the figure.

[0022] Figure 3 A flowchart of a data processing method based on heterogeneous storage provided by the embodiment of the present application is shown in the figure.

[0023] Figure 4 A flowchart of a data processing method based on heterogeneous storage provided by the embodiment of the present application is shown in the figure.

[0024] Figure 5 A flowchart of a data processing method based on heterogeneous storage provided by the embodiment of the present application is shown in the figure.

[0025] Figure 6 A structure diagram of a data processing device based on heterogeneous storage provided by the embodiment of the present application is shown in the figure.

[0026] Figure 7 A structure diagram of an electronic device provided by the embodiment of the present application is shown in the figure.

[0027] The specific embodiments of the application have been shown and described in the above drawings and text. These drawings and text are not meant to limit the scope of the inventive concept in any way but are merely meant to illustrate the inventive concept to one skilled in the art by reference to a particular embodiment. DETAILED DESCRIPTION

[0028] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The same numbers are used in different drawings to represent the same or similar elements. The following detailed description is not meant to limit the application in any way. Rather, the following description is meant to provide example embodiments of apparatuses and methods consistent with the application as detailed in the appended claims.

[0029] The application belongs to the computer technology field of distributed heterogeneous storage and file system, and specifically relates to standardization access of heterogeneous storage devices, unified management of metadata, and transparent access across networks through a virtual file system layer. The heterogeneous storage devices can include block storage, object storage, tape library, optical disk library, etc., and the storage media includes three states of online media, nearline media, and offline media. The online media can be a cache pool, the nearline media can be a tape library or an optical disk library, and the offline media can be a removed tape.

[0030] The heterogeneous storage devices need to develop independent drivers and business logic, and users need to learn multiple APIs (Application Programming Interface), which has high professional requirements for users and affects the application range of the heterogeneous storage devices. In addition, cross-network access to the heterogeneous storage devices needs to explicitly handle protocol differences, such as block uploading, sequential reading and writing, etc., which cannot be operated like local files, affecting user experience. For nearline and offline media, the metadata of the file cannot be persisted, causing a black hole in file management, leading to a broken file system view, affecting the management and access of the file, and further affecting the data processing efficiency and precision of the file.

[0031] The application provides a data processing method and device based on heterogeneous storage, equipment and storage medium, which aims to solve the above technical problems of the prior art.

[0032] It should be noted that the data in this embodiment is not for a specific user and cannot reflect the personal information of a specific user. It should be noted that the data in this embodiment comes from a public data set.

[0033] The collection, storage, use, processing, transmission, provision and disclosure of the user personal information involved in the technical solutions of the present disclosure comply with relevant laws and regulations and do not violate public order and good customs.

[0034] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described again in some examples. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0035] Figure 1 A flowchart of a data processing method based on heterogeneous storage provided for the embodiments of the present application, which can be executed by a data processing apparatus based on heterogeneous storage. As shown in the figure, Figure 1 The method comprises:

[0036] S101, in response to a file write instruction issued by a client device to a virtual disk, storing content information of a file to be written in the virtual disk into a cache server, generating metadata of the file to be written, and storing the metadata of the file to be written into a metadata server; wherein the content information represents the content expressed by the file, the metadata represents the content format and file attributes of the file, and the client device is provided with the virtual disk.

[0037] Exemplarily, the client device can be a terminal device used by a user, for example, it can be a personal computer of the user. The user can open one or more virtual disk spaces in the client device as virtual disks. That is, the client device is an electronic device provided with a virtual disk, which can be created through FUSE (File system in Userspace, user space file system) in the virtual disk, FUSE allows custom file systems to be implemented without modifying the kernel, and can be presented as / mnt / F or F: disk on the client device, that is, the virtual disk can be recognized as a standard block device. The user can operate the virtual disk like a physical disk, and the FUSE driver layer can receive operations on the file, such as open, read, write, etc. Through the back-end storage adapter, the operation can be mapped to the corresponding heterogeneous storage device, so that the user has no awareness.

[0038] That is, the user can read and write files and the like through the virtual disk, and when the user wants to write a file, a new file can be created on the virtual disk, and content is written in the file, that is, a file write instruction is issued. Through the FUSE in the virtual disk, the user's file write instruction can be obtained in real time. After responding to the file write instruction, the content information of the file to be written is obtained, and the cache server is pre-configured, and the cache server contains a cache pool, which is an online storage medium. The content information of the file to be written can be stored in the cache server, that is, in the cache pool. That is, the user writes the file to the virtual disk, and the virtual disk client pushes the file to the cache server. After storing the content information of the file to be written in the cache server, the metadata of the file to be written can be obtained. The metadata server is pre-configured, and the metadata server can contain a preset database for storing metadata. The metadata of the file to be written can be stored in the metadata server, that is, in the preset database. The content information represents the content expressed by the file, that is, the content written by the user can be obtained. The metadata can represent the content format and file attributes of the file, for example, can include the size, length, offset, file handle, creation time, type and the like of the file.

[0039] In this embodiment, in response to the file write instruction issued by the client device to the virtual disk, the content information of the file to be written in the virtual disk is stored in the cache server, comprising: in response to the file write instruction issued by the client device to the virtual disk, obtaining the user's permission information; wherein the permission information represents the user's operation permission on the file; if it is determined that the user has the permission of the file write operation according to the user's permission information, the content information of the file to be written in the virtual disk is stored in the cache server.

[0040] Specifically, the corresponding permission information of each user is pre-configured and stored, and the permission information can be the operation permission of the user on the file. The operation permission can include the file operation allowed to be performed by the user, the heterogeneous storage device allowed to be used by the user, and the size of the file allowed to be read and written by the user. The file operation can include read operation, write operation and the like. For example, each user can correspond to his own client account, and can perform write operation through the virtual disk in his own client. The clients of different users can correspond to different permissions.

[0041] The file write instruction of the user can be received through the FUSE drive layer, and the permission information corresponding to the user is obtained after responding to the file write instruction. According to the permission information of the user, it is judged whether the user has the permission of the file write operation, that is, the write operation. If yes, the file can be written to the virtual disk, the content information of the file to be written and the metadata of the file to be written are obtained, and subsequent S102 and S103 are continued to be executed; if not, S102 and S103 are not executed, and the permission reminding information can be sent to the visual interface of the client device to remind the user that the user does not have the permission of file write.

[0042] The beneficial effect of such setting is that by determining the user's permission, the file can be prevented from being tampered with or leaked, and the security of the file data is improved.

[0043] In the embodiment, if it is determined according to the permission information of the user that the user has the permission of the file write operation, the content information of the file to be written in the virtual disk is stored in the cache server, comprising: if it is determined according to the permission information of the user that the user has the permission of the file write operation, the size information of the file to be written in the virtual disk and the remaining space information of the virtual disk are obtained; wherein the remaining space information represents the remaining space in the bucket size corresponding to the virtual disk configured for the user in advance; if the size information of the file to be written is less than or equal to the remaining space information, the content information of the file to be written is stored in the cache server.

[0044] Specifically, after it is determined according to the permission information of the user that the user has the permission of the file write operation, it only means that the user can perform the write operation, and it does not mean that the user can store all the content information currently written into the corresponding heterogeneous storage device.

[0045] Each virtual disk can correspond to a Bucket (bucket), that is, the Bucket is mounted as a virtual disk. Different Buckets are isolated from each other, so that the data of different users is isolated to realize different access permissions. That is, the user can create a bucket, the size of the bucket can be limited, and the bucket can be mounted on the client to access the virtual disk of the client. The contents of different buckets are not visible to each other. Data transmission between the virtual disk and the heterogeneous storage device can be performed through the Bucket. For example, the native protocol of the heterogeneous storage device can be converted into a uniform interface conforming to a predetermined standard, and can be mounted across the network through a high-speed gateway to realize remote localization of various heterogeneous media file operations. The meaning of localization is to mount a virtual disk locally, which is read and written to the remote heterogeneous storage device, such as a tape library, like a local hard disk. Figure 2 A schematic diagram of the mounting relationship between the virtual disk and the heterogeneous storage device is shown in FIG. 1. Figure 2The heterogeneous storage device in the FUSE can be a tape library, F: refers to a virtual disk, and the FUSE is deployed in the virtual disk.

[0046] A bucket quota can be set for each virtual disk bucket, and the bucket quota is the capacity of the bucket, that is, the maximum capacity of the file that the user can write into the heterogeneous storage device. For example, a user can be set with a bucket quota of 60T. When the user writes a file through the virtual disk, the size of the remaining space in the bucket corresponding to the virtual disk can be determined in real time, that is, the remaining space information of the virtual disk is determined. The size information of the file to be written by the user is compared with the remaining space information, if the size information of the file to be written is less than or equal to the remaining space information, the content information and metadata of the file to be written can be obtained, and subsequent S102 and S103 are executed; if the size information of the file to be written is greater than the remaining space information, the entire content information cannot be transmitted to the heterogeneous storage device, and a pop-up window can be displayed on the visual interface of the electronic device to prompt the user to reduce the size of the file and re-write.

[0047] The beneficial effect of such setting is that the file written by the user is limited, the user abuses the storage medium is avoided, and it is determined in real time whether there is enough space to provide the user for the file writing operation when the user writes, and the user experience is improved.

[0048] S102, the content information of the file to be written is stored in the preset heterogeneous storage device to obtain the storage location information of the file to be written; wherein the preset heterogeneous storage device represents a device different from the storage medium of the electronic device, and the storage location information represents the location of the file in the preset heterogeneous storage device.

[0049] Exemplarily, one or more heterogeneous storage devices are mounted at the remote end of the electronic device, and the heterogeneous storage device can be a device different from the storage medium of the electronic device. The storage medium in different heterogeneous storage devices can be different, for example, the heterogeneous storage device can be block storage, object storage, tape library, optical disk library, etc. The storage medium can include online, nearline, and offline states, in this embodiment, the heterogeneous storage device at the remote end can be a nearline device, if the storage medium such as a tape or an optical disk in the heterogeneous storage device is inserted into the device, the storage medium is in a nearline state; if the storage medium such as a tape or an optical disk in the heterogeneous storage device is pulled out, the storage medium is in an offline state. The electronic device and the heterogeneous storage device can transmit data through a preset protocol interface, for example, the native protocol of the heterogeneous storage device can be converted into a uniform interface conforming to a preset standard.

[0050] After obtaining the content information of the file to be written, the electronic device can transmit the content information to the heterogeneous storage device through a preset interface for storage. One virtual disk can be connected with one or more heterogeneous storage devices, and it can be preset which heterogeneous storage device the file written in the virtual disk is to be stored in. After obtaining the content information, the content information can be directly sent to the corresponding heterogeneous storage device, and the storage location information of the file to be written in the heterogeneous storage device is determined. The storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device, that is, which heterogeneous storage device the file to be written exists in and the storage path in the heterogeneous storage device.

[0051] In this embodiment, storing the content information of the file to be written in the virtual disk into the cache server includes: obtaining size information of the file to be written in the virtual disk, and if the size information is greater than a preset first threshold, performing block processing on the content information of the file to be written to obtain at least two file blocks; wherein the at least two file blocks are combined as the content information of the file to be written; and uploading the at least two file blocks to the cache server in parallel for storage.

[0052] Specifically, obtaining the content information of the file to be written can first write the content information into the cache server and then write it into the preset heterogeneous storage device. When writing the content information into the cache server, the content information can be directly written as a complete file block, or the size of the content information can be determined, and the content information can be block written into the cache server according to the size of the content information.

[0053] The size of the content information is determined as the size information of the file to be written. For example, the size information of the file to be written can be 100KB. A first threshold is preset, and the size information of the file to be written is compared with the preset first threshold. If the size information is less than or equal to the preset first threshold, the entire content information can be directly stored in the preset heterogeneous storage device; if the size information is greater than the preset first threshold, the content information of the file to be written can be block processed according to the preset first threshold to obtain at least two file blocks. For example, the content information of the file to be written is 128MB, and the first threshold is 4MB, which can be block processed in units of 4MB.

[0054] The content information of the file to be written can be divided according to the size of the first threshold value, so that the size of each file block can be the size of the first threshold value. When dividing into the last file block, if the last file block is smaller than the first threshold value, the division can be stopped, that is, the size of the last file block can be smaller than the first threshold value. The divided file blocks can be combined as the complete content information of the file to be written. The divided multiple file blocks can be uploaded in parallel to the preset heterogeneous storage device for storage, for example, can be uploaded in parallel to the object storage or tape library, etc.

[0055] The beneficial effect of such arrangement is that the large file is divided into blocks, and the efficiency of file uploading is improved through parallel uploading.

[0056] In the embodiment, storing the content information of the file to be written in the virtual disk into the cache server includes: if there are at least two files to be written, and the size information of each file to be written is smaller than the preset second threshold value, then the content information of the at least two files to be written is aggregated to obtain an aggregated block; wherein the aggregated block represents the complete content information of the at least two files to be written; and the aggregated block is uploaded to the cache server for storage.

[0057] Specifically, in addition to dividing the large file into blocks, when performing the write operation, the small file can also be aggregated and written in batch. For example, 1000 write operations of 4KB can be combined into a single write operation of 4000KB.

[0058] A second threshold value can be set in advance. After determining that there are multiple files to be written, the size of the content information of each file to be written can be determined, that is, the size information of each file to be written is obtained. Each size information is compared with the second threshold value, and if the size information of each file to be written is smaller than the second threshold value, the content information of these files to be written can be aggregated to obtain an aggregated block. That is, the aggregated block can represent the complete content information of these files to be written. The aggregated block is uploaded to the heterogeneous storage device for storage, that is, the complete content information of these files to be written is written in batch to the heterogeneous storage device.

[0059] If the size information of one or more files to be written is equal to or greater than the second threshold value, the file to be written can be determined as a separately uploaded file, and the files other than the file are aggregated. It is worth noting that the size threshold of the aggregated block can be set in advance, that is, the size of the aggregated block cannot exceed the size threshold, and if it exceeds, the exceeding part can be a new aggregated block.

[0060] The beneficial effect of such an arrangement is that small files are aggregated, reducing metadata overhead and IOPS (Input / Output Operations Per Second) overhead, and improving the efficiency of file uploading.

[0061] In this embodiment, a smart proxy gateway is deployed in the electronic device; the method further includes: when storing the content information of the file to be written into the preset heterogeneous storage device, performing encryption processing on the content information of the file to be written by the smart proxy gateway, obtaining a first hash value of the file to be written, and storing the first hash value of the file to be written; wherein the first hash value is used to represent the content information of the file.

[0062] Specifically, a smart proxy gateway can be deployed in the electronic device, and the smart proxy gateway can be deployed between the electronic device and the remote heterogeneous storage device, providing protocol conversion, cache acceleration, data reading, and the like, and supporting multi-protocol access. In this embodiment, the smart proxy gateway can be specifically deployed between the FUSE driver and the remote heterogeneous storage device, and the smart proxy gateway can first write the content information of the file into the cache server and then write it into the heterogeneous storage device, instead of being directly written into the heterogeneous storage device by the FUSE. That is, after the FUSE obtains the content information of the file to be written, the content information needs to be transmitted to the heterogeneous storage device by the smart proxy gateway.

[0063] The FUSE driver can forward the content information to the smart proxy gateway, and a preset encryption algorithm is used to perform encryption processing on the content information of the file to be written in the smart proxy gateway, to obtain a first hash value of the file to be written, and store the first hash value in the smart proxy gateway. A first hash value can be generated in the smart proxy gateway each time a file is written, and different first hash values correspond to different files. In this embodiment, the preset encryption algorithm is not specifically limited.

[0064] The beneficial effect of such an arrangement is that each written file corresponds to a first hash value, facilitating subsequent integrity verification of the read file and improving data security.

[0065] S103, update the storage location information of the file to be written to the metadata server.

[0066] Exemplarily, the metadata server can be a database located in the electronic device, or can be located in a device independent of the electronic device. After obtaining the metadata of the file to be written, the metadata can be sent to the metadata server. In this embodiment, the type of the metadata server is not specifically limited, for example, the metadata server can be MongoDB, that is, the metadata can be stored in MongoDB.

[0067] After obtaining the storage location information of the file to be written, the storage location information can also be sent to the metadata server for storage. The metadata of the file to be written and the storage location information can be stored in the metadata server in association, so as to facilitate subsequent searching of the corresponding file according to the metadata server.

[0068] The embodiment of the present application provides a data processing method based on heterogeneous storage. A virtual disk is arranged in a client device. A user opens the virtual disk, and issues a file writing instruction through the virtual disk. For example, the user creates a file by using an operation mode consistent with a local disk. In response to the file writing instruction issued by the user, content information of the file to be written is obtained, the content information is stored in a cache server, and metadata of the file to be written is obtained, so that the metadata is stored in a metadata server. There is a protocol connection between an electronic device and a remote heterogeneous storage device. The electronic device can store the content information of the file to be written in the remote heterogeneous storage device, so that the user performs a localized file operation on the remote heterogeneous storage device as if writing a local file. The user has no perception of the heterogeneous storage device on the side of the electronic device, operation consistency of the user is achieved, and differences of underlying media are shielded. Storage location information of the file to be written is determined, and the metadata of the file to be written and the storage location information of the file to be written are stored in the metadata server in association, so that persistent storage of the metadata is achieved, management and access of the file are facilitated, and the efficiency of data processing is improved.

[0069] Figure 3 A flowchart of a data processing method based on heterogeneous storage provided by the embodiment of the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, the embodiment of the present application is based on the embodiment. Figure 1 The embodiment of the present application is based on the embodiment.

[0070] S301, in response to a file writing instruction issued by a client device to a virtual disk, content information of a file to be written in the virtual disk is stored in a cache server, metadata of the file to be written is generated, and the metadata of the file to be written is stored in a metadata server; wherein the content information represents content expressed by the file, and the metadata represents a content format and file attributes of the file; the client device is provided with the virtual disk.

[0071] S302, initial storage location information of the file to be written is searched from the metadata server.

[0072] Exemplarily, the metadata server can be in the form of MongoDB, and metadata of each file written by a user can be stored in the metadata server, i.e., all the metadata can be stored in the metadata server.

[0073] When the file is written into the heterogeneous storage device, storage location information of the file can be obtained and stored in the metadata server as well. That is, the storage location information of the file can also be included in the metadata server. For each file, the metadata and the storage location information of the file can be stored in the metadata server in association, facilitating subsequent searching.

[0074] When the user performs a write operation, metadata of the file to be written can be obtained by the FUSE, and it can be determined whether the storage location information corresponding to the metadata already exists in the metadata server. That is, it is determined whether the file to be written is a newly created file. The storage location information already stored in the metadata server is used as initial storage location information, i.e., the initial storage location information of the file to be written is searched from the metadata server.

[0075] Whether the initial storage location information of the file to be written exists in the metadata server can be determined, for example, whether the path of the file to be written exists in the metadata server. If the path exists, it is considered that the file to be written is not a newly created file; if the path does not exist, it is determined that the file to be written is a newly created file.

[0076] In the embodiment, the metadata of the file to be written can also be searched from the metadata server, for example, the content information of the file to be written and the metadata of the file to be written can be forwarded to the intelligent proxy gateway by the FUSE in the virtual disk; and the metadata of the file to be written can be searched from the metadata server by the intelligent proxy gateway.

[0077] Specifically, the FUSE obtains the content information and the metadata of the file to be written, and forwards the content information and the metadata of the file to be written to the intelligent proxy gateway. The intelligent proxy gateway is disposed between the FUSE and the heterogeneous storage device, and can be used to realize protocol access between the electronic device and the heterogeneous storage device.

[0078] The intelligent proxy gateway can also realize data transmission between the electronic device and the metadata server. The intelligent proxy gateway obtains the metadata of the file to be written, and searches whether the metadata exists in the metadata server.

[0079] The beneficial effects of such an arrangement are that the FUSE driver can forward the obtained data to the intelligent proxy gateway, realize querying the metadata from the metadata server through the intelligent proxy gateway, and thus determine whether the file is repeatedly written, thereby improving the efficiency of data processing.

[0080] S303, if the initial storage location information of the file to be written does not exist in the metadata server, or if the initial storage location information exists in the metadata server and the preset writing condition is met, storing the content information of the file to be written into the preset heterogeneous storage device based on a preset interface through the cache server, to obtain the storage location information of the file to be written.

[0081] For example, if it is determined that the metadata of the file to be written does not exist in the metadata server, it indicates that the file to be written is a newly created file, and the content information of the file to be written can be stored into the heterogeneous storage device to obtain the storage location information of the file to be written, thereby avoiding duplication of file storage. For example, the content information in the cache server can be written into the heterogeneous storage device based on a preset interface. The metadata and the storage location information of the file to be written can also be stored into the metadata server. For example, a record can be created in the metadata server, and the record is the metadata and the storage location information of the file to be written. If the file is written again subsequently, the storage location information can be used as the initial storage location information.

[0082] If the initial storage location information exists in the metadata server, it can be further determined whether the initial storage location information meets the preset writing condition, and if so, the content information of the file to be written in the cache server can be stored into the preset heterogeneous storage device based on a preset interface. In the embodiment, the preset writing condition is not limited. For example, the preset writing condition can be a specific storage location information, and if the initial storage location information is not the specific storage location information, the preset writing condition is met.

[0083] In the embodiment, the metadata is stored in the metadata server, so that the metadata can be persisted. The meaning of metadata persistence is that as long as a file is written into a virtual disk, the file can be retrieved in the file system at all times, but actually the file can have been written into a tape, the tape has been offline, i.e., taken out of a tape library, and the local cache has also been cleared, and when the file is read, the tape needs to be put back into the tape library.

[0084] In the embodiment, the method further includes: after the content information of the file to be written is stored into the preset heterogeneous storage device, determining current storage location information of the content information of the file to be written in the preset heterogeneous storage device; if the initial storage location information of the file to be written exists in the metadata server, storing the current storage location information and the initial storage location information together into the metadata server, or replacing the initial storage location information with the current storage location information.

[0085] Specifically, if the metadata server contains initial storage location information and the preset write conditions are met, then the content information needs to be written to the heterogeneous storage device to obtain the current storage location information. In other words, the file to be written corresponds to two storage location information: the initial storage location information and the current storage location information.

[0086] The metadata server originally stores initial storage location information. Either both storage location information and the initial storage location information can be stored in the metadata server, or the initial storage location information can be replaced, recording only the current storage location information, i.e., the latest storage location information. In this embodiment, a storage strategy for the storage location information can be pre-set according to actual needs to determine whether to replace the initial storage location information.

[0087] The advantage of this setup is that if the metadata server already stores the storage location information of historical versions, the information at the storage location of historical versions can be retained or updated, improving the flexibility of data processing, facilitating data traceability, and enhancing the user experience.

[0088] S304. Update the storage location information of the file to be written to the metadata server.

[0089] This application provides a data processing method based on heterogeneous storage. A virtual disk is set up in the client device. The user opens the virtual disk and issues a file write command through it, for example, the user creates a file using the same operating method as a local disk. In response to the user's file write command, the content information of the file to be written is obtained and stored in a high-speed cache server. The metadata of the file to be written is also obtained and stored in a metadata server. A protocol connection exists between the electronic device and the remote heterogeneous storage device. The electronic device can store the content information of the file to be written in the remote heterogeneous storage device, allowing the user to perform localized file operations on the remote heterogeneous storage device as if writing a local file. The user is unaware of the heterogeneous storage device on the electronic device side, achieving consistency in user operations and shielding the differences in the underlying media. The storage location information of the file to be written is determined, and the metadata of the file to be written and its storage location information are associated and stored in the metadata server, achieving persistent storage of metadata, facilitating file management and access, and improving data processing efficiency.

[0090] Figure 4 This application provides a flowchart illustrating a data processing method based on heterogeneous storage, as shown in the embodiments below. Figure 4 As shown, in this embodiment... Figure 1 Based on the embodiments, a data processing method based on heterogeneous storage is described in detail, the method including:

[0091] S401, in response to a file write instruction issued by the client device to the virtual disk, storing content information of a file to be written in the virtual disk into the cache server, generating metadata of the file to be written, and storing the metadata of the file to be written into the metadata server; wherein the content information represents the content expressed by the file, and the metadata represents the content format and file attributes of the file; the client device is provided with the virtual disk.

[0092] Exemplarily, the cache pool of the cache server can be a storage medium in the form of a hard disk, for example, the cache pool can be a standard cache volume deployed in an electronic device, or a hard disk installed on an electronic device. After obtaining the content information of the file to be written, the content information of the file to be written can be stored into the cache server. The cache server is an online storage medium.

[0093] In the embodiment, it also includes: for the file in the cache server, obtaining file information of the file according to a preset time period; wherein the file information includes at least one of file read frequency, last access time, and file size; if the file information does not meet a preset retention condition, the file is deleted from the cache server.

[0094] Specifically, the cache server cannot perform persistent storage on all content information, and in the embodiment, the intelligent flow of data in the cache server is realized through a three-layer linkage mechanism under the premise of user unawareness. The three-layer linkage mechanism includes hot data acceleration, cold data settlement, and offline medium awakening. The hot data acceleration refers to that for frequently accessed files, the files can be cached in the cache server to reduce the read delay from minutes to microseconds; the cold data settlement refers to that for low-frequency access files, the files are removed from the cache server; the offline medium awakening refers to that when it is detected that the offline medium is reloaded, for example, the tape is returned to the library, the background migration can be automatically triggered, and the files recently queried are preferentially restored to the cache server. That is, the data in the cache server can be continuously updated.

[0095] A time period can be preset, and the cache server is updated regularly according to the preset time period. For example, the file information of each file in the cache server is obtained every preset time period. The file information can include various information categories such as file read frequency, last access time, file size, and occupied space.

[0096] For each file in the cache server, it is judged whether the file information of the file meets a preset retention condition. If yes, the content information of the file is retained in the cache server. If not, the content information of the file is deleted from the cache server. For example, the preset retention condition can be that the file reading frequency is greater than a preset frequency threshold. For a file whose file reading frequency is less than or equal to the frequency threshold, the file can be deleted from the cache server.

[0097] The beneficial effect of such a setting is that the files in the cache server are periodically cleaned up, avoiding redundant data occupying space, so that the files stored in the cache server are relatively important files, facilitating direct data reading from the cache server, thereby improving the efficiency of data reading.

[0098] In this embodiment, if the file information does not meet the preset retention condition, the file is deleted from the cache server, including: determining the weight information of the file according to the file information; wherein the weight information represents the importance of the file; if the weight information is less than a preset weight threshold, the file is deleted from the cache server.

[0099] Specifically, for a file in the cache server, the file information of the file is obtained, and the weight information of the file is determined according to the file information. The weight information can represent the importance of the file, for example, the greater the file reading frequency, the greater the weight information; the earlier the last access time, the smaller the weight information.

[0100] For each information category in the file information, a weight value can be preset, and the file information is weighted and summed according to the preset weight value to determine the calculation result as the weight information. A weight threshold is preset, and the weight information of the file is compared with the weight threshold. If the weight information is less than the preset weight threshold, the file is deleted from the cache server; if the weight information is equal to or greater than the preset weight threshold, the file is retained in the cache server.

[0101] The beneficial effect of such a setting is that by calculating the weight information, the importance of the file can be quantified, facilitating the management of the files in the cache server and improving the efficiency of subsequent file reading.

[0102] S402, the content information of the file to be written is stored in a preset heterogeneous storage device to obtain storage location information of the file to be written; wherein the preset heterogeneous storage device represents a device different from the storage medium of the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the position of the file in the preset heterogeneous storage device.

[0103] S403, update the storage location information of the file to be written to the metadata server.

[0104] S404, in response to the file reading instruction issued by the client device to the virtual disk, acquire the metadata of the file to be read; if the content information of the file to be read is not found in the cache server according to the metadata of the file to be read, find the storage location information corresponding to the metadata of the file to be read in the metadata server; and obtain the content information of the file to be read from the preset heterogeneous storage device according to the storage location information corresponding to the metadata of the file to be read.

[0105] For example, the user can not only perform a write operation through the virtual disk, but also perform a read operation through the virtual disk. For example, the user double-clicks a file in the virtual disk, which is a file reading instruction. The FUSE in the virtual disk responds to the file reading instruction issued by the user, determines the file to be read by the user, for example, the file double-clicked by the user can be determined as the file to be read, that is, the file to be read. Each file stores its own metadata. The metadata of the file to be read is acquired. In this embodiment, the read operation of the file is not necessarily after the write operation.

[0106] The FUSE forwards the metadata of the file to be read to the intelligent agent gateway, and the intelligent agent gateway searches in the cache server according to the metadata of the file to be read to determine whether the file to be read exists. For example, the metadata includes the storage path of the file in the virtual disk, and the files in the cache server store the metadata such as the storage path of the file in the virtual disk. According to the storage path of the file to be read, whether the content information of the file to be read exists in the cache server is found.

[0107] If the content information of the file to be read is found in the cache server, the content information of the file to be read in the cache server can be directly returned to the visual interface of the electronic device for the user to browse. For the user, it is that the user double-clicks a document, and the content of the document is directly opened on the page.

[0108] If the content information of the file to be read is not found from the cache server, metadata of the file to be read can be found from the metadata server, and the metadata of the file to be read corresponds to the storage location information according to the association relationship between the metadata in the metadata server and the storage location information. According to the storage location information corresponding to the metadata of the file to be read, the heterogeneous storage device where the file to be read is located is determined, and the content information of the file to be read is obtained from the heterogeneous storage device based on the preset interface, and returned to the visual interface of the electronic device. For the user, it is also that the user double-clicks a document, and directly opens the content of the document on the page

[0109] The beneficial effect of such a setting is that when accessing data, the cache server is preferentially searched, and if there is none in the cache server, the content information of the file to be read is obtained from the remote heterogeneous storage device according to the metadata in the metadata server and the storage location information. The user is not aware of the query process, and the user experience is improved.

[0110] In the embodiment, the content information of the file to be read is stored in the cache server.

[0111] Specifically, the user wants to access the file to be read, which means that the file is currently important to the user. However, the cache server does not have the content information of the file, so after obtaining the content information of the file to be read from the heterogeneous storage device, the content information of the file to be read can be stored in the cache server, which is convenient for subsequent direct access from the cache server. In the embodiment, the content information in the heterogeneous storage device can be migrated back to the cache pool without the need of the intelligent proxy gateway. The intelligent proxy gateway transmits metadata and content information between the client and each server, wherein the metadata is stored in the metadata server and the content information is stored in the cache server. The heterogeneous storage device is connected to the server end, and the content information can be directly written into the cache server during data migration.

[0112] The beneficial effect of such a setting is that for files in near-line media such as disk libraries and optical disk libraries, automatic loading can be triggered when the user accesses, and the content information is automatically migrated back to the online cache server; for files in offline media, the migration can be automatically triggered after the media is reinserted, that is, the content information is automatically migrated back to the online cache server, which is convenient for subsequent access and improves the efficiency of data processing.

[0113] In the embodiment, the preset heterogeneous storage device includes a pluggable storage medium; the method further includes: if it is determined that the storage location information in the preset heterogeneous storage device cannot be accessed according to the storage location information corresponding to the metadata of the file to be read, issuing a prompt information; wherein the prompt information is used to prompt the user to load the offline storage device.

[0114] Specifically, in the embodiment, a plurality of heterogeneous storage devices can be preset, for example, the heterogeneous storage devices can be a tape library, a Blu-ray disc library, etc. Different heterogeneous storage devices can correspond to different storage media. In the embodiment, the heterogeneous device generally refers to a nearline device, and some heterogeneous storage devices have pluggable storage media. Offline refers to taking out the tapes or optical discs in the nearline device, for example, for a tape library, the tape library can include pluggable tapes, and the taken-out tapes or optical discs are offline storage media.

[0115] After obtaining the storage location information, the file content information can be read from the corresponding heterogeneous storage device. If the corresponding storage medium exists in the heterogeneous storage device, the file content information can be read; if the corresponding storage medium does not exist in the heterogeneous storage device, the file content information cannot be read. That is, if the storage location information in the heterogeneous storage device cannot be accessed, it means that the storage medium of the storage location information is unplugged, and a prompt information can be issued on the electronic device to prompt the user to load the offline storage medium. For example, the user can be reminded to insert a tape or a disc. In the embodiment, a mechanical hand can also be used to automatically insert a tape or a disc.

[0116] The beneficial effect of such a setting is that for offline media, the user can be reminded to load in time to read the file, realize the interaction with the user in the data reading process, improve the efficiency of data reading, and improve the user experience.

[0117] In the embodiment, an intelligent agent gateway is deployed in the electronic device; the method further includes: after obtaining the content information of the file to be read, the content information of the file to be read is encrypted by the intelligent agent gateway to obtain a second hash value of the file to be read; wherein the second hash value represents the content information of the file; if a first hash value consistent with the second hash value is found, it is determined that the content information of the file to be read is verified; wherein the first hash value is a hash value representing the content information of the file generated when the file is written.

[0118] Specifically, when the file to be read is written, the intelligent agent gateway calculates a first hash value of the file to be read through a preset encryption algorithm. When the user needs to read the file, the intelligent agent gateway obtains content information of the file to be read, encrypts the content information of the file to be read based on the preset encryption algorithm, and obtains a second hash value of the file to be read. The encryption algorithms used to generate the first hash value and the second hash value can be the same.

[0119] The intelligent agent gateway compares the generated second hash value with all the first hash values determined previously. If a first hash value consistent with the second hash value is found, it is determined that the content information of the file to be read is complete, and the content information of the file to be read is verified. If there is no first hash value consistent with the second hash value, it is determined that the content information of the file to be read fails to be verified, and the content information of the file to be read may be tampered with or lost, and the user can be prompted to confirm.

[0120] The beneficial effect of such a setting is that the first hash value and the second hash value are used to verify the integrity of the file content, thereby improving the security of data processing.

[0121] Figure 5 A schematic diagram of the data reading process is shown in FIG. 5. As shown in FIG. 5, read is a read operation, the client application can be a virtual disk on an electronic device, and the cache pool is a cache server in the embodiment, which can be in the form of an SSD (Solid State Drive) and is an online storage medium. Figure 5 The process is as follows:

[0122] 1. The user makes a read operation on the client application.

[0123] 2. The FUSE drive layer responds to the read operation and determines the metadata of the file to be read, and forwards the metadata to the intelligent agent gateway.

[0124] 3. The intelligent agent gateway checks whether the file to be read has been cached in the cache pool.

[0125] 4. If it is hit in the cache pool, the file to be read is returned to the intelligent agent gateway; if it is not hit, the result of the miss is fed back to the intelligent agent gateway.

[0126] 5. The intelligent agent gateway receives the result of the miss, and finds the storage location information corresponding to the metadata from the MongoDB based on the metadata. The MongoDB is a metadata server in the embodiment.

[0127] 6. The MongoDB returns the storage location information to the intelligent agent gateway.

[0128] 7、The intelligent gateway agent acquires the content information of the file to be read from the heterogeneous storage device according to the storage location information. It is worth noting that the cache pool is an online medium, and the heterogeneous storage device can be a nearline or offline medium. That is, the content information can be acquired from the nearline or offline heterogeneous storage device.

[0129] 8、The heterogeneous storage device writes the content information back to the cache pool.

[0130] 9、The content information is displayed on the interface of the client through the intelligent gateway agent.

[0131] The embodiment of the application provides a data processing method based on heterogeneous storage. A virtual disk is arranged in an electronic device, and a FUSE is arranged in the virtual disk. A user opens the virtual disk, and issues a file write instruction through the virtual disk. For example, the user writes a file by creating a new document in a manner consistent with a local disk. The FUSE in the virtual disk responds to the file write instruction issued by the user, and acquires content information of the file to be written and metadata of the file to be written. There is a protocol connection between the electronic device and a remote heterogeneous storage device. The electronic device can store the content information of the file to be written in the remote heterogeneous storage device, so that the user can perform a localized file operation on the remote heterogeneous storage device in a manner consistent with writing a local file. The user has no perception of the heterogeneous storage device on the side of the electronic device, the consistency of user operation is achieved, and the difference between underlying media is shielded. Storage location information of the file to be written is determined, the metadata of the file to be written and the storage location information of the file to be written are stored in a metadata server in association, and the metadata is stored persistently, so that the file is managed and accessed conveniently, and the efficiency of data processing is improved.

[0132] Figure 6 A structure diagram of a data processing device based on heterogeneous storage provided by the embodiment of the application is shown in FIG. 1. As shown in FIG. 1, the data processing device based on heterogeneous storage 60 provided by the embodiment of the application includes: Figure 6

[0133] a first response unit 601, configured to store content information of a file to be written in a virtual disk in a cache server in response to a file write instruction issued by a client device to the virtual disk, generate metadata of the file to be written, and store the metadata of the file to be written in a metadata server; wherein the content information represents content expressed by the file, and the metadata represents a content format and file attributes of the file; and the client device is provided with the virtual disk.

[0134] ​The position determining unit 602 is configured to store the content information of the file to be written into a preset heterogeneous storage device, to obtain storage position information of the file to be written; wherein the preset heterogeneous storage device represents a device different from a storage medium of the electronic device, and the storage position information represents a preset heterogeneous storage device where the file is located and a position of the file in the preset heterogeneous storage device.

[0135] The data storage unit 603 is configured to update the storage position information of the file to be written into a metadata server.

[0136] In a possible implementation, the position determining unit 602 comprises:

[0137] The metadata searching module is configured to search, from the metadata server, initial storage position information of the file to be written.

[0138] The content writing module is configured to, if the initial storage position information of the file to be written does not exist in the metadata server, or if the initial storage position information exists in the metadata server and a preset writing condition is met, store, by the cache server, the content information of the file to be written into the preset heterogeneous storage device based on a preset interface.

[0139] In a possible implementation, the method further comprises:

[0140] The current position determining unit is configured to, after the content information of the file to be written is stored into the preset heterogeneous storage device, determine current storage position information of the content information of the file to be written in the preset heterogeneous storage device.

[0141] The position storage unit is configured to, if the initial storage position information of the file to be written exists in the metadata server, store the current storage position information and the initial storage position information together into the metadata server, or replace the initial storage position information with the current storage position information.

[0142] In a possible implementation, the first response unit 601 comprises:

[0143] The block dividing module is configured to acquire size information of a file to be written in a virtual disk, and perform block dividing processing on the content information of the file to be written if the size information is greater than a preset first threshold, to obtain at least two file blocks; wherein the at least two file blocks are combined as the content information of the file to be written; and the at least two file blocks are uploaded to the cache server in parallel for storage.

[0144] In a possible implementation, the first response unit 601 comprises:

[0145] an aggregating module, configured to, if there are at least two files to be written and size information of each file to be written is less than a preset second threshold, perform aggregation processing on content information of the at least two files to be written to obtain an aggregated block; the aggregated block represents all content information of the at least two files to be written; and the aggregated block is uploaded to the cache server for storage.

[0146] In a possible implementation, the first response unit 601 comprises:

[0147] an authority obtaining module, configured to, in response to a file write instruction issued by a client device to a virtual disk, obtain authority information of the user; the authority information represents an operation authority of the user on a file;

[0148] an authority judging module, configured to, if it is determined according to the authority information of the user that the user has the authority of file write operation, store content information of a file to be written in the virtual disk into the cache server.

[0149] In a possible implementation, the authority judging module is specifically configured to:

[0150] if it is determined according to the authority information of the user that the user has the authority of file write operation, obtain size information of the file to be written in the virtual disk and remaining space information of the virtual disk; the remaining space information represents a remaining space in a bucket size corresponding to the virtual disk configured for the user in advance;

[0151] if the size information of the file to be written is less than or equal to the remaining space information, store the content information of the file to be written into the cache server.

[0152] In a possible implementation, the method further comprises:

[0153] a cache cleaning unit, configured to, for a file in the cache server, obtain file information of the file according to a preset time period; the file information comprises at least one of a file read frequency, a last access time and a file size; and if the file information does not satisfy a preset retention condition, delete the file from the cache server.

[0154] In a possible implementation, the cache cleaning unit is specifically configured to:

[0155] determine weight information of the file according to the file information; the weight information represents an importance of the file;

[0156] If the weight information is less than a preset weight threshold, the file is deleted from the cache server.

[0157] In a possible implementation, the method further includes:

[0158] The second response unit is configured to, in response to a file reading instruction issued by the client device to the virtual disk, acquire metadata of a file to be read;

[0159] The cache searching unit is configured to, if the content information of the file to be read is not found from the cache server according to the metadata of the file to be read, search, from the metadata server, storage location information corresponding to the metadata of the file to be read.

[0160] The file reading unit is configured to, according to the storage location information corresponding to the metadata of the file to be read, obtain the content information of the file to be read from a preset heterogeneous storage device.

[0161] In a possible implementation, the method further includes:

[0162] The content caching unit is configured to store the content information of the file to be read into the cache server.

[0163] In a possible implementation, the preset heterogeneous storage device includes a pluggable storage medium; and the method further includes:

[0164] The prompting unit is configured to, if it is determined that the storage location information in the preset heterogeneous storage device cannot be accessed according to the storage location information corresponding to the metadata of the file to be read, issue a prompt information; wherein the prompt information is used to prompt a user to load a storage medium.

[0165] The data processing apparatus based on heterogeneous storage provided in this embodiment can execute the method provided in the method embodiment, and has similar implementation principles and technical effects, which will not be described here in detail.

[0166] The embodiment of the present application provides a heterogeneous storage virtual file system, which can be used to execute the data processing method based on the heterogeneous storage in the embodiment of the present application. The system realizes the standardized access, unified management of metadata and transparent access across networks of heterogeneous storage devices such as object storage, tape library, optical disc library and the like. The file queryability is maintained in the offline state of the medium, and the data back migration is automatically triggered after the accessed nearline or offline medium is reloaded to the heterogeneous storage device. In the heterogeneous storage virtual file system, the native protocol of the heterogeneous storage device such as block storage, object storage, tape library, optical disc library and the like can be converted into a unified interface, and the cross-network mounting can be performed through a high-speed gateway, so as to realize the remote localization file operation of various heterogeneous media, thereby enabling the system to be used to solve the problems of multi-media compatibility, nearline and offline data accessibility and user experience consistency and the like.

[0167] Figure 7 A structural schematic diagram of an electronic device provided by the embodiment of the present application is provided. As shown in the figure, Figure 7 The electronic device 70 provided by the embodiment of the present application includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected through a bus 704.

[0168] In the specific implementation process, the at least one processor 701 executes the computer execution instruction stored in the memory 702, so that the at least one processor 701 executes the method described above.

[0169] The specific implementation process of the processor 701 can refer to the method embodiment described above, which has similar implementation principles and technical effects, and will not be described here in detail.

[0170] In the above embodiment, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC) and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The steps of the method disclosed in the application can be directly embodied as the execution of the hardware processor, or the execution of the combination of the hardware and software modules in the processor.

[0171] The memory can include a Random Access Memory (RAM) and can also include a Non-volatile Memory (NVM), such as at least one disk memory.

[0172] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0173] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above method.

[0174] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the above method is implemented.

[0175] The above readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0176] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0177] The division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0178] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0179] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0180] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0181] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various program code storage media.

[0182] It should be understood that many of the materials and devices exemplified in this disclosure are articles of manufacture (i.e., articles of manufacture) according to this disclosure. The articles of manufacture can be manufactured as such or can be manufactured by combining the materials and devices exemplified in this disclosure. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It should be understood that, in some embodiments, equivalents to the specific electrode structures and / or methods described herein can be employed without departing from the scope of the application. Accordingly, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized into," and variations thereof herein, is meant to encompass the items listed thereafter, and equivalents thereof as well as additional items. Although the foregoing application has been described in some detail by way of illustration and example, it is not to be limited thereby, but rather, only by the scope of the appended claims.

Claims

1. A data processing method based on heterogeneous storage, characterized in that, The method comprises the following steps: In response to a file write instruction issued by a client device to a virtual disk, storing content information of a file to be written in the virtual disk into a cache server, generating metadata of the file to be written, and storing the metadata of the file to be written into a metadata server; wherein the content information represents the content expressed by the file, and the metadata represents the content format and file attributes of the file; the client device is provided with a virtual disk; Storing the content information of the file to be written into a preset heterogeneous storage device to obtain storage location information of the file to be written; wherein the preset heterogeneous storage device represents a device different from the storage medium of the electronic device, and the storage location information represents the preset heterogeneous storage device where the file is located and the location of the file in the preset heterogeneous storage device; Updating the storage location information of the file to be written into the metadata server.

2. The method of claim 1, wherein, Storing the content information of the file to be written into a preset heterogeneous storage device comprises the following steps: Finding initial storage location information of the file to be written from the metadata server; If the initial storage location information of the file to be written does not exist in the metadata server, or if the initial storage location information exists in the metadata server and a preset write condition is met, storing the content information of the file to be written into a preset heterogeneous storage device based on a preset interface through the cache server.

3. The method of claim 2, wherein, After storing the content information of the file to be written into a preset heterogeneous storage device, the method further comprises the following steps: Determining the current storage location information of the content information of the file to be written in the preset heterogeneous storage device; If the initial storage location information of the file to be written exists in the metadata server, storing the current storage location information and the initial storage location information together in the metadata server, or replacing the initial storage location information with the current storage location information.

4. The method of claim 1, wherein, Storing the content information of the file to be written in the virtual disk into the cache server comprises the following steps: Obtaining size information of the file to be written in the virtual disk, and if the size information is greater than a preset first threshold, performing block processing on the content information of the file to be written to obtain at least two file blocks; wherein the at least two file blocks are combined as the content information of the file to be written; Uploading the at least two file blocks to the cache server in parallel for storage.

5. The method of claim 1, wherein, Storing the content information of the file to be written in the virtual disk into the cache server comprises the following steps: If there are at least two files to be written, and the size information of each file to be written is less than a preset second threshold, performing aggregation processing on the content information of the at least two files to be written to obtain an aggregation block; wherein the aggregation block represents the total content information of the at least two files to be written; Uploading the aggregation block to the cache server for storage.

6. The method of claim 1, wherein, In response to a file write instruction issued by the client device to the virtual disk, storing content information of a file to be written in the virtual disk into the cache server, comprising: In response to a file write instruction issued by the client device to the virtual disk, obtaining the permission information of the user; wherein the permission information represents the operation permission of the user on the file; If it is determined that the user has the permission of file write operation according to the permission information of the user, then storing the content information of the file to be written in the virtual disk into the cache server.

7. The method of claim 6, wherein, If it is determined that the user has the permission of file write operation according to the permission information of the user, then storing the content information of the file to be written in the virtual disk into the cache server, comprising: If it is determined that the user has the permission of file write operation according to the permission information of the user, then obtaining the size information of the file to be written in the virtual disk and the remaining space information of the virtual disk; wherein the remaining space information represents the remaining space in the bucket size corresponding to the virtual disk configured for the user in advance; If the size information of the file to be written is less than or equal to the remaining space information, then storing the content information of the file to be written into the cache server.

8. The method of claim 1, wherein, Further comprising: For the file in the cache server, obtaining file information of the file according to a preset time period; wherein the file information comprises at least one of file read frequency, last access time, and file size; If the file information does not satisfy a preset retention condition, then deleting the file from the cache server.

9. The method of claim 8, wherein, If the file information does not satisfy a preset retention condition, then deleting the file from the cache server, comprising: According to the file information, determining weight information of the file; wherein the weight information represents the importance of the file; If the weight information is less than a preset weight threshold, then deleting the file from the cache server.

10. The method of claim 1, wherein, Further comprising: In response to a file read instruction issued by the client device to the virtual disk, obtaining metadata of a file to be read; If the content information of the file to be read is not found from the cache server according to the metadata of the file to be read, then finding storage location information corresponding to the metadata of the file to be read from the metadata server; According to the storage location information corresponding to the metadata of the file to be read, obtaining the content information of the file to be read from a preset heterogeneous storage device.

11. The method of claim 10, wherein, Further comprising: Storing the content information of the file to be read into the cache server.

12. The method of claim 10, wherein, The preset heterogeneous storage device comprises a pluggable storage medium; the method further comprises: If it is determined that the storage location information in the preset heterogeneous storage device cannot be accessed according to the storage location information corresponding to the metadata of the file to be read, then issuing a prompt information; wherein the prompt information is used to prompt the user to load the storage medium.

13. A data processing apparatus based on heterogeneous storage, characterized in that, Comprising: The first response unit is configured to store content information of a file to be written in the virtual disk into the cache server, generate metadata of the file to be written, and store the metadata of the file to be written into the metadata server in response to a file write instruction issued by the client device to the virtual disk; wherein the content information represents content expressed by the file, and the metadata represents content format and file attributes of the file; and the client device is provided with the virtual disk. The position determination unit is configured to store the content information of the file to be written into a preset heterogeneous storage device to obtain storage position information of the file to be written; wherein the preset heterogeneous storage device represents a device different from a storage medium of the electronic device, and the storage position information represents a preset heterogeneous storage device where the file is located and a position of the file in the preset heterogeneous storage device. The data storage unit is configured to update the storage position information of the file to be written into the metadata server.

14. An electronic device, comprising: The electronic device comprises: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the method according to any one of claims 1-12.

15. A computer readable storage medium characterized by: The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method according to any one of claims 1-12.

16. A computer program product, characterised in that, The computer program is executed by the processor to implement the method according to any one of claims 1-12.

Images