A data operation method, device, equipment and medium

Abstract

The present disclosure relates to a data operation method, device, equipment and medium, the method is applied to a data node, including: determining a hard disk connected to the data node, and reporting the hard disk information to a metadata server, wherein the hard disk information includes the hard disk medium type, available capacity and location information; receiving a data operation instruction sent by the metadata server, wherein the data operation instruction carries the medium type and location information of the hard disk where the operated data block is located; according to the medium type, calling a service unit corresponding to the medium type to perform a data block operation according to the location information of the hard disk. The present disclosure simplifies the use of a distributed storage cluster and reduces the difficulty of cluster deployment and maintenance.

Description

Technical Field

[0001] The present disclosure relates to the field of data operation technology, and in particular to a data operation method, device, equipment and medium. Background Art

[0002] In video surveillance, the real-time business data stream generated by the front-end video / image acquisition equipment needs to be connected to the distributed storage cluster in a timely manner. The specific method is: first, the metadata server will divide the user file into several data blocks of the same size according to the data redundancy rule; then, the metadata server will select data nodes with the same number of data blocks as the file after segmentation according to the load balancing strategy to save the data blocks of the business data stream; finally, after receiving the business data block, the data node will use the load balancing strategy again to select one of the hard disks to save the data block.

[0003] It can be seen that in the existing distributed storage cluster, the metadata server is responsible for maintaining "user file names, names of each data block after segmentation, and the data node ID (Identity Document, identity identification number) where these data blocks are located"; the data node is responsible for maintaining "data blocks and the location information of the data blocks in the hard disk" and managing the hard disk space. Through two-layer load balancing, fault isolation between data nodes and hard disks can be achieved, but there is a certain degree of logical redundancy.

[0004] With the diversification of security monitoring services, the type of data written into the distributed storage cluster is no longer single. Since the metadata server cannot sense the type of hard disk connected to the data node, it is unable to deliver different types of business data to the appropriate hard disk. Summary of the invention

[0005] The present disclosure provides a data operation method, apparatus, device and medium, which mix and connect CMR (Conventional Magneting Recording) hard disks and SMR (Shingled Magneting Recording) hard disks in data nodes to achieve cluster-level load balancing, data reading and writing, and management, simplify the use of distributed storage clusters, and reduce the difficulty of cluster deployment and maintenance.

[0006] According to a first aspect of an embodiment of the present disclosure, a data operation method is provided, which is applied to a data node, and the method includes:

[0007] Determine a hard disk connected to the data node, and report information of the hard disk to a metadata server, wherein the information of the hard disk includes a media type, available capacity, and location information of the hard disk;

[0008] receiving a data operation instruction sent by the metadata server, wherein the data operation instruction carries the media type and location information of the hard disk where the operated data block is located;

[0009] According to the medium type, a service unit corresponding to the medium type is called to perform a data block operation according to the location information of the hard disk.

[0010] In a possible implementation manner, before determining the hard disk connected to the data node, the method further includes:

[0011] In response to a hard disk access instruction, determining a medium type of the hard disk;

[0012] According to the medium type of the hard disk and the corresponding relationship between different medium types and service units, the hard disk is associated with the corresponding service unit.

[0013] In a possible implementation, the data operation instruction includes a storage operation instruction carrying a data block to be stored, and the calling, according to the medium type, of a service unit corresponding to the medium type to perform a data block operation according to the hard disk location information includes:

[0014] According to the storage operation instruction, determining the hard disk where the data block in the storage operation instruction is located;

[0015] According to the determined medium type of the hard disk, a service unit corresponding to the medium type is called to store data blocks.

[0016] In a possible implementation, after invoking a service unit corresponding to the medium type to store data blocks according to the determined medium type of the hard disk, the method further includes:

[0017] Determine the hard disk corresponding to the data block that has not been successfully stored, and store the data block that has not been successfully stored to another hard disk of a corresponding medium type according to the instruction of the metadata server, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that has not been successfully stored belongs;

[0018] The hard disk corresponding to the data block successfully stored in the data file is determined, and the location information of the hard disk corresponding to the data block successfully stored is returned to the metadata server.

[0019] In a possible implementation, associating the hard disk with a corresponding service unit according to the medium type of the hard disk and the correspondence between different medium types and service units includes:

[0020] Determining that the medium type of the hard disk is a conventional magnetic recording CMR hard disk, and associating the hard disk with a CMR hard disk service unit;

[0021] It is determined that the medium type of the hard disk is a shingled magnetic recording (SMR) hard disk, and the hard disk is associated with an SMR hard disk service unit.

[0022] According to a second aspect of an embodiment of the present disclosure, a data operation method is provided, which is applied to a metadata server, and the method includes:

[0023] Obtaining hard disk information reported by the data node, wherein the hard disk information includes a media type, available capacity, and location information of the hard disk;

[0024] In response to a user's data operation instruction, determining the media type and location information of the hard disk to be operated;

[0025] The determined medium type and location information of the hard disk to be operated is indicated to the data node through a data operation instruction.

[0026] In a possible implementation, the determining, in response to a data operation instruction from a user, the media type and location information of the hard disk to be operated includes:

[0027] In response to the storage operation instruction, determining a data block to be stored;

[0028] Determine that the data block to be stored belongs to the first data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the first medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node;

[0029] Determine that the data block to be stored belongs to the second data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the second medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node;

[0030] The priority strategy is to select a hard disk of a corresponding media type according to the type of data block to be stored. If the available capacity of the hard disk of the corresponding media type is smaller than the size of the data block to be stored, select a hard disk of another media type.

[0031] In a possible implementation, the first data file type is a data file with an indefinite life cycle and an occupied hard disk capacity less than a set threshold, the second data file type is a data file with a fixed life cycle and an occupied hard disk capacity greater than a set threshold, the first media type hard disk is a CMR hard disk, and the second media type hard disk is an SMR hard disk.

[0032] In a possible implementation, the data operation instruction includes a storage operation instruction carrying a data block to be stored, and after indicating the determined medium type and location information of the hard disk to be operated to the data node through the data operation instruction, the method further includes:

[0033] According to the hard disk corresponding to the data block that was not successfully stored fed back by the data node, instructing the data block that was not successfully stored to be stored in another hard disk of a corresponding medium type, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that was not successfully stored belongs;

[0034] According to the hard disk corresponding to the successfully stored data block fed back by the data node, the location information of the hard disk corresponding to the successfully stored data block returned by the data node is obtained;

[0035] A user file relationship table is established according to the location information of the hard disk corresponding to the successfully stored data block returned by the data node, wherein the user relationship table includes the data file name, the data block name and the location information of the successfully stored hard disk.

[0036] According to a third aspect of an embodiment of the present disclosure, a data operation device is provided, the device comprising:

[0037] A hard disk information reporting module, used to determine the hard disk connected to the data node through the data node, and report the hard disk information to the metadata server, wherein the hard disk information includes the media type, available capacity and location information of the hard disk;

[0038] A data operation instruction receiving module, used to receive a data operation instruction sent by the metadata server through a data node, wherein the data operation instruction carries the media type and location information of the hard disk where the operated data block is located;

[0039] The data block operation module is used to call the service unit corresponding to the media type through the data node according to the media type to perform data block operation according to the location information of the hard disk.

[0040] According to a fourth aspect of an embodiment of the present disclosure, a data operation device is provided, the device comprising:

[0041] A hard disk information acquisition module, used to acquire hard disk information reported by a data node through a metadata server, wherein the hard disk information includes a media type, available capacity, and location information of the hard disk;

[0042] A data operation instruction execution module, used to determine the media type and location information of the hard disk to be operated in response to the user's data operation instruction through the metadata server;

[0043] The data operation instruction sending module is used to indicate the determined medium type and location information of the hard disk to be operated to the data node through the metadata server through the data operation instruction.

[0044] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device comprising: a processor; and a memory for storing processor executable instructions; wherein the processor implements the steps of the above-mentioned data operation method by running the executable instructions.

[0045] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the above-mentioned data operation method are implemented.

[0046] In addition, the technical effects brought about by any implementation method in the third to sixth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, or refer to the technical effects brought about by different implementation methods in the second aspect, and will not be repeated here.

[0047] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0048] First, it adds SMR hard disk management capabilities to data nodes, supports mixed access of SMR hard disks and CMR hard disks, eliminates the need for physical isolation during cluster deployment, and reduces the difficulty of maintaining distributed storage clusters.

[0049] Second, the data node will report the type of hard disk it is connected to to the metadata server, which will maintain the "user file name, the name of the data block after segmentation, and the location information of the hard disk of the data node where each data block is located", thereby improving the availability of the distributed storage cluster in various business scenarios and the writing performance of the data node, and reducing the waste of resources in maintaining multiple data relationships;

[0050] Third, after the data node reports the type of hard disk connected to the metadata server, the distributed storage cluster selects the appropriate storage medium when processing business data of different life cycles, which can reasonably utilize the cluster capacity, improve read and write performance, and reduce the difficulty of data management and space recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 is a schematic diagram of an application scenario according to an exemplary embodiment;

[0053] Figure 2 is a flow chart showing a data operation method applied to a data node according to an exemplary embodiment;

[0054] Figure 3 is a flow chart showing hard disk access in a data operation method according to an exemplary embodiment;

[0055] Figure 4 is a flow chart showing a data operation method applied to a metadata server according to an exemplary embodiment;

[0056] Figure 5 is a flowchart showing data block storage in a data operation method according to an exemplary embodiment;

[0057] Figure 6 is a schematic diagram of a module structure of a data operation device according to an exemplary embodiment;

[0058] Figure 7 is a schematic diagram of a module structure of a data operation device according to an exemplary embodiment;

[0059] Figure 8 is a schematic diagram of an electronic device showing a data operation method according to an exemplary embodiment;

[0060] Fig. 9 is a schematic diagram of an electronic device showing a data operation method according to an exemplary embodiment;

[0061] Fig.10 It is a schematic diagram of a program product showing a data operation method according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0063] The following are explanations of some of the terms that appear in the text:

[0064] 1. In the embodiments of the present disclosure, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0065] 2. The terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein.

[0066] 3. The term "CMR hard disk" in the embodiments of the present disclosure refers to a mechanical hard disk that uses traditional magnetic recording to manage data, and is currently the mainstream mechanical hard disk medium. CMR hard disks support random read, write, and delete, and can be used to construct a general file system to manage hard disk files; however, the hard disk capacity is small, and the use of the file system will occupy a certain amount of read and write bandwidth, and the performance of sequential writes will be reduced.

[0067] 4. The term "SMR hard disk" in the embodiments of the present disclosure refers to a large-capacity, cost-effective hard disk that improves storage density by reducing the sector spacing. It supports continuous reading, writing, deletion, and random reading, but does not support random writing and deletion. The business layer is required to perform continuous writing and deletion of the entire zone data in units of zones. The current common zone length is 256 megabytes (MB). SMR hard disks have large storage space and perform better than CMR disks in sequential writing scenarios; however, since they do not support random writing and deletion, it is impossible to use a general file system to manage hard disk files. In business scenarios where file life cycles are inconsistent, the recovery of hard disk space will take up a large amount of system resources, thereby affecting read and write performance.

[0068] 5. The term "distributed storage cluster" in the embodiments of the present disclosure refers to a cluster where multiple servers provide data operation services. There are multiple implementation forms. In the present disclosure, it specifically refers to a distributed storage cluster composed of "metadata servers, data nodes, and hard disks." Among them, the metadata server is responsible for managing the data nodes in the distributed storage cluster and maintaining the distribution relationship of user data. The data node is responsible for managing large-capacity mechanical hard disks. In the present disclosure, mechanical hard disks specifically refer to SMR and CMR hard disks connected to the data nodes, and store the data block storage information in the specified location of the hard disk.

[0069] 6. The term "data block" in the embodiments of the present disclosure is "data block storage information", which refers to the data stored in the data node hard disk after the user's data file is segmented. When the business layer reads and writes data files, it needs to follow the data redundancy rules supported by the metadata server, so that the user's original data file is segmented into data blocks of the same size and then stored in the corresponding data node hard disk; or after reading the data block from the data node hard disk, the user data file is restored according to the data redundancy rules.

[0070] 7. In the embodiments of the present disclosure, the term "life cycle of user data files" refers to the field of security video surveillance. In order to use the distributed storage cluster quickly and conveniently, when the distributed storage cluster is connected to the front-end video / image service, a life cycle will be set for each data file, that is, the number of hours that the data file can be saved in the distributed storage cluster. Once the storage time of the data file exceeds the life cycle set by the user, the distributed storage cluster will actively delete the data files when the capacity is insufficient to achieve the reuse of storage space.

[0071] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0072] The problems existing in the data operation method in the prior art are as follows:

[0073] In the existing distributed storage cluster, the metadata server is responsible for maintaining "user file names, names of each data block after segmentation, and the data node ID where these data blocks are located"; the data node is responsible for maintaining "data blocks and the location information of the data blocks in the hard disk" and managing the hard disk space. Through two-layer load balancing, fault isolation between data nodes and hard disks can be achieved, but there is a certain degree of logical redundancy.

[0074] With the diversification of security monitoring services, the type of data written into the distributed storage cluster is no longer single. Since the metadata server cannot perceive the type of hard disk connected to the data node, it is unable to deliver different types of business data to the appropriate hard disk.

[0075] In order to solve the above problems, the present disclosure provides a data operation method, device, equipment and medium, which mix and connect hard disks of different media types in the data node to achieve cluster-level load balancing, data reading and writing and management, simplify the use of distributed storage clusters, and reduce the difficulty of cluster deployment and maintenance.

[0076] First reference Figure 1 , which is a schematic diagram of an application scenario of an embodiment of the present disclosure. A user 10 logs in to a metadata server 12 in a distributed storage cluster storage system through a client software installed in a user device 11, wherein the client may be a web browser or an application client installed in a mobile user device such as a mobile phone, a tablet computer, etc.

[0077] The user device 11 and the metadata server 12 are connected to each other through a network, which may be a local area network, a wide area network, etc. The user device 11 may be a portable device (e.g., a mobile phone, a tablet, a laptop, etc.) or a personal computer (PC), and the metadata server 12 may be any device that can provide Internet services.

[0078] One possible form of communication between the user device 11 and the metadata server 12 is that the user 10 can log in to the corresponding data operation platform, send the user's data operation instructions to the metadata server 12 through the communication network, and according to the data operation instructions, the data node 13 connected to the metadata server 12 performs corresponding data operations.

[0079] In the disclosed embodiment, the data node 13 determines the hard disk connected to the data node, and reports the information of the hard disk to the metadata server, wherein the information of the hard disk includes the media type, available capacity and location information of the hard disk; receives the data operation instruction sent by the metadata server, wherein the data operation instruction carries the media type and location information of the hard disk where the operated data block is located; according to the media type, calls the service unit corresponding to the media type to perform the data block operation according to the location information of the hard disk. The metadata server 12 obtains the information of the hard disk reported by the data node, wherein the information of the hard disk includes the media type, available capacity and location information of the hard disk; in response to the user's data operation instruction, determines the media type and location information of the hard disk to be operated; indicates the determined media type and location information of the hard disk to be operated to the data node through the data operation instruction.

[0080] In an embodiment of the present disclosure, a data operation method is provided. Based on the same concept, the present disclosure also provides a data operation device, an electronic device and a computer-readable storage medium.

[0081] Example 1

[0082] A data operation method provided by the present disclosure is described below through a specific embodiment, which is applied to a data node, such as Figure 2 As shown, including:

[0083] Step 201, determining a hard disk connected to the data node, and reporting information of the hard disk to a metadata server, wherein the information of the hard disk includes a media type, available capacity, and location information of the hard disk;

[0084] Specifically, the location information of the hard disk includes the identity identifier (dnID) of the data node where the hard disk is located and the identity identifier (diskID) assigned to the hard disk by the data node after the hard disk is successfully connected. The location information of each hard disk is unique within the distributed storage cluster. A data node can connect to multiple hard disks of different media types, but a hard disk can only connect to one data node. Compared with the existing requirement that data nodes can only connect to hard disks of the same media type, there is no need for physical isolation during cluster deployment, which reduces the difficulty of maintaining the distributed storage cluster.

[0085] Step 202, receiving a data operation instruction sent by the metadata server, wherein the data operation instruction carries the media type and location information of the hard disk where the operated data block is located;

[0086] Specifically, the data block may be obtained by correspondingly segmenting the data file to be operated by the user, or may be obtained by performing other operations on the data file to be operated by the user. The data operation instructions include deletion operation instructions, storage operation instructions, data reading operation instructions, and write operation instructions. The storage operation instruction carries not only the medium type and location information of the hard disk where the operated data block is located, but also the relevant information of the data block.

[0087] Step 203: According to the medium type, a service unit corresponding to the medium type is called to perform a data block operation according to the location information of the hard disk.

[0088] Specifically, the service unit refers to a program unit that provides management methods corresponding to the media types for hard disks of different media types. The management methods include hard disk access, reading, writing, deleting and formatting.

[0089] The data node in the present disclosure supports simultaneous access to and management of hard disks of different media types. Hard disks of each media type are managed by a service unit corresponding to the media type. The data node reports the media type of the hard disk to the metadata server. The data node shields the differences in operating data in hard disks of different media types from the business layer, reduces the difficulty of system maintenance, and facilitates on-site implementation.

[0090] In the related technology, data blocks are obtained from the user's data files and stored in the hard disk directory of the data node in the form of files of the general file system. The data management method based on erasure code is also implemented in the form of files. However, since SMR hard disks cannot be written and deleted randomly, it is impossible to build a general file system on SMR hard disks, and it is impossible to connect CMR hard disks and SMR hard disks to the data nodes of the existing cluster at the same time.

[0091] As an optional implementation manner, before determining the hard disk connected to the data node, the method further includes:

[0092] In response to a hard disk access instruction, determining a medium type of the hard disk;

[0093] According to the medium type of the hard disk and the corresponding relationship between different medium types and service units, the hard disk is associated with the corresponding service unit.

[0094] Specifically, when a hard disk is connected to a data node, its media type, available capacity and other information will be reported to the data node. The data node determines the media type of the hard disk based on this information, and thus associates the connected hard disk with the corresponding service unit.

[0095] As an optional implementation manner, the medium type of the hard disk in the embodiment of the present application includes a CMR hard disk and an SMR hard disk, and the service unit corresponding to the hard disk is associated according to the medium type of the hard disk and the correspondence between different medium types and service units, including:

[0096] Determine that the medium type of the hard disk is a CMR hard disk, and associate the hard disk with a CMR hard disk service unit. That is, when the data node senses that the CMR hard disk is online, it formats it into a general file system, and the data blocks of the business layer are saved in the hard disk directory in the form of files, supporting the deletion of any file and the release of space;

[0097] Determine that the medium type of the hard disk is an SMR hard disk, and associate the hard disk with an SMR hard disk service unit. That is, when the data node senses that the SMR hard disk is online, the SMR hard disk is directly read, written and managed through system call instructions, and data blocks are deleted and space is reclaimed in units of zones.

[0098] Figure 3 FIG. 1 is a flowchart of hard disk access in a data operation method according to an exemplary embodiment. Figure 3 As shown in the figure, the specific steps for connecting a hard disk to a data node are:

[0099] Step 301, obtaining the media type of the hard disk connected to the data node;

[0100] Step 302: If it is determined that the connected hard disk is a CMR hard disk, then step 304 is executed;

[0101] Step 303: If it is determined that the connected hard disk is an SMR hard disk, then step 305 is executed;

[0102] Step 304, associating the hard disk with the CMR hard disk service unit;

[0103] Step 305, associating the hard disk with the SMR hard disk service unit;

[0104] Step 306: Report the information of the connected hard disk to the metadata server.

[0105] The data nodes of the disclosed embodiment shield the differences in managing CMR and SMR hard disks through software means, so that the metadata server can deliver the segmented data blocks to the hard disks of any node and any medium, reducing the difficulty of maintaining the distributed storage cluster.

[0106] As an optional implementation, the data operation instructions include storage operation instructions carrying data blocks to be stored, deletion operation instructions, read data operation instructions, write data operation instructions and other instructions. The implementation methods corresponding to various operation instructions are described below.

[0107] 1. When a user sends a storage operation instruction carrying a data block to be stored, the hard disk where the data block in the storage operation instruction is located is determined according to the storage operation instruction; and according to the determined medium type of the hard disk, a service unit corresponding to the medium type is called to store the data block.

[0108] Specifically, after receiving the storage operation instruction with the data block to be stored from the metadata server, the data node will find the corresponding hard disk according to the location information in the storage operation instruction. When storing data blocks, it is necessary to ensure that different data blocks of the same data file are distributed in different hard disks.

[0109] After calling the service unit corresponding to the medium type to store data blocks according to the determined medium type of the hard disk, the following steps are further performed:

[0110] Determine the hard disk corresponding to the data block that has not been successfully stored, and store the data block that has not been successfully stored to another hard disk of a corresponding medium type according to the instruction of the metadata server, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that has not been successfully stored belongs;

[0111] The hard disk corresponding to the data block successfully stored in the data file is determined, and the location information of the hard disk corresponding to the data block successfully stored is returned to the metadata server.

[0112] Specifically, the data node returns the location information of the hard disk where the successfully stored data block is written. The location information of the SMR hard disk includes the starting address and length of the data block on the hard disk, and the location information of the CMR hard disk includes the directory structure and offset address of the data block file. The metadata server establishes a user file relationship table including the data file name, data block name and the location information of the successfully stored hard disk based on the location information of the hard disk corresponding to the successfully stored data block returned by the data node, so as to manage the user's data files in the distributed storage cluster.

[0113] 2. When the user sends a deletion operation instruction, the instruction carries the media type and location information of the hard disk where the data block to be deleted is located. According to the location information of the hard disk where the data block to be deleted in the instruction is located, the user file relationship table is used to determine the hard disk corresponding to the location information of the hard disk; according to the media type of the hard disk, the corresponding service unit is called to delete the data block; after the deletion is completed, the information of the hard disk is reported to the metadata server, and the metadata server updates the relevant content in the user file relationship table.

[0114] 3. When the user sends a data read operation instruction, the instruction carries the media type and location information of the hard disk where the data block to be read is located. According to the location information of the hard disk where the data block to be read in the instruction is located, the user file relationship table is used to determine the hard disk corresponding to the location information of the hard disk; according to the determined media type of the hard disk, the corresponding service unit is called to perform a read operation on the data block; after the write operation is completed, the information of the hard disk is reported to the metadata server, and the metadata server updates the relevant content in the user file relationship table.

[0115] 4. When the user sends a write data operation instruction, the instruction carries the media type and location information of the hard disk where the data block to be written is located. According to the location information of the hard disk where the data block to be written in the instruction is located, the user file relationship table is used to determine the hard disk corresponding to the location information of the hard disk; according to the determined media type of the hard disk, the corresponding service unit is called to write the data block; after the write operation is completed, the information of the hard disk is reported to the metadata server, and the metadata server updates the relevant content in the user file relationship table.

[0116] Example 2

[0117] The data operation method provided by the embodiment of the present disclosure is applied to a metadata server, such as Figure 4 As shown, including:

[0118] Step 401, obtaining hard disk information reported by a data node, wherein the hard disk information includes a media type, available capacity, and location information of the hard disk;

[0119] In the prior art, the metadata server cannot sense the media type of the hard disk connected to the data node, and thus cannot store different types of data blocks in the appropriate hard disk. The metadata server disclosed in the present invention obtains the hard disk information reported by the data node, and thus manages the hard disk.

[0120] Step 402, in response to a user's data operation instruction, determining the media type and location information of the hard disk to be operated;

[0121] Specifically, if the data operation instruction is a storage operation instruction, the media type and location information of the hard disk to be operated is determined according to the type of data block of the data file in the storage operation instruction; if the data operation instruction is other data operation instructions, such as a read operation instruction or a delete operation instruction, the media type and location information of the hard disk to be operated is directly determined according to the data operation instruction.

[0122] Step 403: Instruct the data node via a data operation instruction on the determined medium type and location information of the hard disk to be operated.

[0123] Specifically, the metadata server determines the data nodes corresponding to the location information of the hard disk to be operated according to the determined location information of the hard disk to be operated, and indicates the data operation instructions to these data nodes.

[0124] In the present disclosure, after the data node reports the type of hard disk media connected to the metadata server, the distributed storage cluster selects the appropriate storage medium when processing data files of different life cycles, which can reasonably utilize the cluster capacity, improve read and write performance, and reduce the difficulty of data management and space recovery.

[0125] In the metadata server, the user file relationship table is established based on the hard disk location information corresponding to the successfully stored data block returned by the data node, which includes the data file name, data block name and the location information of the successfully stored hard disk. When performing other operations, the metadata server finds the hard disk location information corresponding to the data block to be operated based on the user file relationship table to facilitate the corresponding operation.

[0126] Image files occupy a small amount of hard disk capacity and have an irregular life cycle, and belong to the first data file type. These files may come from different business scenarios, and their life cycles are not consistent. If they are stored in SMR hard disks, it will be inconvenient to recycle hard disk space. In order to avoid wasting space, a large amount of resources will be consumed for garbage collection. Therefore, these image files with inconsistent life cycles should be stored in CMR hard disks. Once the life cycle of an image file expires, it can be deleted from the hard disk, and the corresponding storage space can be released by deleting as many image files as possible.

[0127] Video files are continuously written, have a fixed life cycle, and occupy a large amount of hard disk capacity. They belong to the second data file type. These files can be stored in SMR hard disks, and when the file life cycle expires, they can be deleted in units of zones.

[0128] When the metadata server receives a storage operation instruction for a resource requested by a user, the available hard disk is determined based on the hard disk information of the data node where the image file is segmented according to the data blocks of the erasure code redundancy rule, and the CMR hard disk and its location information are preferentially selected from the available hard disks according to the priority policy of the data node. If the available capacity of the CMR hard disk is smaller than the size of the data block to be stored, the SMR hard disk and its location information are allowed to be selected; and the available hard disk is determined based on the hard disk information of the data node where the video file is segmented according to the data blocks of the erasure code redundancy rule, and the SMR hard disk and its location information are preferentially selected from the available hard disks according to the priority policy of the data node. If the available capacity of the SMR hard disk is smaller than the size of the data block to be stored, the CMR hard disk and its location information are allowed to be selected.

[0129] As an optional implementation, the metadata server receives a storage operation instruction of a resource requested by a user;

[0130] In response to the storage operation instruction, the data blocks to be stored are determined, that is, the user's data files, such as video files or picture files, are divided into data blocks of the same size, and after the check data is calculated according to the redundancy rule of the erasure code and the data blocks, it is distributed to the data nodes to complete the storage. The specific process is referred to the existing process and will not be described in detail here;

[0131] Determine that the data block to be stored belongs to the first data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the first medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node;

[0132] Determine that the data block to be stored belongs to the second data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the second medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node;

[0133] The priority strategy is to select a hard disk of a corresponding media type according to the type of data block to be stored. If the available capacity of the hard disk of the corresponding media type is smaller than the size of the data block to be stored, select a hard disk of another media type.

[0134] Among them, the first data file type is a data file with an indefinite life cycle and the occupied hard disk capacity is less than a set threshold, the second data file type is a data file with a fixed life cycle and the occupied hard disk capacity is greater than a set threshold, the first medium type hard disk is a CMR hard disk, and the second medium type hard disk is an SMR hard disk.

[0135] When processing data files of different life cycles, the metadata server selects appropriate storage media according to the type of data file, which can reasonably utilize cluster capacity, improve read and write performance, and reduce the difficulty of data management and space recovery.

[0136] As an optional implementation manner, the data operation instruction includes a storage operation instruction carrying a data block to be stored, and after indicating the determined medium type and location information of the hard disk to be operated to the data node through the data operation instruction, the method further includes:

[0137] According to the hard disk corresponding to the data block that was not successfully stored fed back by the data node, instructing the data block that was not successfully stored to be stored in another hard disk of a corresponding medium type, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that was not successfully stored belongs;

[0138] Specifically, the data node feeds back the location information of the hard disk corresponding to the data block that was not successfully stored and the data block that was not successfully stored, the metadata server receives the data block that was not successfully stored fed back by the data node, determines that the data block was not successfully stored, receives the location information of the hard disk corresponding to the data block that was not successfully stored fed back by the data node, determines the location of the hard disk that was not successfully stored, and tries to find other hard disks of the same media type on the data node other than this hard disk or hard disks of the same media type on other data nodes. For example, if the data file is a picture file of the first data file type, if there are data blocks that were not successfully stored in the data blocks of the picture file, the metadata server instructs the other CMR hard disks of the original data node or the CMR hard disks of other data nodes to store it. If the data file is a video file of the second data file type, if there are data blocks that were not successfully stored in the data blocks of the video file, the metadata server instructs the other SMR hard disks of the original data node or the SMR hard disks of other data nodes to store it.

[0139] According to the hard disk corresponding to the successfully stored data block fed back by the data node, the location information of the hard disk corresponding to the successfully stored data block returned by the data node is obtained;

[0140] A user file relationship table is established according to the location information of the hard disk corresponding to the successfully stored data block returned by the data node, wherein the user relationship table includes the data file name, the data block name and the location information of the successfully stored hard disk.

[0141] Figure 5 is a flowchart showing data file storage in a data operation method according to an exemplary embodiment. Figure 5 As shown, the steps for data file storage are as follows:

[0142] Step 501, obtaining a storage operation instruction of a user, wherein the storage operation instruction carries relevant information of a data block of a data file of the user;

[0143] Step 502, the data file is a second data file type, such as a video file, and the SMR hard disk to be operated and its location information are determined;

[0144] Step 503: The data file is of the first data file type, such as a picture file, and the CMR hard disk to be operated and its location information are determined;

[0145] Step 504, obtaining a storage operation instruction carrying the determined medium type and location information of the hard disk to be operated, the instruction also carrying relevant information of the data block of the user's data file;

[0146] Step 505, sending the storage operation instruction to the data node corresponding to the location information of the hard disk to be operated;

[0147] Step 506, obtaining the data block storage status fed back by the data node;

[0148] Step 507, determine whether the data block is stored successfully, if not, execute step 508, if successful, execute steps 509-510;

[0149] Step 508, indicating other hard disks of the media type corresponding to the data block to the data node, and then executing steps 505-507;

[0150] Step 509, obtaining the hard disk location information of the successfully stored data block returned by the data node;

[0151] Step 510: Establish a user file relationship table according to the hard disk location information of the successfully stored data block returned by the data node.

[0152] In the existing distributed storage cluster, both the metadata server and the data node are responsible for data management, which leads to the following problems:

[0153] First, in order to meet the data redundancy of erasure codes, a certain number of data nodes in the cluster must be met. Otherwise, the metadata server cannot select appropriate data nodes when certain redundancy requirements are met. For example, if the number of data nodes is less than 7, 7+2 data redundancy cannot be performed.

[0154] The second is that both the metadata server and the data node need to perform load balancing and data redundancy verification, that is, to ensure that different data blocks of the same user file are distributed in different storage media, there is a certain degree of logical redundancy, and it will increase the system load of the data node.

[0155] In the disclosed embodiment, only the metadata server is responsible for executing the load balancing strategy at the cluster hard disk level, giving priority to hard disks with more remaining capacity and less IO pressure to complete data block storage. After the data block is successfully stored, the metadata server establishes a user file relationship table based on the location information of the hard disk corresponding to the successfully stored data block returned by the data node to manage the user's data files in the distributed storage cluster, which has the following advantages:

[0156] In scenarios with a small cluster size and a small number of data nodes, as long as there are enough hard disks, the data redundancy of the erasure code can be met, thereby improving the availability of the distributed storage cluster in various business scenarios; integrating the pressure of user data redundancy verification and the load balancing strategy of data nodes for hard disks into a metadata server with stronger performance and more system resources can improve the write performance of data nodes and reduce the waste of resources in maintaining multiple data relationships.

[0157] Example 3

[0158] Based on the same inventive concept, the embodiment of the present disclosure also provides a data operation device. Since the device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0159] like Figure 6 As shown, the above device includes the following modules:

[0160] The hard disk information reporting module 601 is used to determine the hard disk connected to the data node through the data node, and report the hard disk information to the metadata server, wherein the hard disk information includes the media type, available capacity and location information of the hard disk;

[0161] The data operation instruction receiving module 602 is used to receive the data operation instruction sent by the metadata server through the data node, wherein the data operation instruction carries the media type and location information of the hard disk where the operated data block is located;

[0162] The data block operation module 603 is used to call the service unit corresponding to the medium type through the data node according to the medium type to perform data block operation according to the location information of the hard disk.

[0163] As an optional implementation manner, before the hard disk information reporting module determines the hard disk connected to the data node through the data node, it is further used to:

[0164] In response to a hard disk access instruction, determining a medium type of the hard disk;

[0165] According to the medium type of the hard disk and the corresponding relationship between different medium types and service units, the hard disk is associated with the corresponding service unit.

[0166] As an optional implementation, the data operation instruction includes a storage operation instruction carrying a data block to be stored, and the data block operation module, through the data node, calls a service unit corresponding to the media type according to the media type to perform the data block operation according to the hard disk location information, and is further used to:

[0167] According to the storage operation instruction, determining the hard disk where the data block in the storage operation instruction is located;

[0168] According to the determined medium type of the hard disk, a service unit corresponding to the medium type is called to store data blocks.

[0169] As an optional implementation manner, the data block operation module, after invoking the service unit corresponding to the media type to store the data block according to the determined media type of the hard disk, is further configured to:

[0170] Determine the hard disk corresponding to the data block that has not been successfully stored, and store the data block that has not been successfully stored to another hard disk of a corresponding medium type according to the instruction of the metadata server, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that has not been successfully stored belongs;

[0171] The hard disk corresponding to the data block successfully stored in the data file is determined, and the location information of the hard disk corresponding to the data block successfully stored is returned to the metadata server.

[0172] As an optional implementation manner, the hard disk information reporting module associates the hard disk with the corresponding service unit through the data node according to the medium type of the hard disk and the correspondence between different medium types and service units, and is further used to:

[0173] Determining that the medium type of the hard disk is a conventional magnetic recording CMR hard disk, and associating the hard disk with a CMR hard disk service unit;

[0174] It is determined that the medium type of the hard disk is a shingled magnetic recording (SMR) hard disk, and the hard disk is associated with an SMR hard disk service unit.

[0175] Example 4

[0176] Based on the same inventive concept, the embodiment of the present disclosure also provides a data operation device. Since the device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0177] like Figure 7 As shown, the above device includes the following modules:

[0178] The hard disk information acquisition module 701 is used to acquire the hard disk information reported by the data node through the metadata server, wherein the hard disk information includes the medium type, available capacity and location information of the hard disk;

[0179] The data operation instruction execution module 702 is used to determine the media type and location information of the hard disk to be operated in response to the user's data operation instruction through the metadata server;

[0180] The data operation instruction sending module 703 is used to indicate the determined medium type and location information of the hard disk to be operated to the data node through the metadata server through the data operation instruction.

[0181] As an optional implementation, the data operation instruction execution module determines the media type and location information of the hard disk to be operated in response to the user's data operation instruction through the metadata server, and is also used to:

[0182] In response to the storage operation instruction, determining a data block to be stored;

[0183] Determine that the data block to be stored belongs to the first data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the first medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node;

[0184] Determine that the data block to be stored belongs to the second data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the second medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node;

[0185] The priority strategy is to select a hard disk of a corresponding media type according to the type of data block to be stored. If the available capacity of the hard disk of the corresponding media type is smaller than the size of the data block to be stored, select a hard disk of another media type.

[0186] As an optional implementation, the first data file type is a data file whose life cycle is not fixed and whose occupied hard disk capacity is less than a set threshold, the second data file type is a data file whose life cycle is fixed and whose occupied hard disk capacity is greater than a set threshold, the first media type hard disk is a CMR hard disk, and the second media type hard disk is an SMR hard disk.

[0187] As an optional implementation, the data operation instruction includes a storage operation instruction carrying a data block to be stored, and the data operation instruction sending module, after indicating the determined medium type and location information of the hard disk to be operated to the data node through the metadata server through the data operation instruction, is further used to:

[0188] According to the hard disk corresponding to the data block that was not successfully stored fed back by the data node, instructing the data block that was not successfully stored to be stored in another hard disk of a corresponding medium type, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that was not successfully stored belongs;

[0189] According to the hard disk corresponding to the successfully stored data block fed back by the data node, the location information of the hard disk corresponding to the successfully stored data block returned by the data node is obtained;

[0190] A user file relationship table is established according to the location information of the hard disk corresponding to the successfully stored data block returned by the data node, wherein the user relationship table includes the data file name, the data block name and the location information of the successfully stored hard disk.

[0191] Example 5

[0192] Based on the same inventive concept, a data operation electronic device is also provided in the embodiment of the present disclosure. Since the electronic device is the electronic device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0193] Refer to the following Figure 8 An electronic device 80 according to this embodiment of the present disclosure will be described. Figure 8 The electronic device 80 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0194] like Figure 8 As shown, the electronic device 80 may be in the form of a general-purpose computing device, for example, it may be a terminal device. The components of the electronic device 80 may include, but are not limited to: the at least one processor 81, the at least one memory 82 storing processor executable instructions, and a bus 83 connecting different system components (including the memory 82 and the processor 81).

[0195] The processor executes the executable instructions to implement the following steps:

[0196] Determine a hard disk connected to the data node, and report information of the hard disk to a metadata server, wherein the information of the hard disk includes a media type, available capacity, and location information of the hard disk;

[0197] receiving a data operation instruction sent by the metadata server, wherein the data operation instruction carries the media type and location information of the hard disk where the operated data block is located;

[0198] According to the medium type, a service unit corresponding to the medium type is called to perform a data block operation according to the location information of the hard disk.

[0199] As an optional implementation manner, before the processor determines the hard disk connected to the data node, it is further configured to execute:

[0200] In response to a hard disk access instruction, determining a medium type of the hard disk;

[0201] According to the medium type of the hard disk and the corresponding relationship between different medium types and service units, the hard disk is associated with the corresponding service unit.

[0202] As an optional implementation manner, the data operation instruction includes a storage operation instruction carrying a data block to be stored, and according to the medium type, a service unit corresponding to the medium type is called to perform a data block operation according to the hard disk location information, and the processor executes:

[0203] According to the storage operation instruction, determining the hard disk where the data block in the storage operation instruction is located;

[0204] According to the determined medium type of the hard disk, a service unit corresponding to the medium type is called to store data blocks.

[0205] As an optional implementation manner, after the processor calls the service unit corresponding to the media type to store the data block according to the determined media type of the hard disk, it is further configured to execute:

[0206] Determine the hard disk corresponding to the data block that has not been successfully stored, and store the data block that has not been successfully stored to another hard disk of a corresponding medium type according to the instruction of the metadata server, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that has not been successfully stored belongs;

[0207] The hard disk corresponding to the data block successfully stored in the data file is determined, and the location information of the hard disk corresponding to the data block successfully stored is returned to the metadata server.

[0208] As an optional implementation manner, according to the medium type of the hard disk and the correspondence between different medium types and service units, the hard disk is associated with the corresponding service unit, and the processor executes:

[0209] Determining that the medium type of the hard disk is a conventional magnetic recording CMR hard disk, and associating the hard disk with a CMR hard disk service unit;

[0210] It is determined that the medium type of the hard disk is a shingled magnetic recording (SMR) hard disk, and the hard disk is associated with an SMR hard disk service unit.

[0211] Bus 83 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0212] The memory 82 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 821 and / or a cache memory 822 , and may further include a read-only memory (ROM) 823 .

[0213] The memory 82 may also include a program / utility 825 having a set (at least one) of program modules 824, such program modules 824 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0214] The electronic device 80 may also communicate with one or more external devices 84 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 80, and / or may communicate with any device that enables the electronic device 80 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 85. Furthermore, the electronic device 80 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 86. As shown, the network adapter 86 communicates with other modules of the electronic device 80 via a bus 83. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 80, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0215] Refer to the following Fig. 9 An electronic device 90 according to this embodiment of the present disclosure will be described. Fig. 9 The electronic device 90 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0216] like Fig. 9As shown, the electronic device 90 may be in the form of a general-purpose computing device, for example, it may be a terminal device. The components of the electronic device 90 may include, but are not limited to: the at least one processor 91, the at least one memory 92 storing processor executable instructions, and a bus 93 connecting different system components (including the memory 92 and the processor 91).

[0217] The processor executes the executable instructions to implement the following steps:

[0218] Obtaining hard disk information reported by the data node, wherein the hard disk information includes a media type, available capacity, and location information of the hard disk;

[0219] In response to a user's data operation instruction, determining the media type and location information of the hard disk to be operated;

[0220] The determined medium type and location information of the hard disk to be operated is indicated to the data node through a data operation instruction.

[0221] As an optional implementation manner, in response to a user's data operation instruction, determining the medium type and location information of the hard disk to be operated, the processor executes:

[0222] In response to the storage operation instruction, determining a data block to be stored;

[0223] Determine that the data block to be stored belongs to the first data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the first medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node;

[0224] Determine that the data block to be stored belongs to the second data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the second medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node;

[0225] The priority strategy is to select a hard disk of a corresponding media type according to the type of data block to be stored. If the available capacity of the hard disk of the corresponding media type is smaller than the size of the data block to be stored, select a hard disk of another media type.

[0226] As an optional implementation, the first data file type is a data file whose life cycle is not fixed and whose occupied hard disk capacity is less than a set threshold, the second data file type is a data file whose life cycle is fixed and whose occupied hard disk capacity is greater than a set threshold, the first media type hard disk is a CMR hard disk, and the second media type hard disk is an SMR hard disk.

[0227] As an optional implementation manner, the data operation instruction includes a storage operation instruction carrying a data block to be stored, and after the processor indicates the determined medium type and location information of the hard disk to be operated to the data node through the data operation instruction, it is further used to execute:

[0228] According to the hard disk corresponding to the data block that was not successfully stored fed back by the data node, instructing the data block that was not successfully stored to be stored in another hard disk of a corresponding medium type, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that was not successfully stored belongs;

[0229] According to the hard disk corresponding to the successfully stored data block fed back by the data node, the location information of the hard disk corresponding to the successfully stored data block returned by the data node is obtained;

[0230] A user file relationship table is established according to the location information of the hard disk corresponding to the successfully stored data block returned by the data node, wherein the user relationship table includes the data file name, the data block name and the location information of the successfully stored hard disk.

[0231] Bus 93 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0232] The memory 92 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922 , and may further include a read-only memory (ROM) 923 .

[0233] The memory 92 may also include a program / utility 925 having a set (at least one) of program modules 924, such program modules 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0234] The electronic device 90 may also communicate with one or more external devices 94 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 90, and / or may communicate with any device that enables the electronic device 90 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 95. Furthermore, the electronic device 90 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 96. As shown, the network adapter 96 communicates with other modules of the electronic device 90 via a bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 90, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0235] Example 6

[0236] In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute the steps of each module in the data operation device according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of this specification. For example, the terminal device can be used to determine the hard disk connected to the data node and report the information of the hard disk to the metadata server, wherein the information of the hard disk includes the media type, available capacity and location information of the hard disk; receive data operation instructions sent by the metadata server, wherein the data operation instructions carry operations such as the media type and location information of the hard disk where the operated data block is located.

[0237] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0238] like Fig.10As shown, a program product 100 for data operation according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0239] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0240] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0241] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0242] It should be noted that although several modules or submodules of the system are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0243] In addition, although the operations of the modules of the disclosed system are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some operations may be omitted, multiple operations may be combined into one operation, and / or one operation may be decomposed into multiple operations.

[0244] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0245] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.

[0246] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0247] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0248] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0249] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data operation method, applied to a data node, characterized in that: The method includes: Determine a hard disk connected to the data node, and report information of the hard disk to a metadata server, wherein the information of the hard disk includes a media type, available capacity, and location information of the hard disk; receiving a data operation instruction sent by the metadata server, wherein the data operation instruction carries the media type and location information of the hard disk where the data block to be operated is located; According to the medium type, calling a service unit corresponding to the medium type to perform a data block operation according to the location information of the hard disk; The metadata server determines the media type and location information of the hard disk where the data block to be operated in the data operation instruction is located by: In response to a storage operation instruction from a user, determining a data block to be stored; Determine that the data block to be stored belongs to the first data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the first medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node; Determine that the data block to be stored belongs to the second data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the second medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node; The priority strategy is to select a hard disk of a corresponding media type according to the type of data block to be stored. If the available capacity of the hard disk of the corresponding media type is smaller than the size of the data block to be stored, select a hard disk of another media type.

2. The method according to claim 1, characterized in that Before determining the hard disk connected to the data node, the method further includes: In response to a hard disk access instruction, determining a medium type of the hard disk; According to the medium type of the hard disk and the corresponding relationship between different medium types and service units, the hard disk is associated with the corresponding service unit.

3. The method according to claim 1, characterized in that The data operation instruction includes a storage operation instruction carrying a data block to be stored, and calling a service unit corresponding to the medium type to perform a data block operation according to the hard disk location information according to the medium type includes: According to the storage operation instruction, determining the hard disk where the data block in the storage operation instruction is located; According to the determined medium type of the hard disk, a service unit corresponding to the medium type is called to store data blocks.

4. The method according to claim 3, characterized in that: After calling the service unit corresponding to the medium type to store the data block according to the determined medium type of the hard disk, the method further comprises: Determine the hard disk corresponding to the data block that has not been successfully stored, and store the data block that has not been successfully stored to another hard disk of a corresponding medium type according to the instruction of the metadata server, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that has not been successfully stored belongs; The hard disk corresponding to the data block successfully stored in the data file is determined, and the location information of the hard disk corresponding to the data block successfully stored is returned to the metadata server.

5. The method according to claim 2, characterized in that: The associating the hard disk with the corresponding service unit according to the medium type of the hard disk and the correspondence between different medium types and service units includes: Determining that the medium type of the hard disk is a conventional magnetic recording CMR hard disk, and associating the hard disk with a CMR hard disk service unit; It is determined that the medium type of the hard disk is a shingled magnetic recording (SMR) hard disk, and the hard disk is associated with an SMR hard disk service unit.

6. A data operation method, applied to a metadata server, characterized in that: The method includes: Obtaining hard disk information reported by the data node, wherein the hard disk information includes a media type, available capacity, and location information of the hard disk; In response to a user's data operation instruction, determining the media type and location information of the hard disk to be operated; Indicating the determined medium type and location information of the hard disk to be operated to the data node through a data operation instruction; The step of determining the media type and location information of the hard disk to be operated in response to the user's data operation instruction includes: In response to the storage operation instruction, determining a data block to be stored; Determine that the data block to be stored belongs to the first data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the first medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node; Determine that the data block to be stored belongs to the second data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the second medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node; The priority strategy is to select a hard disk of a corresponding media type according to the type of data block to be stored. If the available capacity of the hard disk of the corresponding media type is smaller than the size of the data block to be stored, select a hard disk of another media type.

7. The method according to claim 6, characterized in that The first data file type is a data file with an indefinite life cycle and the occupied hard disk capacity is less than a set threshold, the second data file type is a data file with a fixed life cycle and the occupied hard disk capacity is greater than a set threshold, the first medium type hard disk is a CMR hard disk, and the second medium type hard disk is an SMR hard disk.

8. The method according to claim 6, characterized in that The data operation instruction includes a storage operation instruction carrying a data block to be stored. After indicating the determined medium type and location information of the hard disk to be operated to the data node through the data operation instruction, the method further includes: According to the hard disk corresponding to the data block that was not successfully stored fed back by the data node, instructing the data block that was not successfully stored to be stored in another hard disk of a corresponding medium type, where the corresponding medium type is a medium type corresponding to the data file type to which the data block that was not successfully stored belongs; According to the hard disk corresponding to the successfully stored data block fed back by the data node, the location information of the hard disk corresponding to the successfully stored data block returned by the data node is obtained; A user file relationship table is established according to the location information of the hard disk corresponding to the successfully stored data block returned by the data node, wherein the user file relationship table includes the data file name, the data block name and the location information of the successfully stored hard disk.

9. A data operation device, characterized in that: The device comprises: A hard disk information reporting module, used to determine the hard disk connected to the data node through the data node, and report the hard disk information to the metadata server, wherein the hard disk information includes the media type, available capacity and location information of the hard disk; A data operation instruction receiving module, used to receive a data operation instruction sent by the metadata server through a data node, wherein the data operation instruction carries the media type and location information of the hard disk where the data block to be operated is located; A data block operation module, configured to call a service unit corresponding to the medium type through a data node according to the medium type to perform a data block operation according to the location information of the hard disk; The metadata server determines the media type and location information of the hard disk where the data block to be operated is located in the following manner: In response to a storage operation instruction from a user, determining a data block to be stored; Determine that the data block to be stored belongs to the first data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the first medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node; Determine that the data block to be stored belongs to the second data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the second medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node; The priority strategy is to select a hard disk of a corresponding media type according to the type of data block to be stored. If the available capacity of the hard disk of the corresponding media type is smaller than the size of the data block to be stored, select a hard disk of another media type.

10. A data operation device, characterized in that: The device comprises: A hard disk information acquisition module, used to acquire hard disk information reported by a data node through a metadata server, wherein the hard disk information includes a media type, available capacity, and location information of the hard disk; A data operation instruction execution module, used to respond to the user's data operation instruction through the metadata server and determine the media type and location information of the hard disk to be operated; A data operation instruction sending module, used to indicate the determined medium type and location information of the hard disk to be operated to the data node through the metadata server through the data operation instruction; Wherein, the data operation instruction execution module is used for: In response to the storage operation instruction, determining a data block to be stored; Determine that the data block to be stored belongs to the first data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the first medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node; Determine that the data block to be stored belongs to the second data file type, determine an available hard disk according to the hard disk information of the data node, and select a hard disk of the second medium type and location information corresponding to the data block to be stored from the available hard disks according to the priority strategy of the data node; The priority strategy is to select a hard disk of a corresponding media type according to the type of data block to be stored. If the available capacity of the hard disk of the corresponding media type is smaller than the size of the data block to be stored, select a hard disk of another media type.

11. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in any one of claims 1 to 5, or implements the steps of the method described in any one of claims 6 to 8 by running the executable instructions.

12. A computer readable and writable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented, or the steps of the method described in any one of claims 6 to 8 are implemented.

Images