Data storage method, device, equipment and machine-readable storage medium

Abstract

The present disclosure provides a data storage method, apparatus, device and machine-readable storage medium, the method comprising: in response to a data copy command, parsing a data source address and a data destination address associated with the data copy command; parsing data distribution information of a data source file based on a judgment result that both the data source address and the data destination address belong to the storage cluster; establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file, and performing local device data replication. Through the technical solution of the present disclosure, when executing a data copy command, based on the data distribution of the data source file, the data destination file generated by the replication is also stored in the storage device storing the data source file in the storage cluster, so that when copying files, data transfer between storage devices does not occur, thereby avoiding network data transmission and reducing network bandwidth resource consumption.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a data storage method, apparatus, device, and machine-readable storage medium. Background Art

[0002] With the advancement of technologies such as high-definition video, image processing, and video surveillance, user data capacity is increasing, and user requirements for the read and write performance of stored data are also increasing. When purchasing storage products, users often have clear performance requirements to meet business needs. In many business scenarios, storage performance is not just a matter of speed; it can even affect the normal operation of the business itself. Take file digitization as an example. After archives and libraries digitize paper books, they store them on storage servers. As more and more digitized books are stored, the capacity of the storage cluster will continue to grow. At the same time, with a large number of users accessing them concurrently, poor read and write performance and high read and write latency in the storage cluster will reduce user friendliness and hinder the advancement of digitization. Therefore, improving the read and write performance of the storage cluster is crucial.

[0003] With the development of digitalization, distributed storage is used in more and more fields, and the data on the storage system is also increasing. As a result, the consumption of network bandwidth resources caused by read and write operations increases, making network bandwidth resources very important.

[0004] In some cases, it is necessary to perform file and data replication operations within a storage cluster. Since the storage devices in the storage cluster are connected to the storage cluster through network devices and exchange data through the network within the cluster, this replication operation consumes a lot of network bandwidth resources. Summary of the Invention

[0005] In view of this, the present disclosure provides a data storage method, device, electronic device, and machine-readable storage medium to improve the problem of excessive network resource consumption caused by data replication within the cluster.

[0006] The specific technical solutions are as follows:

[0007] The present disclosure provides a data storage method, which is applied to a storage cluster device, wherein the storage cluster device includes several storage devices for storing data. The method includes: in response to a data copy command, parsing a data source address and a data destination address associated with the data copy command, and determining whether the data source address and the data destination address both belong to the current storage cluster; based on the determination result that the data source address and the data destination address both belong to the current storage cluster, parsing data distribution information of a data source file; based on the data distribution information of the data source file, establishing a data target file distributed on the same storage device as the data source file, and executing local device data replication.

[0008] As a technical solution, the method of establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file and performing local device data replication includes: establishing a data target file distributed on the same disk as the data source file based on the data distribution information of the data source file and performing local disk data replication.

[0009] As a technical solution, the method of parsing the data distribution information of the data source file based on the judgment result that the data source address and the data destination address both belong to the present storage cluster includes: parsing the data distribution information of the copy file of the data source file based on the judgment result that the data source address and the data destination address both belong to the present storage cluster; establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file, and performing local device data replication, including: establishing a copy file of the data target file distributed on the same storage device as the copy file of the data source file based on the data distribution information of the copy file of the data source file, and performing local device data replication of the copy file.

[0010] As a technical solution, the executing local device data replication includes: in response to an event of executing local device data replication of the data target file and the copy file of the data target file being completed, feeding back a reply message indicating that the data replication is completed.

[0011] The present disclosure also provides a data storage device, which is applied to a storage cluster device, wherein the storage cluster device includes several storage devices for storing data. The method includes: an address module, which is used to respond to a data copy command, parse a data source address and a data destination address associated with the data copy command, and determine whether the data source address and the data destination address both belong to the current storage cluster; a parsing module, which is used to parse data distribution information of a data source file based on a determination result that the data source address and the data destination address both belong to the current storage cluster; and a replication module, which is used to establish a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file, and perform local device data replication.

[0012] As a technical solution, the method of establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file and performing local device data replication includes: establishing a data target file distributed on the same disk as the data source file based on the data distribution information of the data source file and performing local disk data replication.

[0013] As a technical solution, the method of parsing the data distribution information of the data source file based on the judgment result that the data source address and the data destination address both belong to the present storage cluster includes: parsing the data distribution information of the copy file of the data source file based on the judgment result that the data source address and the data destination address both belong to the present storage cluster; establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file, and performing local device data replication, including: establishing a copy file of the data target file distributed on the same storage device as the copy file of the data source file based on the data distribution information of the copy file of the data source file, and performing local device data replication of the copy file.

[0014] As a technical solution, the executing local device data replication includes: in response to an event of executing local device data replication of the data target file and the copy file of the data target file being completed, feeding back a reply message indicating that the data replication is completed.

[0015] The present disclosure also provides an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the aforementioned data storage method.

[0016] The present disclosure also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the aforementioned data storage method.

[0017] The above technical solution provided by the present disclosure brings at least the following beneficial effects:

[0018] When executing a data copy command, it first analyzes whether the data generated by the copy will be stored in the same data cluster as the data source. If so, based on the data distribution of the data source file, the data destination file generated by the copy will also be stored in the storage device that stores the data source file in the storage cluster. In this way, when copying files, data transfer between storage devices does not occur, avoiding network data transmission, not occupying the business network and storage network, and reducing network bandwidth resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present disclosure or the description of the prior art. Obviously, the drawings described below are only some of the embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present disclosure.

[0020] Figure 1 is a flow chart of a data storage method in one embodiment of the present disclosure;

[0021] Figure 2 is a structural diagram of a data storage device in one embodiment of the present disclosure;

[0022] Figure 3 It is a hardware structure diagram of an electronic device in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The terms used in the embodiments of the present disclosure are intended only to describe specific embodiments and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the present disclosure and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to any or all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" used may be interpreted as "at the time of" or "when" or "in response to determining".

[0025] In one technical solution, the data to be backed up is first read in full, and then written in full to the destination address. In the same cluster data backup scenario, this is equivalent to performing a full data read and full data write operation.

[0026] Specifically, for the current data replication in the same distributed cluster, taking distributed storage 2 copies and copying a file as an example, the process is as follows: the user selects the data to be backed up, and then executes the "cp source data file destination file" command; for distributed storage, the CP command will trigger the following actions: first, read the source file data, find the disk location of the primary copy according to the data location allocation algorithm, read the data into the memory, and then perform create writes at the destination. According to the data allocation algorithm, select the disk that carries the copy, and then create the destination file. Finally, write the content read from the memory to the destination file (multiple copies complete writing).

[0027] The data in the above technical solutions needs to go through a large amount of network transmission. When reading data and writing data, the data will be fully transmitted once through the business network, and the copy data will also be fully transmitted once through the storage network.

[0028] In view of this, the present disclosure provides a data storage method, device, electronic device, and machine-readable storage medium to improve the problem of excessive network resource consumption caused by data replication within the cluster.

[0029] Specifically, the technical solution is described below.

[0030] In one embodiment, the present disclosure provides a data storage method applied to a storage cluster device, wherein the storage cluster device includes several storage devices for storing data, and the method includes: in response to a data copy command, parsing a data source address and a data destination address associated with the data copy command, and determining whether the data source address and the data destination address both belong to the present storage cluster; based on the determination result that the data source address and the data destination address both belong to the present storage cluster, parsing data distribution information of a data source file; based on the data distribution information of the data source file, establishing a data target file distributed on the same storage device as the data source file, and performing local device data replication.

[0031] Specifically, if Figure 1 , including the following steps:

[0032] Step S11, in response to a data copy command, parsing a data source address and a data destination address associated with the data copy command, and determining whether both the data source address and the data destination address belong to the current storage cluster;

[0033] Step S12: parsing data distribution information of the data source file based on the result of determining that both the data source address and the data destination address belong to the current storage cluster;

[0034] Step S13: Based on the data distribution information of the data source file, a data target file is created that is distributed on the same storage device as the data source file, and data replication is performed on the local device.

[0035] If the result of determining whether the data source address and the data destination address both belong to the current storage cluster is that both do not belong to the current storage cluster, the data copy command is executed according to the original process.

[0036] When executing a data copy command, it first analyzes whether the data generated by the copy will be stored in the same data cluster as the data source. If so, based on the data distribution of the data source file, the data destination file generated by the copy will also be stored in the storage device that stores the data source file in the storage cluster. In this way, when copying files, data transfer between storage devices does not occur, avoiding network data transmission, not occupying the business network and storage network, and reducing network bandwidth resource consumption.

[0037] In one embodiment, the method of establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file and performing local device data replication includes: establishing a data target file distributed on the same disk as the data source file based on the data distribution information of the data source file and performing local disk data replication.

[0038] In one embodiment, the parsing of data distribution information of the data source file based on the judgment result that both the data source address and the data destination address belong to the present storage cluster includes: parsing the data distribution information of the copy file of the data source file based on the judgment result that both the data source address and the data destination address belong to the present storage cluster; establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file, and performing local device data replication, includes: establishing a copy file of the data target file distributed on the same storage device as the copy file of the data source file based on the data distribution information of the copy file of the data source file, and performing local device data replication of the copy file.

[0039] In one embodiment, executing the local device data replication includes: in response to an event that the local device data replication of the data target file and the copy file of the data target file is completed, feeding back a reply message indicating that the data replication is completed.

[0040] In one embodiment, data in the storage cluster is replicated in situ, which can achieve zero network transmission of data and avoid additional network bandwidth occupation. Since the data replication process is performed locally, the replication performance can be improved.

[0041] Specifically, the client sends a data replication command (CP) to the storage cluster. Upon receiving the CP command, the storage cluster first queries the MDS for the data distribution information of the source file. After obtaining the source file distribution information, the MDS creates the destination file. The destination file creation adopts a data distribution strategy consistent with the source file. The underlying object content of the destination file is copied from the source file object to the data destination file object via local storage device replication. To further conserve hardware bus bandwidth and CPU performance within the storage device, data replication executed in response to the CP command can be further limited to local disk replication, that is, the data destination file is distributed on the same disk as the source file.

[0042] Specifically, the client initiates a same-cluster data replication command to the distributed storage cluster. The distributed storage confirms that it is same-cluster data replication through source and destination detection. The metadata management module first queries the data distribution information of the source file and then creates the destination file. The data distribution information of the destination file adopts the data distribution information of the source file to ensure that the data storage object of the source file and the data storage object of the new destination file are distributed on the same disk. The MDS sends a data replication request to the primary object of the source file and passes the destination object name at the same time. After the source file primary object receives the same-host data replication request, it performs local disk data replication and sends the local object data replication request to the slave copy. After the slave copy receives the local data object replication request, it performs local disk data replication in the same manner as the primary copy. After the local data replication of the slave copy is completed, it will reply to the primary copy that the local data replication is completed. After the primary copy's own local data replication and the slave copy's data replication are completed, it will reply to the client that the data replication is completed.

[0043] In one embodiment, the present disclosure also provides a data storage device, such as Figure 2 , applied to a storage cluster device, the storage cluster device includes several storage devices for storing data, the method includes: an address module 21, for responding to a data copy command, parsing a data source address and a data destination address associated with the data copy command, and judging whether the data source address and the data destination address both belong to the present storage cluster; a parsing module 22, for parsing data distribution information of a data source file based on a judgment result that the data source address and the data destination address both belong to the present storage cluster; a replication module 23, for establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file, and executing local device data replication.

[0044] In one embodiment, the method of establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file and performing local device data replication includes: establishing a data target file distributed on the same disk as the data source file based on the data distribution information of the data source file and performing local disk data replication.

[0045] In one embodiment, the parsing of data distribution information of the data source file based on the judgment result that both the data source address and the data destination address belong to the present storage cluster includes: parsing the data distribution information of the copy file of the data source file based on the judgment result that both the data source address and the data destination address belong to the present storage cluster; establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file, and performing local device data replication, includes: establishing a copy file of the data target file distributed on the same storage device as the copy file of the data source file based on the data distribution information of the copy file of the data source file, and performing local device data replication of the copy file.

[0046] In one embodiment, executing the local device data replication includes: in response to an event that the local device data replication of the data target file and the copy file of the data target file is completed, feeding back a reply message indicating that the data replication is completed.

[0047] The device implementation is the same as or similar to the corresponding method implementation, and will not be repeated here.

[0048] In one embodiment, the present disclosure provides an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the aforementioned data storage method. From a hardware perspective, the hardware architecture diagram can be found in Figure 3 shown.

[0049] In one embodiment, the present disclosure provides a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the aforementioned data storage method.

[0050] Here, the machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage media, or a combination thereof.

[0051] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0052] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0053] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of a fully hardware implementation, a fully software implementation, or an implementation combining software and hardware. Furthermore, embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] Furthermore, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0057] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of a fully hardware implementation, a fully software implementation, or an implementation combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (which may include, but are not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure.

Claims

1. A data storage method, characterized in that: Applied to a storage cluster device, the storage cluster device includes several storage devices for storing data, and the method includes: In response to a data copy command, parsing a data source address and a data destination address associated with the data copy command, and determining whether both the data source address and the data destination address belong to the current storage cluster; Based on the result that both the data source address and the data destination address belong to the current storage cluster, the data distribution information of the data source file is parsed; According to the data distribution information of the data source file, a data target file is created which is distributed on the same storage device as the data source file, and data replication is performed on the local device.

2. The method according to claim 1, characterized in that The method of establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file and performing data replication on the local device includes: According to the data distribution information of the data source file, a data target file distributed on the same disk as the data source file is established, and local disk data replication is performed.

3. The method according to claim 1, characterized in that The step of parsing the data distribution information of the data source file based on the judgment result that both the data source address and the data destination address belong to the current storage cluster includes: The data distribution information of the replica file of the data source file is parsed based on the judgment result that the data source address and the data destination address both belong to the current storage cluster; The method of establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file and performing data replication on the local device includes: According to the data distribution information of the copy file of the data source file, a copy file of the data target file distributed on the same storage device as the copy file of the data source file is created, and local device data replication of the copy file is performed.

4. The method according to claim 3, characterized in that The executing local device data replication includes: In response to an event indicating that the local device data copying of the data target file and the replica file of the data target file is completed, a reply message indicating that the data copying is completed is fed back.

5. A data storage device, characterized in that Applied to a storage cluster device, the storage cluster device includes several storage devices for storing data, and the apparatus includes: An address module, configured to, in response to a data copy command, parse a data source address and a data destination address associated with the data copy command, and determine whether both the data source address and the data destination address belong to the current storage cluster; A parsing module is used to parse the data distribution information of the data source file based on the judgment result that the data source address and the data destination address both belong to the storage cluster; The replication module is used to establish a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file, and perform local device data replication.

6. The device according to claim 5, characterized in that The method of establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file and performing data replication on the local device includes: According to the data distribution information of the data source file, a data target file distributed on the same disk as the data source file is established, and local disk data replication is performed.

7. The device according to claim 5, characterized in that The step of parsing the data distribution information of the data source file based on the judgment result that both the data source address and the data destination address belong to the current storage cluster includes: The data distribution information of the replica file of the data source file is parsed based on the judgment result that the data source address and the data destination address both belong to the current storage cluster; The method of establishing a data target file distributed on the same storage device as the data source file based on the data distribution information of the data source file and performing data replication on the local device includes: According to the data distribution information of the copy file of the data source file, a copy file of the data target file distributed on the same storage device as the copy file of the data source file is created, and local device data replication of the copy file is performed.

8. The device according to claim 7, characterized in that The executing local device data replication includes: In response to an event indicating that the local device data copying of the data target file and the replica file of the data target file is completed, a reply message indicating that the data copying is completed is fed back.

9. An electronic device, characterized in that: include: A processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the method according to any one of claims 1 to 4.

10. A machine-readable storage medium, characterized in that The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the method according to any one of claims 1 to 4.

Images