Data reading and writing method, device, electronic equipment and medium

Abstract

The embodiments of the present invention provide a data reading and writing method, device, electronic device and medium, which relate to the field of computer technology, and can shorten the path of read and write operations, reduce the time consumed by read and write operations, and thus reduce the occurrence of read and write operation failures due to the excessive time consumed by read and write operations. The embodiments of the present invention include: according to the storage location in the virtual disk for storing the data to be read and written, determining the slices in the virtual disk for storing the data to be read and written, the virtual disk includes multiple slices, each slice is used to store data of a specified size. Then determine the virtual node mapped to the slice for storing the data to be read and written, and the virtual node corresponds to a preset number of servers in the storage cluster. Then send an input / output I / O request to the server corresponding to the virtual node, and the I / O request is used to request to read and write the data to be read and written from the storage location.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data reading and writing method, device, electronic equipment and medium. Background Art

[0002] In the related art, data storage forms include: local storage and distributed storage. Local storage means that any read and write operations are performed only in the local disk without going through the network. Since the read and write operations of local storage are all performed in the local disk, the disk capacity of local storage is limited, and if the server using local storage crashes, the read and write operations cannot be performed. In order to solve the above problems, distributed storage has emerged.

[0003] Distributed storage means that the storage system includes metadata nodes and storage nodes. Metadata nodes are used to store metadata information of files or data blocks. Any read and write operations must first pass through the metadata nodes to obtain the device identification, disk identification and offset address of the storage data block or file, and then send read and write requests to the data storage server corresponding to the obtained device identification to perform corresponding read and write operations.

[0004] However, in a distributed storage system, each read and write operation must first pass through the metadata node, which makes the execution path of the read and write operation longer, takes longer time, and even causes the read and write operation to fail. For example, if the network jitters, the read and write requests of each terminal in the network will be queued, and the delay caused by the queue will affect the data delay of the transmission from the client to the server, which is easy to cause the read and write request to fail. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a data reading and writing method, device, electronic device and medium, so as to shorten the path of the reading and writing operation, reduce the time consumed by the reading and writing operation, and thus reduce the occurrence of the reading and writing operation failure caused by the reading and writing operation consuming too long time. The specific technical solution is as follows:

[0006] In a first aspect, a data reading and writing method is provided, which is applied to a terminal, and the method includes:

[0007] According to a storage location in the virtual disk for storing the data to be read or written, determining a slice in the virtual disk for storing the data to be read or written, wherein the virtual disk includes a plurality of slices, each slice being used to store data of a specified size;

[0008] Determine a virtual node for storing the shard mapping of the data to be read and written, the virtual node corresponding to a preset number of servers in the storage cluster;

[0009] An input / output (I / O) request is sent to a server corresponding to the virtual node, where the I / O request is used to request to read and write the data to be read or written from the storage location.

[0010] Optionally, before determining, according to the storage location for storing the to-be-read / written data in the virtual disk, the slice for storing the to-be-read / written data in the virtual disk, the method further comprises:

[0011] A virtual node list is obtained, wherein the virtual node list includes a hard disk identifier corresponding to each virtual node and an Internet Protocol IP address of a server to which the hard disk belongs.

[0012] Optionally, obtaining a virtual node list includes:

[0013] Sending a request to a designated server, wherein the request is used to request to obtain the virtual node list;

[0014] The virtual node list is received, and a persistent connection is established with a server that sends the virtual node list.

[0015] Optionally, determining, according to a storage location in the virtual disk for storing the data to be read or written, a slice in the virtual disk for storing the data to be read or written, includes:

[0016] Determine the starting slice corresponding to the data to be read or written according to the starting storage position for storing the data to be read or written in the virtual disk and the size of each slice in the virtual disk;

[0017] According to the size of the starting segment and the data to be read or written, the ending segment corresponding to the data to be read or written is determined, and the starting segment, the ending segment and the segments between the starting segment and the ending segment are determined as the segments for storing the data to be read or written.

[0018] Optionally, after determining the virtual node for storing the shard mapping of the data to be read or written, the method further includes:

[0019] According to the virtual node list, a hard disk set for storing the data to be read or written is determined, wherein the hard disk set includes hard disks corresponding to virtual nodes for storing shard mapping of the data to be read or written.

[0020] Optionally, the I / O request is a write request, and sending the input / output I / O request to the server corresponding to the virtual node includes:

[0021] Determine the last hard disk of the hard disk set, and send the write request to the server to which the last hard disk belongs, so that after the last hard disk writes the data to be read and written, the data to be written written by the last hard disk is synchronized to other hard disks in the hard disk set according to the hard disk identifiers in the hard disk set, wherein the write request includes: the data to be written and the identifier of the virtual node; or,

[0022] The write request is sent to the server to which each hard disk in the hard disk set belongs respectively.

[0023] Optionally, the I / O request is a read request, and the sending of the input / output I / O request to the server corresponding to the virtual node includes:

[0024] The last hard disk of the hard disk set is determined, and the read request is sent to the server to which the last hard disk belongs, wherein the read request includes: an identifier of the data to be read and an identifier of the virtual node.

[0025] In a second aspect, a data reading and writing device is provided, which is applied to a terminal, and the device includes:

[0026] A determination module, configured to determine, according to a storage location in the virtual disk for storing the data to be read or written, a slice in the virtual disk for storing the data to be read or written, wherein the virtual disk includes a plurality of slices, each slice being used to store data of a specified size;

[0027] The determination module is further used to determine a virtual node for storing the shard mapping of the data to be read and written, wherein the virtual node corresponds to a preset number of servers in the storage cluster;

[0028] The sending module is used to send an input / output I / O request to the server corresponding to the virtual node determined by the determining module, wherein the I / O request is used to request to read and write the data to be read and written from the storage location.

[0029] Optionally, the device further includes: an acquisition module;

[0030] The acquisition module is used to obtain a virtual node list before determining the storage location of the data to be read and written in the virtual disk and the shards to store the data to be read and written in the virtual disk. The virtual node list includes the hard disk identifier corresponding to each virtual node and the Internet Protocol IP address of the server to which the hard disk belongs.

[0031] Optionally, the acquisition module is specifically used to:

[0032] Sending a request to a designated server, wherein the request is used to request to obtain the virtual node list;

[0033] The virtual node list is received, and a persistent connection is established with a server that sends the virtual node list.

[0034] Optionally, the determining module is specifically used to:

[0035] Determine the starting slice corresponding to the data to be read or written according to the starting storage position for storing the data to be read or written in the virtual disk and the size of each slice in the virtual disk;

[0036] According to the size of the starting segment and the data to be read or written, the ending segment corresponding to the data to be read or written is determined, and the starting segment, the ending segment and the segments between the starting segment and the ending segment are determined as the segments for storing the data to be read or written.

[0037] Optionally, the determination module is also used to determine a hard disk set for storing the data to be read and written according to the virtual node list after determining the virtual node for storing the shard mapping of the data to be read and written, and the hard disk set includes hard disks corresponding to the virtual nodes for storing the shard mapping of the data to be read and written.

[0038] Optionally, the I / O request is a write request, and the sending module is specifically used to:

[0039] Determine the last hard disk of the hard disk set, and send the write request to the server to which the last hard disk belongs, so that after the last hard disk writes the data to be read and written, the data to be written written by the last hard disk is synchronized to other hard disks in the hard disk set according to the hard disk identifiers in the hard disk set, wherein the write request includes: the data to be written and the identifier of the virtual node; or,

[0040] The write request is sent to the server to which each hard disk in the hard disk set belongs respectively.

[0041] Optionally, the I / O request is a read request, and the sending module is specifically used to:

[0042] The last hard disk of the hard disk set is determined, and the read request is sent to the server to which the last hard disk belongs, wherein the read request includes: an identifier of the data to be read and an identifier of the virtual node.

[0043] In a third aspect, an electronic device is provided, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0044] Memory, used to store computer programs;

[0045] The processor is used to implement any of the above-mentioned data reading and writing method steps when executing the program stored in the memory.

[0046] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned steps of the data reading and writing method.

[0047] In a fifth aspect, an embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned steps of the data reading and writing method.

[0048] The data reading and writing method, device, electronic device and medium provided by the embodiments of the present invention can determine the shards in the virtual disk for storing the data to be read and written according to the storage location in the virtual disk for storing the data to be read and written. Then determine the virtual node for storing the shard mapping of the data to be read and written, wherein the virtual node corresponds to a preset number of servers in the storage cluster. Then send an I / O request to the server corresponding to the virtual node. Since the terminal can directly interact with the server in the storage cluster without going through the metadata node in the middle, the embodiments of the present invention can shorten the path of the read and write operations and reduce the time consumed by the read and write operations, thereby reducing the occurrence of read and write operation failures due to excessive time consumed by the read and write operations.

[0049] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram of the structure of a data reading and writing system provided by an embodiment of the present invention;

[0052] Figure 2 A flow chart of a data reading and writing method provided by an embodiment of the present invention;

[0053] Figure 3 An exemplary schematic diagram of a data reading and writing system provided by an embodiment of the present invention;

[0054] Figure 4An exemplary schematic diagram of another data reading and writing system provided by an embodiment of the present invention;

[0055] Figure 5 An exemplary schematic diagram of another data reading and writing system provided by an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of the structure of a data reading and writing device provided by an embodiment of the present invention;

[0057] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1 , Figure 1 A data reading and writing system shown includes: a terminal and a storage cluster.

[0060] The terminal may be an electronic device such as a mobile phone, a tablet computer or a computer that can be installed with client software. The terminal may read data from a server of the storage cluster or write data to a server of the storage cluster.

[0061] The storage cluster includes various servers, each of which includes multiple hard disks, which are used to store data written by the terminal.

[0062] A virtual node corresponds to one or more hard disks in the storage cluster. The hard disks corresponding to each virtual node can be different, and the number of hard disks corresponding to each virtual node can also be different.

[0063] It should be noted that Figure 1 A terminal and two servers capable of communicating with the terminal are exemplarily shown, and the two servers are located in a storage cluster. The number of servers included in the storage cluster in the embodiment of the present application is not limited thereto.

[0064] Figure 2 A data reading and writing method flow chart provided in an embodiment of the present invention, the data reading and writing method is applied to Figure 1 The terminal of the data reading and writing system shown in Figure 2 , the method comprises the following steps:

[0065] Step 201 : determining a slice in the virtual disk for storing the data to be read or written according to a storage location in the virtual disk for storing the data to be read or written.

[0066] The virtual disk includes a plurality of slices, each of which is used to store data of a specified size, for example, the specified size is 4 megabytes (MB).

[0067] In one implementation, the method for determining a shard for storing data to be read or written includes the following two steps:

[0068] Step 1: Determine the starting slice corresponding to the data to be read or written according to the starting storage position for storing the data to be read or written in the virtual disk and the size of each slice in the virtual disk.

[0069] Optionally, before obtaining the starting storage location in the virtual disk for storing the data to be read or written, the terminal may request the storage cluster to obtain metadata information of the virtual disk. Metadata information is information that describes data attributes, and the terminal may determine the starting storage location in the virtual disk for storing the data to be read or written based on the metadata information. If the created virtual disk does not exist in the storage cluster, the storage cluster creates a virtual disk and then sends the metadata information of the virtual disk to the terminal. If the created virtual disk exists in the storage cluster, the metadata information of the virtual disk is sent to the terminal.

[0070] Specifically, Figure 1 The storage cluster of the data reading and writing system shown determines whether a header file (eg, AAA-header file) is stored in the storage cluster. If the header file is stored in the storage cluster, it indicates that a virtual disk has been created in the storage cluster.

[0071] As an example, if the starting storage position for storing the data to be read and written in the virtual disk obtained by the terminal is: the position of the 300MB of data stored in the virtual disk, and the size of each slice in the virtual disk is 4MB, the terminal can determine that the starting slice corresponding to the data to be read and written is: 300MB / 4MB=75, which means that the 300MB of data of the virtual disk is stored in the 75th slice, so the starting slice is the 75th slice of the virtual disk.

[0072] It is understandable that the terminal can determine the starting storage position for storing the data to be written in the virtual disk according to the size of the data stored in the virtual disk. For example, if the virtual disk has stored 200MB of data, the starting storage position of the data to be written in the virtual disk is 201MB.

[0073] The terminal may determine a starting storage position in the virtual disk for storing the data to be read according to the identifier of the data to be read.

[0074] Step 2: Determine the ending segment corresponding to the data to be read or written according to the size of the starting segment and the data to be read or written, and determine the starting segment, the ending segment, and the segments between the starting segment and the ending segment as the segments for storing the data to be read or written.

[0075] For example, if the size of the data to be read or written is 12MB, and the starting slice is the second slice of the virtual disk, then the ending slice is the fourth slice of the virtual disk. The second slice of the virtual disk stores 0MB-4MB of the data to be read or written, the third slice stores 5MB-8MB of the data to be read or written, and the fourth slice stores 9MB-12MB of the data to be read or written. The second, third, and fourth slices of the virtual disk are determined as the slices used to store the data to be read or written.

[0076] Step 202: determine a virtual node for storing a shard mapping of data to be read or written.

[0077] The virtual node corresponds to a preset number of servers in the storage cluster. It is understandable that the server corresponding to the virtual node is: the server to which the hard disk corresponding to the virtual node belongs.

[0078] Optionally, the virtual node for storing the shard mapping of the data to be read or written may be determined according to the identifier of the shard used to store the data to be read or written and the total number of virtual nodes.

[0079] For example, if the identifier of the shard used to store the data to be read or written is 3 and the total number of virtual nodes is 10, then the identifier of the virtual node mapped to the shard used to store the data to be read or written is 3%10=3, where % represents a remainder operation.

[0080] After determining the virtual node for storing the shard mapping of the data to be read or written, a hard disk set for storing the data to be read or written may be determined according to the virtual node list, wherein the hard disk set includes the hard disk corresponding to the virtual node for storing the shard mapping of the data to be read or written.

[0081] It is understandable that one virtual node corresponds to multiple hard disks in the storage cluster. To achieve data redundancy, each data to be written is stored as multiple copies when written to the storage cluster, and each hard disk corresponding to the virtual node mapped by the data to be written stores one copy.

[0082] like Figure 1 If the number of servers in the storage cluster of the data reading and writing system shown is greater than or equal to the number of hard disks corresponding to the virtual nodes, the hard disks corresponding to the virtual nodes are respectively located in different servers; if the number of servers in the storage cluster is less than the number of hard disks corresponding to the virtual nodes, the hard disks corresponding to the virtual nodes include: hard disks in different servers and different hard disks in the same server.

[0083] For example, if the number of servers in the storage cluster is 2, namely server 1 and server 2, and the number of hard disks corresponding to the virtual nodes is 2, then the hard disks corresponding to the virtual nodes include: hard disk A in server 1 and hard disk B in server 2.

[0084] For another example, the number of servers in the storage cluster is 2, namely: server 1 and server 2. The number of hard disks corresponding to the virtual nodes is 3, and the hard disks corresponding to the virtual nodes include: one hard disk in server 1 and two hard disks in server 2, or two hard disks in server 1 and one hard disk in server 2.

[0085] Since the hard disks corresponding to the virtual nodes are in different servers, if one of the servers corresponding to the virtual nodes crashes, other servers corresponding to the virtual nodes can still respond to the I / O requests sent by the terminal, thereby improving the success rate of the terminal reading or writing data.

[0086] Step 203: Send an input / output (I / O) request to the server corresponding to the virtual node.

[0087] The I / O request is used to request to read and write the data to be read or written from a storage location in the virtual disk for storing the data to be read or written.

[0088] It can be understood that the terminal reads and writes the data to be read and written from the storage location in the virtual disk, and the terminal may read and write the data to be read and written from the hard disk of the server corresponding to the virtual node mapped by the shard of the virtual disk.

[0089] In one implementation, the terminal sends an I / O request to a server to which the hard disk corresponding to the virtual node belongs. After receiving the I / O request, the server writes the data to be written into the hard disk corresponding to the virtual node, or reads the data to be read from the hard disk corresponding to the virtual node.

[0090] The data reading and writing method provided by the embodiment of the present invention can determine the shards in the virtual disk for storing the data to be read and written according to the storage location in the virtual disk for storing the data to be read and written. Then determine the virtual nodes mapped to the shards for storing the data to be read and written, wherein the virtual nodes correspond to a preset number of servers in the storage cluster. Then send an I / O request to the server corresponding to the virtual node. Since the terminal can directly interact with the server in the storage cluster without passing through the metadata node, the embodiment of the present invention can shorten the path of the read and write operations and reduce the time consumed by the read and write operations, thereby reducing the occurrence of read and write operation failures due to excessive time consumed by the read and write operations.

[0091] In the embodiment of the present invention, before determining the slice for storing the data to be read and written in step 202, the terminal also needs to obtain a virtual node list. The virtual node list includes the hard disk identifier corresponding to each virtual node and the Internet Protocol (IP) address of the server to which the hard disk belongs. The method for obtaining the virtual node list includes the following steps:

[0092] Step 1: Send a request to the specified server.

[0093] Among them, the acquisition request is used to request to obtain the virtual node list, and the specified server is: a server corresponding to an IP address in a pre-acquired configuration file. The IP address in the configuration file is: a preset number of server IP addresses in the storage cluster. In order to avoid being unable to establish a connection with the server corresponding to the IP address in the configuration file, or the server corresponding to the IP address in the configuration file is unstable, the configuration file can include multiple IP addresses. If the terminal cannot successfully connect to the server corresponding to an IP address in the configuration file, it connects to the server corresponding to another IP address in the configuration file. It can be seen that since the configuration file includes multiple IP addresses, the occurrence of the terminal being unable to obtain the virtual node list is reduced.

[0094] Optionally, the terminal can call a software development kit (SDK) to connect Figure 1 The storage cluster in the data reading and writing system shown in the figure. Then establish a long connection with the specified server, and after the connection is successful, send a request to the specified server.

[0095] Step 2: Receive the virtual node list and establish a persistent connection with the server that sends the virtual node list.

[0096] If the designated server stores a virtual node list, the terminal may receive the virtual node list sent by the designated server.

[0097] If the specified server does not store the virtual node list, the specified server will ask Figure 1 Whether other servers in the storage cluster of the data reading and writing system shown have a virtual node list stored in them. If a server in the storage cluster has a virtual node list stored in it, the server establishes a long connection with the terminal and sends the virtual node list to the terminal.

[0098] Optionally, specify that the server can query in ascending order of IP address. Figure 1 The data reading and writing system shown in the figure queries other servers in the storage cluster until a server storing a virtual node list is queried.

[0099] Optionally, in step 203, the terminal sends an I / O request to the server corresponding to the virtual node in two ways:

[0100] Case 1: When the I / O request is a write request, the terminal sends the I / O request to the server corresponding to the virtual node in the following two ways:

[0101] Optionally, the first method of sending a write request by the terminal may be applicable to a scenario where a large number of I / O requests sent by the terminal may cause a bottleneck in the use of a network card of the terminal, but is not limited thereto.

[0102] Method 1: Determine the last hard disk in the hard disk set and send a write request to the server to which the last hard disk belongs.

[0103] In the embodiment of the present invention, combined with Figure 3 , the terminal sends the last hard disk in the hard disk set (such as Figure 3 After the last hard disk writes the data to be read or written, the server to which the last hard disk belongs can send a write request to other hard disks in the hard disk set (such as Figure 3 Hard disk 2 and hard disk 1 in the synchronization synchronize the data to be written by the last hard disk.

[0104] The write request includes: data to be written, an identifier of the virtual node, and identifiers of each hard disk in the hard disk set.

[0105] After writing the data to be written, the server may also send a confirmation message to the terminal, where the confirmation message is used to indicate that the hard disk has successfully written the data to be written.

[0106] Optionally, the last hard disk of the hard disk set may be determined in descending order of hard disk IDs.

[0107] It is understandable that each server in the storage cluster includes multiple hard disks, and the server needs to determine the hard disk to which the data to be written is written according to the identifier of the virtual node. And each data to be written needs to be written to a different hard disk. After writing the data to be written, the server to which the last hard disk belongs needs to synchronize the data to be written to other hard disks in the hard disk set according to the identifiers of each hard disk in the hard disk set.

[0108] It is understandable that when some hard disks in the hard disk set have not written the data to be written, the server to which the hard disks to which the data to be written belongs receive a read request for the data to be written. At this time, the server cannot read the data to be written and the terminal fails to read the data.

[0109] To reduce the occurrence of the above situation, the terminal can send a write request to the server to which the last hard disk belongs, so that the server to which the last hard disk belongs writes the data to be written first, and then synchronizes the data to be written to other hard disks in the hard disk cluster. When the terminal reads data, it reads from the last hard disk. Since the last hard disk is the first to write the data to be written, when the terminal reads the data to be written, it is most likely to read it from the last hard disk.

[0110] Optionally, in order to distinguish data written by different users, the write request may further include a user identifier. The user identifier may be an account identifier for logging in to the terminal. It is understandable that different accounts may correspond to different virtual disks, and different users may be distinguished according to different accounts logged in by the terminal.

[0111] The write request may also include a virtual disk identifier. It is understandable that the same account may correspond to multiple virtual disks, and different virtual disks of the same account may be distinguished by the virtual disk identifier. For example, the virtual disk identifier is: 11A, which indicates virtual disk A of user 11.

[0112] It can be seen that after the terminal sends a write request to the server to which the last hard disk in the hard disk set belongs, the data to be written in each hard disk in the hard disk set can be written synchronously. Compared with the method of sending write requests to the servers to which each hard disk belongs separately, this method can reduce the flow of write requests. It avoids the problem of being unable to send write requests or sending write requests at a slow speed due to the excessive flow of write requests sent by the terminal and the limited processing capacity of the terminal's network card.

[0113] Optionally, the second method of sending a write request by the terminal may be applicable to a scenario where the number of I / O requests sent by the terminal is small and a shorter I / O operation time is required, but is not limited thereto.

[0114] Method 2: Combination Figure 4 , the terminal sends the data to each hard disk in the hard disk collection (such as Figure 4 The server to which the hard disk 1, hard disk 2 and hard disk 3 belong sends a write request. After writing the data to be written, the server can also send a confirmation message to the terminal, and the confirmation message is used to indicate that the hard disk has successfully written the data to be written.

[0115] The write request includes: the data to be written and the identifier of the virtual node. Optionally, in order to distinguish data written by different users, the write request may also include a user identifier.

[0116] It can be seen that compared with method 1, the method in which the terminal sends write requests to the servers to which each hard disk in the hard disk set belongs respectively reduces the time for writing the data to be written because each hard disk in the hard disk set can write the data to be written at the same time.

[0117] Case 2: When the I / O request is a read request, the method for the terminal to send the read request to the server corresponding to the virtual node is: determine the last hard disk in the hard disk set, and send the read request to the server to which the last hard disk belongs.

[0118] The read request includes: the identifier of the data to be read and the identifier of the virtual node. Optionally, in order to distinguish data read by different users, the read request may also include a user identifier.

[0119] It is understandable that after the server to which the last hard disk belongs receives the read request, if the server stores the data to be read, the data to be read is sent to the terminal. If the server does not store the data to be read, the data to be read is read from other hard disks corresponding to the virtual node in descending order of the hard disk identifiers. If the data to be read is read, the data to be read is sent to the terminal.

[0120] In order to more clearly explain the data reading and writing method provided by the embodiment of the present invention, the embodiment of the present invention also provides another exemplary schematic diagram of a data reading and writing system. Figure 5 shown.

[0121] Combination Figure 5 The system shown in the figure, the data reading and writing method provided by the embodiment of the present invention comprises the following steps:

[0122] Step 1: The terminal sends a request to the specified server corresponding to the IP address in the configuration file, and receives a list of virtual nodes sent by the specified server.

[0123] For example, combined with Figure 5 , the virtual node list sent by the specified server includes: virtual node 1, virtual node 2 and virtual node 3. Virtual node 1 corresponds to hard disk X of server 1 and hard disk Y of server 2.

[0124] Step 2: The terminal determines the shard for storing the data to be read or written according to the location of the data to be read or written in the virtual disk and the size of the data to be read or written, and determines the virtual node corresponding to the shard for storing the data to be read or written.

[0125] For example, combined with Figure 5 The terminal determines that the virtual node corresponding to the shard used to store the data to be read and written is virtual node 1.

[0126] Step 3: The terminal determines the hard disk corresponding to the virtual node determined in step 2 according to the virtual node list.

[0127] For example, combined with Figure 5 , the terminal determines that the hard disks corresponding to virtual node 1 are hard disk X and hard disk Y.

[0128] Step 4: The terminal writes the data to be written into each hard disk determined in step 3, or reads the data to be read from the hard disk determined in step 3.

[0129] For example, combined with Figure 5 , the terminal sends write requests to server 1 and server 2 respectively. After receiving the write request, server 1 writes the data to be written to hard disk X and sends a confirmation message to the terminal. After receiving the write request, server 2 writes the data to be written to hard disk Y and sends a confirmation message to the terminal.

[0130] Alternatively, the terminal sends a read request to server 1 or server 2. After receiving the read request, server 1 reads the data to be read from hard disk X and sends it to the terminal. Alternatively, after receiving the read request, server 2 reads the data to be read from hard disk Y and sends it to the terminal.

[0131] The embodiments of the present invention may also include the following beneficial effects: since the storage cluster of the embodiments of the present invention does not include metadata nodes, compared with the storage cluster including metadata nodes, the embodiments of the present invention can reduce the storage components of the storage cluster, reduce the total number of devices in the storage cluster, and save resources.

[0132] Moreover, if a metadata node in a storage cluster including a metadata node fails, the terminal cannot obtain the hard disk identifier corresponding to the data to be read or written, cannot read or write data in the hard disk corresponding to the data to be read or written, and the server for storing data in the storage cluster cannot process I / O requests. However, in the embodiment of the present invention, the terminal does not need to obtain the hard disk corresponding to the data to be read or written from the metadata node, and the terminal can directly interact with the server for storing data, so the embodiment of the present invention can avoid single point failure.

[0133] In addition, in an embodiment of the present invention, the terminal can directly send a write request to the server, and the server can also send a confirmation message to the terminal to indicate that the data to be written is successfully written, thereby ensuring the consistency of the data written by the terminal and the data stored on the hard disk, and improving the quality and integrity of the stored data.

[0134] In the prior art, if a hard disk in a storage cluster fails, data stored in a large number of hard disks in the storage cluster needs to be migrated.

[0135] This is because in the prior art, the identifier of the virtual node is calculated by taking the modulo of the total number of hard disks in the storage cluster, and the calculation result is determined as the hard disk corresponding to the virtual node. If a hard disk in the storage cluster fails, the total number of available hard disks in the storage cluster will change, and then the hard disks corresponding to each virtual node will change. It is necessary to re-determine the hard disk corresponding to each virtual node, and migrate the data stored in the hard disk originally corresponding to each virtual node to the hard disk newly corresponding to the virtual node. The amount of data to be migrated is huge, and it takes too much time, which is not conducive to implementation.

[0136] In the embodiment of the present invention, the hard disk corresponding to each virtual node is predetermined. If a hard disk in the storage cluster fails, it is only necessary to transfer the data stored in the failed hard disk to another hard disk, and update the hard disk corresponding to the virtual node in the virtual node list. Therefore, the embodiment of the present invention can also reduce the amount of data that needs to be migrated when a hard disk in the storage cluster fails.

[0137] For example, the virtual node list includes: virtual node 1 and virtual node 2, virtual node 1 corresponds to hard disk A and hard disk B, and virtual node 2 corresponds to hard disk C. If hard disk 1 fails, the data stored in hard disk 1 is migrated to hard disk C. And the virtual node list is updated to: virtual node 1 corresponds to hard disk C and hard disk B, and virtual node 2 corresponds to hard disk C.

[0138] Corresponding to the above method embodiment, Figure 6 As shown, an embodiment of the present invention provides a data reading and writing device, which is applied to Figure 1 The terminal of the data reading and writing system shown includes: a determining module 601 and a sending module 602.

[0139] A determination module 601 is used to determine a slice in the virtual disk for storing the data to be read or written according to a storage location in the virtual disk for storing the data to be read or written, wherein the virtual disk includes a plurality of slices, each of which is used to store data of a specified size;

[0140] The determination module 601 is further used to determine a virtual node for storing a shard mapping of the data to be read and written, where the virtual node corresponds to a preset number of servers in the storage cluster;

[0141] The sending module 602 is used to send an input / output I / O request to the server corresponding to the virtual node determined by the determining module. The I / O request is used to request to read and write the data to be read and written from the storage location.

[0142] Optionally, the device may further include: an acquisition module;

[0143] The acquisition module is used to obtain a virtual node list before determining the shard for storing the data to be read and written in the virtual disk according to the storage location of the data to be read and written in the virtual disk, wherein the virtual node list includes the hard disk identifier corresponding to each virtual node and the Internet Protocol IP address of the server to which the hard disk belongs.

[0144] Optionally, the acquisition module can be used to:

[0145] Send a request to the specified server to obtain a list of virtual nodes.

[0146] Receive the virtual node list and establish a persistent connection with the server that sent the virtual node list.

[0147] Optionally, the determination module 601 may be specifically configured to:

[0148] Determine the starting slice corresponding to the data to be read or written according to the starting storage position for storing the data to be read or written in the virtual disk and the size of each slice in the virtual disk;

[0149] According to the size of the starting segment and the data to be read or written, the ending segment corresponding to the data to be read or written is determined, and the starting segment, the ending segment and the segments between the starting segment and the ending segment are determined as the segments for storing the data to be read or written.

[0150] Optionally, determination module 601 can also be used to determine a hard disk set for storing the data to be read and written according to a virtual node list after determining the virtual node for storing the shard mapping of the data to be read and written, the hard disk set including the hard disk corresponding to the virtual node for storing the shard mapping of the data to be read and written.

[0151] Optionally, the I / O request is a write request, and the sending module 602 may be specifically used for:

[0152] Determine the last hard disk in the hard disk set, and send a write request to the server to which the last hard disk belongs, so that after the last hard disk writes the data to be read or written, the data to be written written by the last hard disk is synchronized to other hard disks in the hard disk set according to the hard disk identifiers in the hard disk set, and the write request includes: the data to be written and the identifier of the virtual node; or,

[0153] Send a write request to the server to which each hard disk in the hard disk set belongs.

[0154] Optionally, the I / O request is a read request, and the sending module 602 may be specifically used for:

[0155] The last hard disk of the hard disk set is determined, and a read request is sent to the server to which the last hard disk belongs, wherein the read request includes: an identifier of the data to be read and an identifier of the virtual node.

[0156] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0157] Memory 703, used for storing computer programs;

[0158] The processor 701 is used to implement the steps performed by the terminal in the above method embodiment when executing the program stored in the memory 703.

[0159] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0160] The communication interface is used for communication between the above electronic device and other devices.

[0161] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0162] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0163] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data reading and writing methods are implemented.

[0164] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any one of the data reading and writing methods in the above embodiments.

[0165] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0166] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0167] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data reading and writing method, characterized in that: Applied to a terminal, the method comprises: According to a storage location in the virtual disk for storing the data to be read or written, determining a slice in the virtual disk for storing the data to be read or written, wherein the virtual disk includes a plurality of slices, each slice being used to store data of a specified size; Determine a virtual node for storing the shard mapping of the data to be read and written, the virtual node corresponds to a preset number of servers in the storage cluster, the server corresponding to the virtual node is the server to which the hard disk corresponding to the virtual node belongs, and the preset number of hard disks corresponding to the virtual node are all used to store a copy of the data to be read and written; Sending an input / output (I / O) request to a server corresponding to the virtual node, wherein the I / O request is used to request reading and writing the data to be read or written from the storage location; The determining of a virtual node for storing the shard mapping of the data to be read or written comprises: According to the identifier of the shard used to store the data to be read or written and the total number of virtual nodes, a virtual node for storing the shard mapping of the data to be read or written is determined.

2. The method according to claim 1, characterized in that Before determining, according to the storage location in the virtual disk for storing the data to be read or written, the slice in the virtual disk for storing the data to be read or written, the method further comprises: A virtual node list is obtained, wherein the virtual node list includes a hard disk identifier corresponding to each virtual node and an Internet Protocol IP address of a server to which the hard disk belongs.

3. The method according to claim 2, characterized in that The step of obtaining a virtual node list includes: Sending a request to a designated server, wherein the request is used to request to obtain the virtual node list; The virtual node list is received, and a persistent connection is established with a server that sends the virtual node list.

4. The method according to claim 3, characterized in that: The determining, according to a storage location in the virtual disk for storing the data to be read or written, a slice in the virtual disk for storing the data to be read or written, comprises: Determine the starting slice corresponding to the data to be read or written according to the starting storage position for storing the data to be read or written in the virtual disk and the size of each slice in the virtual disk; According to the size of the starting segment and the data to be read or written, the ending segment corresponding to the data to be read or written is determined, and the starting segment, the ending segment and the segments between the starting segment and the ending segment are determined as the segments for storing the data to be read or written.

5. The method according to claim 3, characterized in that: After determining the virtual node for storing the shard mapping of the data to be read and written, the method further includes: According to the virtual node list, a hard disk set for storing the data to be read or written is determined, wherein the hard disk set includes hard disks corresponding to virtual nodes for storing shard mapping of the data to be read or written.

6. The method according to claim 5, characterized in that The I / O request is a write request, and the sending of the input / output I / O request to the server corresponding to the virtual node includes: Determine the last hard disk of the hard disk set, and send the write request to the server to which the last hard disk belongs, so that after the last hard disk writes the data to be read and written, the data to be written written by the last hard disk is synchronized to other hard disks in the hard disk set according to the hard disk identifiers in the hard disk set, wherein the write request includes: the data to be written and the identifier of the virtual node; or, The write request is sent to the server to which each hard disk in the hard disk set belongs respectively.

7. The method according to claim 5, characterized in that The I / O request is a read request, and the sending of the input / output I / O request to the server corresponding to the virtual node includes: The last hard disk of the hard disk set is determined, and the read request is sent to the server to which the last hard disk belongs, wherein the read request includes: an identifier of the data to be read and an identifier of the virtual node.

8. A data reading and writing device, characterized in that: Applied to a terminal, the device comprises: A determination module, configured to determine, according to a storage location in the virtual disk for storing the data to be read or written, a slice in the virtual disk for storing the data to be read or written, wherein the virtual disk includes a plurality of slices, each slice being used to store data of a specified size; The determination module is further used to determine a virtual node for storing the shard mapping of the data to be read and written, the virtual node corresponds to a preset number of servers in the storage cluster, the server corresponding to the virtual node is the server to which the hard disk corresponding to the virtual node belongs, and the preset number of hard disks corresponding to the virtual node are all used to store a copy of the data to be read and written; A sending module, used for sending an input / output I / O request to the server corresponding to the virtual node determined by the determining module, wherein the I / O request is used for requesting to read and write the data to be read and written from the storage location; The determination module is specifically used for: According to the identifier of the shard used to store the data to be read or written and the total number of virtual nodes, a virtual node for storing the shard mapping of the data to be read or written is determined.

9. The device according to claim 8, characterized in that The device also includes: an acquisition module; The acquisition module is used to obtain a virtual node list before determining the storage location of the data to be read and written in the virtual disk and the shards to store the data to be read and written in the virtual disk. The virtual node list includes the hard disk identifier corresponding to each virtual node and the Internet Protocol IP address of the server to which the hard disk belongs.

10. The device according to claim 9, characterized in that The acquisition module is specifically used for: Sending a request to a designated server, wherein the request is used to request to obtain the virtual node list; The virtual node list is received, and a persistent connection is established with a server that sends the virtual node list.

11. The device according to claim 10, characterized in that The determination module is specifically used for: Determine the starting slice corresponding to the data to be read or written according to the starting storage position for storing the data to be read or written in the virtual disk and the size of each slice in the virtual disk; According to the size of the starting segment and the data to be read or written, the ending segment corresponding to the data to be read or written is determined, and the starting segment, the ending segment and the segments between the starting segment and the ending segment are determined as the segments for storing the data to be read or written.

12. The device according to claim 10, characterized in that The determination module is also used to determine, after determining the virtual node for storing the shard mapping of the data to be read and written, a hard disk set for storing the data to be read and written according to the virtual node list, wherein the hard disk set includes hard disks corresponding to the virtual nodes for storing the shard mapping of the data to be read and written.

13. The device according to claim 12, characterized in that The I / O request is a write request, and the sending module is specifically used for: Determine the last hard disk of the hard disk set, and send the write request to the server to which the last hard disk belongs, so that after the last hard disk writes the data to be read and written, the data to be written written by the last hard disk is synchronized to other hard disks in the hard disk set according to the hard disk identifiers in the hard disk set, wherein the write request includes: the data to be written and the identifier of the virtual node; or, The write request is sent to the server to which each hard disk in the hard disk set belongs respectively.

14. The device according to claim 12, characterized in that The I / O request is a read request, and the sending module is specifically used for: The last hard disk of the hard disk set is determined, and the read request is sent to the server to which the last hard disk belongs, wherein the read request includes: an identifier of the data to be read and an identifier of the virtual node.

15. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images