Data transmission method, data writing method, data reading method and equipment

Abstract

The invention provides a data transmission method, a data writing method, a data reading method and equipment. The data transmission method is applied to a master node of the raft storage system, the raft storage system comprises the master node and slave nodes, the number of the slave nodes is at least one, and the method comprises the steps that in response to a target data write-in request, target data, the data size of the target data and a first offset position pre-distributed by a target disk are obtained, the target disk is used for writing the target data; and sending the target data, the data size of the target data and the first offset position to a slave node, so that the slave node stores the target data to a target disk based on the data size and the first offset position.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a data transmission method, a data writing method, a data reading method and a device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

[0003] In multi-copy storage systems, many use Raft to ensure cluster data consistency. After all requests reach the Leader node, they are sent to other nodes through the Raft protocol to ensure multiple copies of data. However, Raft consumes a lot of performance when transmitting large blocks of data, thus affecting the overall system performance.

[0004] Based on this, the present application urgently needs to propose a data transmission method, a data writing method, a data reading method and a device that can solve the above-mentioned technical problems. Summary of the invention

[0005] Multiple aspects of the present application provide a data transmission method, a data writing method, a data reading method and a device to reduce the consumption of system performance during data transmission and improve the overall system performance.

[0006] In one aspect of the present application, a data transmission method is provided, which is applied to a master node of a raft storage system, wherein the raft storage system includes a master node and a slave node, and the number of the slave node is at least one, and the method includes:

[0007] In response to a target data write request, acquiring the target data, the data size of the target data, and a first offset position pre-allocated by a target disk, the target disk being used to write the target data;

[0008] The target data, the data size of the target data, and the first offset position are sent to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position.

[0009] Furthermore, the method also includes: establishing a custom connection channel to communicatively connect the master node and the slave node; establishing a raft protocol to communicatively connect the master node and the slave node, and the master node and the target disk; wherein the master node obtains the data size of the target data and the first offset position allocated to the target disk in response to a target data write request, including: the master node obtains the data size of the target data and the first offset position allocated to the target disk through the raft protocol in response to the target data write request; the master node sends the data size of the target data and the first offset position to the slave node, including: the master node sends the target data, the data size of the target data and the first offset position through the custom connection channel.

[0010] Furthermore, the master node sends the target data, the data size of the target data and the first offset position to the slave node so that after the slave node writes the target data to the target disk, the method also includes: receiving the data writing status returned by the slave node and generating a data writing result; wherein, generating the data writing result includes: if the data writing status returned by any of the slave nodes is successful, then it is determined that the target data is successfully written to the target disk; otherwise, it is determined that the target data is failed to be written to the target disk.

[0011] Furthermore, after the master node sends the target data, the data size of the target data and the first offset position to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position, the method also includes: the master node obtains the data storage location of the target data and the disk identifier of the target disk; the master node generates meta information based on the data storage location, the disk identifier and the data size of the target data; the master node writes the meta information to the slave node through the raft protocol.

[0012] Furthermore, after the master node writes the meta-information to the slave node through the raft protocol, the method further includes: receiving the meta-information writing status returned by the slave node, and generating a meta-information writing result; wherein generating the meta-information writing result includes: if the meta-information writing status returned by the slave nodes exceeding a preset ratio is successful, then determining that the writing of the meta-information to the slave node is successful; otherwise, determining that the writing of the meta-information to the slave node fails; wherein the preset ratio is that the number of target slave nodes accounts for 50% of the number of slave nodes, and the target slave node is a slave node that returns the meta-information writing status as successful.

[0013] Furthermore, the method further includes: the master node sends the meta information writing result and the data writing result to a client of the raft storage system.

[0014] Furthermore, after sending the target data, the data size of the target data and the first offset position to the slave node, the method also includes: determining a second offset position of the target disk based on the first offset position and the data size of the target data; and updating the second offset position of the target disk to the first offset position pre-allocated to the target disk.

[0015] Another aspect of the present application provides a data writing method for writing target data, wherein the data is transmitted to a slave node based on the data transmission method as described above, and the method comprises:

[0016] In response to the target data write request, the slave node receives the first offset position, the target data, and the data size of the target data sent by the master node;

[0017] The target data is written to a target disk based on the data size and the first offset position.

[0018] In another aspect of the present application, a data reading method is provided for reading target data, wherein the data is written to a target disk based on the data writing method as described above, and the method comprises:

[0019] In response to the target data read request, the slave node obtains the meta information of the target data;

[0020] The slave node obtains the data storage location of the target data and the data size of the target data based on the meta information of the target data, and reads the target data.

[0021] In another aspect of the present application, a data transmission device is provided, the device comprising:

[0022] a response unit, configured to obtain, in response to a target data write request, the target data, the data size of the target data, and a first offset position pre-allocated on a target disk, the target disk being used to write the target data;

[0023] A sending unit is used to send the target data, the data size of the target data and the first offset position to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position.

[0024] In another aspect of the present application, a data writing device is provided, the device comprising:

[0025] A receiving unit, configured to receive, from the slave node in response to a target data write request, a first offset position, target data, and a data size of the target data sent by the master node;

[0026] A writing unit is used to write the target data from the node to the target disk based on the data size and the first offset position.

[0027] In another aspect of the present application, a data reading device is provided, the device comprising:

[0028] an acquisition unit, configured to acquire meta information of the target data from a node in response to a target data read request;

[0029] The reading unit is used to obtain the data storage location of the target data and the data size of the target data from the node based on the meta information of the target data, and read the target data.

[0030] In another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0031] at least one processor; and

[0032] a memory communicatively coupled to the at least one processor;

[0033] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0034] In another aspect of the present application, a computer-readable medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method as described above.

[0035] In another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method as described above is implemented.

[0036] The present application proposes a data transmission method, a data writing method, a data reading method and a device applied to a raft storage system. The data transmission method comprises the following steps: in response to a target data writing request, obtaining the target data, the data size of the target data and a first offset position pre-allocated by a target disk, the target disk being used to write the target data; sending the target data, the data size of the target data and the first offset position to a slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position;

[0037] The above data transmission method realizes decoupling processing of target data and metadata (data size of target data and first offset position) by sending target data, data size of target data and first offset position respectively, and the target data and metadata are sent to slave nodes respectively; the performance of the system is greatly improved by controlling the target data and metadata separately; and the write flow of target data is controlled separately, which is also beneficial for expanding system functions (such as controlling the flow rate of requests). BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0040] Figure 1 A flowchart of a data transmission method provided in one embodiment of the present application;

[0041] Figure 2 A schematic diagram of a data writing method according to an embodiment of the present application;

[0042] Figure 3 A schematic diagram of a data reading method according to an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of a data transmission device provided in one embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of a data writing device provided in yet another embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a data reading device provided in yet another embodiment of the present application;

[0046] Figure 7 A schematic diagram of the structure of an electronic device suitable for implementing the solution in the embodiments of the present application;

[0047] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0050] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0051] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0052] An embodiment of the present application provides a data transmission method, which is applied to a master node of a raft storage system, wherein the raft storage system includes a master node and a slave node, and the number of the slave node is at least one, and the method includes:

[0053] In response to a target data write request, acquiring the target data, the data size of the target data, and a first offset position pre-allocated by a target disk, the target disk being used to write the target data;

[0054] The target data, the data size of the target data, and the first offset position are sent to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position.

[0055] In actual scenarios, the execution subject of the method can be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0056] Embodiment 1

[0057] Figure 1 The processing flow of a data transmission method provided by an embodiment of the present application is shown. The method is applied to a master node of a raft storage system. The raft storage system includes a master node and a slave node. The number of the slave node is at least one. The method includes at least the following processing steps:

[0058] Step S101, in response to a target data write request, obtaining the target data, the data size of the target data and a first offset position pre-allocated by a target disk, the target disk being used to write the target data;

[0059] Step S102: Send the target data, the data size of the target data, and the first offset position to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position.

[0060] It should be understood that there is no binding relationship between the disk and all nodes of the storage system. All nodes can correspond to multiple disks. After one of the disks corresponding to each node is made into a raft group, the group of disks can be managed as a whole.

[0061] The data size of the target data is size, and the first offset position is offset, so that the slave node stores the target data to the target disk based on the data size and the first offset position as follows: storing the target data to the target disk based on size+offset.

[0062] In one embodiment, the method further includes: establishing a custom connection channel to communicatively connect the master node and the slave node; establishing a raft protocol to communicatively connect the master node and the slave node, and the master node and the target disk; wherein the master node obtains the data size of the target data and the first offset position allocated to the target disk in response to a target data write request, including: the master node obtains the data size of the target data and the first offset position allocated to the target disk through the raft protocol in response to the target data write request; the master node sends the data size of the target data and the first offset position to the slave node, including: the master node sends the target data, the data size of the target data and the first offset position through the custom connection channel.

[0063] Among them, the custom connection channel is different from the intra-group connection established by the raft protocol itself, and can send data by itself. The custom channel is used to process replication work. By establishing a custom channel, data with large data volume is extracted and then transmitted by the custom link channel; metadata with small data volume is transmitted through the raft protocol to avoid the energy consumption of large data blocks in the raft protocol, so as to improve system performance.

[0064] In one embodiment, the master node sends the target data, the data size of the target data and the first offset position to the slave node so that after the slave node writes the target data to the target disk, the method further includes: receiving the data write status returned by the slave node and generating a data write result; wherein, generating the data write result includes: if the data write status returned by any of the slave nodes is successful, then it is determined that the target data is successfully written to the target disk; otherwise, it is determined that the target data is failed to be written to the target disk.

[0065] Specifically, to determine whether the target data is written to the target disk successfully, it is necessary to ensure that the data writing status returned by any slave node is successful. If the data writing status returned by one of the slave nodes is failed, it is considered that the target data has failed to be written to the target disk, and the target data writing step is re-executed.

[0066] In one embodiment, after the master node sends the target data, the data size of the target data and the first offset position to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position, the method further includes: the master node obtains the data storage location of the target data and the disk identifier of the target disk; the master node generates metadata based on the data storage location, the disk identifier and the data size of the target data; the master node writes the metadata to the slave node through the raft protocol.

[0067] Specifically, the metadata of each data is generated and sent to each slave node respectively, which ensures the retrievability of the metadata (to prevent the metadata of some slave nodes from being lost and difficult to retrieve); on the other hand, ensures the data reading authority of each slave node.

[0068] In one embodiment, after the master node writes the meta-information to the slave node through the raft protocol, the method further includes: receiving the meta-information writing status returned by the slave node, and generating a meta-information writing result; wherein generating the meta-information writing result includes: if the meta-information writing status returned by the slave nodes exceeding a preset ratio is successful, then determining that the writing of the meta-information to the slave node is successful; otherwise, determining that the writing of the meta-information to the slave node fails; wherein the preset ratio is that the number of target slave nodes accounts for 50% of the number of slave nodes, and the target slave node is a slave node that returns the meta-information writing status as successful.

[0069] Among them, if the metadata writing status returned by slave nodes exceeding the preset ratio is successful, the reason for determining that the metadata is written to the slave node successfully is that the raft protocol can ensure that when the metadata writing status returned by more than n / 2 slave nodes is successful, the metadata is considered to be written correctly, and the remaining rafts have logs that can be synchronized to the failed slave nodes; where n is the total number of slave nodes.

[0070] In one embodiment, the method further includes: the master node sends the meta information writing result and the data writing result to the client of the raft storage system, so as to ensure that the client can obtain the meta information writing result and the data writing result in the first time, and ensure that the client has sufficient right to know the data writing status.

[0071] In one embodiment, after sending the target data, the data size of the target data and the first offset position to the slave node, the method further includes: determining a second offset position of the target disk based on the first offset position and the data size of the target data; and updating the second offset position of the target disk to the first offset position pre-allocated to the target disk.

[0072] The data transmission logic of the commonly used data transmission methods in the prior art is often as follows: when data is written in raft, it is written to the disk as a log, and then the data is read from the log disk through submission and written to the target disk; while the data transmission method described in the present application can independently control the data writing stream, and can directly send the data to the target disk, eliminating the intermediate reading and writing process of the log disk, thereby greatly improving the performance of the system; and due to the lightweight attribute of the meta-information, data consistency can still be guaranteed through the raft protocol, and independently controlling the data writing stream can ensure that the system has better functional scalability (such as controlling the data request speed, etc.).

[0073] Embodiment 2

[0074] Figure 2 The processing flow of a data writing method provided by an embodiment of the present application is shown, which is used to write target data. The data is transmitted to the slave node based on the data transmission method as described above. The method includes at least the following processing steps:

[0075] Step S201: In response to a target data write request, the slave node receives a first offset position, target data, and a data size of the target data sent by the master node;

[0076] Step S202: writing the target data to a target disk based on the data size and the first offset position.

[0077] Embodiment 3

[0078] Figure 3 The processing flow of a data acquisition method provided by an embodiment of the present application is shown, which is used to read target data. The data is written to the target disk based on the data writing method as described above. The method at least includes the following processing steps:

[0079] Step S301: The slave node obtains the meta information of the target data in response to a target data read request;

[0080] Step S302: The slave node obtains the data storage location of the target data and the data size of the target data based on the meta-information of the target data, and reads the target data.

[0081] It should be understood that when reading the target data written based on the data writing method as described in Example 2, the storage location of the target data can be found according to the metadata of the target data to read the local data from the node, and all nodes can perform concurrent reading.

[0082] Embodiment 4

[0083] Figure 4A schematic diagram of the structure of a data transmission device provided in an embodiment of the present application is shown, and the device at least includes:

[0084] a response unit, configured to obtain, in response to a target data write request, the target data, the data size of the target data, and a first offset position pre-allocated on a target disk, the target disk being used to write the target data;

[0085] A sending unit is used to send the target data, the data size of the target data and the first offset position to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position.

[0086] Embodiment 5

[0087] Figure 5 A schematic diagram of the structure of a data writing device provided in an embodiment of the present application is shown, and the device at least includes:

[0088] A receiving unit, configured to receive, from the slave node in response to a target data write request, a first offset position, target data, and a data size of the target data sent by the master node;

[0089] A writing unit is used to write the target data from the node to the target disk based on the data size and the first offset position.

[0090] Embodiment 6

[0091] Figure 6 A schematic diagram of the structure of a data reading device provided in an embodiment of the present application is shown, and the device at least includes:

[0092] an acquisition unit, configured to acquire meta information of the target data from a node in response to a target data read request;

[0093] The reading unit is used to obtain the data storage location of the target data and the data size of the target data from the node based on the meta information of the target data, and read the target data.

[0094] Embodiment 7

[0095] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the data transmission method, data writing method and data reading method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0096] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0097] Figure 7 The structure of an electronic device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 500 includes a central processing unit (CPU, Central Processing Unit) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 502 or the program loaded from the storage part 508 to the random access memory (RAM, Random Access Memory) 503. In RAM503, various programs and data required for system operation are also stored. CPU 501, ROM 502 and RAM 503 are connected to each other through a bus 504. An input / output (I / O, Input / Output) interface 505 is also connected to the bus 504.

[0098] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 508 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet.

[0099] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, the above functions defined in the method of the present application are executed.

[0100] Embodiment 8

[0101] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0102] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0103] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device.

[0104] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0105] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0110] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0111] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0113] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A data transmission method, characterized in that: The method is applied to a master node of a raft storage system, the raft storage system includes a master node and a slave node, the number of the slave node is at least one, and the method includes: In response to a target data write request, acquiring the target data, the data size of the target data, and a first offset position pre-allocated by a target disk, the target disk being used to write the target data; The target data, the data size of the target data, and the first offset position are sent to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position.

2. The data transmission method according to claim 1, characterized in that: The method further comprises: Establishing a custom connection channel to communicate with the master node and the slave node; Establishing a raft protocol to communicate between the master node and the slave node, and the master node and the target disk; The master node obtains the data size of the target data and the first offset position of the target disk allocation in response to the target data write request, including: the master node obtains the data size of the target data and the first offset position of the target disk allocation through the raft protocol in response to the target data write request; The master node sends the data size of the target data and the first offset position to the slave node, including: the master node sends the target data, the data size of the target data and the first offset position through the custom connection channel.

3. The data transmission method according to claim 1 or 2, characterized in that: The master node sends the target data, the data size of the target data and the first offset position to the slave node so that the slave node writes the target data to the target disk. The method further includes: Receive the data writing status returned by the slave node and generate a data writing result; The generated data writing result includes: If the data writing status returned by any of the slave nodes is successful, it is determined that the target data is written into the target disk successfully; otherwise, it is determined that the target data is written into the target disk unsuccessfully.

4. The data transmission method according to claim 3, characterized in that: After the master node sends the target data, the data size of the target data, and the first offset position to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position, the method further includes: The master node obtains the data storage location of the target data and the disk identifier of the target disk; The master node generates meta information based on the data storage location, the disk identifier and the data size of the target data; The master node writes the meta information to the slave node through the raft protocol.

5. The data transmission method according to claim 4, characterized in that: After the master node writes the meta information to the slave node through the raft protocol, the method further includes: Receiving the meta information writing status returned by the slave node, and generating a meta information writing result; The generating of the meta information writing result includes: if the meta information writing status returned by the slave nodes exceeding the preset ratio is successful, then determining that the writing of the meta information to the slave node is successful; otherwise, then determining that the writing of the meta information to the slave node fails; The preset ratio is that the number of target slave nodes accounts for 50% of the number of slave nodes, and the target slave node is a slave node that returns the meta-information writing status as successful.

6. The data transmission method according to claim 5, characterized in that: The method further comprises: The master node sends the meta information writing result and the data writing result to the client of the raft storage system.

7. The data transmission method according to claim 6, characterized in that: After sending the target data, the data size of the target data, and the first offset position to the slave node, the method further includes: Determining a second offset position of the target disk based on the first offset position and the data size of the target data; The second offset position of the target disk is updated to be the first offset position pre-allocated to the target disk.

8. A data writing method, characterized in that: For writing target data, the data is transmitted to the slave node based on the data transmission method according to any one of claims 1 to 7, the method comprising: In response to the target data write request, the slave node receives the first offset position, the target data, and the data size of the target data sent by the master node; The target data is written to a target disk based on the data size and the first offset position.

9. A data reading method, characterized in that: For reading target data, the data is written into the target disk based on the data writing method according to claim 8, the method comprising: In response to the target data read request, the slave node obtains the meta information of the target data; The slave node obtains the data storage location of the target data and the data size of the target data based on the meta information of the target data, and reads the target data.

10. A data transmission device, characterized in that: The device comprises: a response unit, configured to obtain, in response to a target data write request, the target data, the data size of the target data, and a first offset position pre-allocated on a target disk, the target disk being used to write the target data; A sending unit is used to send the target data, the data size of the target data and the first offset position to the slave node, so that the slave node stores the target data to the target disk based on the data size and the first offset position.

11. A data writing device, characterized in that: The device comprises: A receiving unit, configured to receive, from the slave node in response to a target data write request, a first offset position, target data, and a data size of the target data sent by the master node; A writing unit is used to write the target data from the node to the target disk based on the data size and the first offset position.

12. A data reading device, characterized in that: The device comprises: an acquisition unit, configured to acquire meta information of the target data from a node in response to a target data read request; The reading unit is used to obtain the data storage location of the target data and the data size of the target data from the node based on the meta information of the target data, and read the target data.

13. An electronic device, comprising: at least one processor; as well as A memory communicatively connected to the at least one processor; characterized in that The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

14. A computer readable medium having computer program instructions stored thereon, characterized in that: The computer program instructions are executable by a processor to implement the method according to any one of claims 1 to 9.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images