A data reading and writing system and a data writing method

By introducing a separate structure of computing and storage in the data read and write system, and synchronizing data with control devices, write devices and shared devices, data loss and sharing difficulties caused by stand-alone failures are solved, and high availability and consistency of data are achieved , improves system fault tolerance and application startup efficiency.

CN113760926BActive Publication Date: 2024-11-15BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202011401261.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-02
Publication Date
2024-11-15
Estimated Expiration
2040-12-02

AI Technical Summary

Technical Problem

In the prior art, single-machine failures lead to easy data loss, difficult data sharing, and inconsistent backup machine reading, resulting in large delay in application startup.

Method used

A data reading and writing system is designed, including control devices, writing devices, shared devices and shared storage. Through the separation structure of calculation and storage, data is written to the shared storage and uploaded to the log in response to the data write request. Servers, shared devices synchronize data to realize multi-machine sharing and consistency of data.

Benefits of technology

High availability and consistency of data is achieved, data transmission between devices is avoided, fault tolerance of the system is improved, multiple data writing operations can be performed simultaneously, and application startup delay is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113760926B_ABST
    Figure CN113760926B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses a data reading and writing system and a data writing method, the system comprising: a control device, a writing device, a sharing device, a shared storage and a log server, wherein: the control device is used to respond to a detected data writing request, determine the writing device and the sharing device associated with the data writing request, and send the data to be written to the writing device; the writing device is used to write the data to be written to the shared storage and upload the writing information to the log server; the sharing device is used to obtain the writing information in the log server and synchronize the data to be written according to the writing information. The system provided by the embodiment of the present invention adopts a structure in which computing and storage are separated, so that multiple data writing operations can be executed simultaneously, avoiding data transmission between devices and improving the fault tolerance of the system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a data reading and writing system and a data writing method. Background Art

[0002] At present, data storage is mainly stored on a single machine, or on a backup machine through backup. In the process of implementing the present invention, the inventor found that there are at least the following technical problems in the prior art: when a single machine fails, data is easily lost; data cannot be shared by other machines, and data needs to be reloaded on the single machine, resulting in a large delay in application startup; and the backup machine reads inconsistent data. Summary of the invention

[0003] The embodiment of the present invention provides a data reading and writing system writing method and a data writing method to realize data sharing, so that data can be directly accessed when a service is started, and the consistency of data is guaranteed.

[0004] In a first aspect, an embodiment of the present invention provides a data reading and writing system, including: a control device, a writing device, a sharing device and a shared storage, wherein:

[0005] The control device is used to determine the writing device and the shared device associated with the data writing request in response to the detected data writing request, and send the data to be written to the writing device;

[0006] The writing device is used to write the data to be written into the shared storage and upload the writing information to the log server;

[0007] The shared device is used to obtain the write information in the log server and synchronize the data to be written according to the write information.

[0008] In a second aspect, an embodiment of the present invention further provides a data writing method, which is executed by a control device in a data reading and writing system provided by any embodiment of the present invention, comprising:

[0009] In response to the detected data write request, determining the write device and the shared device associated with the data write request;

[0010] The data to be written is sent to the writing device, so that the writing device writes the data to be written into the shared storage, and uploads the writing information to the log server, and the shared device performs data synchronization according to the writing information in the log server.

[0011] The data reading and writing system provided by the embodiment of the present invention responds to the detected data writing request through the control device to determine the writing device and the shared device associated with the data writing request; the writing device writes the data to be written into the shared storage and uploads the writing information to the log server; the shared device obtains the writing information in the log server and synchronizes the data to be written according to the writing information, adopts a structure in which computing and storage are separated, so that multiple data writing operations can be executed simultaneously, avoiding data transmission between devices and improving the fault tolerance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a structural schematic diagram of a data reading and writing system provided by the first embodiment of the present invention;

[0013] Figure 2a This is a system architecture diagram of a data reading and writing system provided by Embodiment 2 of the present invention;

[0014] Figure 2b This is a schematic diagram of generating a block identifier provided by the second embodiment of the present invention;

[0015] Figure 2c It is a flowchart of a data writing method provided by Embodiment 2 of the present invention;

[0016] Figure 2d is a flow chart of a data merging method provided in Embodiment 2 of the present invention;

[0017] Figure 3 is a flow chart of a data writing method provided by Embodiment 3 of the present invention;

[0018] Figure 4 is a structural schematic diagram of a data writing device provided by a fourth embodiment of the present invention;

[0019] Figure 5 It is a structural diagram of a computer device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0021] Embodiment 1

[0022] Figure 1 1 is a schematic diagram of the structure of a data reading and writing system provided in Embodiment 1 of the present invention. The system provided in this embodiment can be used for reading and writing data.

[0023] like Figure 1 As shown, the system includes a control device 110, a writing device 120, a sharing device 130, a shared storage 140 and a log server 150. Among them: the control device 110 is used to respond to a detected data writing request and determine the writing device and the sharing device associated with the data writing request; the writing device 120 is used to write the data to be written into the shared storage 140 and upload the writing information to the log server; the sharing device 130 is used to obtain the writing information in the log server and synchronize the data to be written according to the writing information.

[0024] In this embodiment, the control device determines the writing device and the shared device associated with the data to be written, the writing device writes the data to be written into the shared storage and uploads the writing information to the log server, and the shared device synchronizes the data to be written through the writing information in the log server, thereby realizing the decoupling of computing and storage, so that multiple writing operations can be performed simultaneously without conflicts, and the fault tolerance of the system is improved. Specifically, after receiving the data writing request, the control device divides the data to be written in the data writing request according to the number of shards to obtain multiple shards, and for each shard, determines the writing device and the shared device corresponding to the shard, and sends the shard to the writing device, the writing device writes the received shard into the shared storage, and then uploads the writing information to the log server, the shared device monitors the writing information in the log server, and obtains data from the shared storage for synchronization according to the writing information.

[0025] In one embodiment of the present invention, the determination of the write device and the shared device associated with the data write request includes: determining the write shard of the data to be written in the data write request, and determining the write device and the shared device based on the replica device associated with the write shard. Optionally, the shared device includes multiple shards, each shard corresponds to multiple replica devices, and data is written through the replica devices corresponding to the shards. By setting a shard to correspond to multiple replica devices, different write operations within the shard can be performed separately through the multiple replica devices associated with the shard, thereby realizing multiple write operations on the same shard at the same time.

[0026] Specifically, multiple replica devices associated with the write shard are obtained, one of the replica devices is selected as the write device, and the replica devices other than the write device are used as the shared devices. Any replica device associated with the shard can be randomly selected as the write device, the data to be written is written to the shared storage, the replica devices other than the write device are used as the shared devices, and the write information in the log server is monitored, so that after the write device writes the data to be written to the shared storage, the data to be written is synchronized through the log server.

[0027] In one embodiment of the present invention, the step of writing the data to be written into the shared storage and uploading the write information to the log server includes: writing the data to be written into the shared storage, generating a write storage identifier corresponding to the data to be written based on the data write path, and generating the write information based on the write storage identifier and uploading it to the log server. Specifically, the write information is generated based on the storage identifier of the data to be written. After the writing device writes the data to be written into the shared storage, a storage identifier is generated according to the data write path, and write information is generated according to the storage identifier, so that the shared device can synchronize the data to be written according to the storage identifier in the write information.

[0028] In this embodiment, the write storage identifier includes the partition identifier corresponding to the data to be written and the block identifier of the data to be written. Among them, the partition identifier is carried in the data to be written and is used to characterize the data service to be written; the block identifier is determined based on the data write path of the data to be written. Optionally, the writing device applies for a reference block identifier for data writing, generates a self-increment identifier based on the applied reference block identifier, and uses the generated self-increment identifier as the block identifier of the data to be written. Exemplarily, the storage identifier can be represented by a partition identifier + a block identifier.

[0029] On the basis of the above scheme, it is considered that each time data is written, a data block will be generated, resulting in an increase in the number of data blocks when the number of data writes increases, which reduces the performance of the database. Therefore, small data blocks can be merged. Optionally, the writing device is also used to: determine the data to be merged according to the storage identifier; merge the data to be merged to obtain merged data, and determine the merge storage identifier corresponding to the merged data; upload the merge information to the log server. Among them, the storage identifier includes a partition identifier and a block identifier. Based on the storage identifier, the data to be merged can be determined as the data to be merged, the data to be merged can be merged to obtain new merged data, and the mark of the old data to be merged can be modified to complete the data merge. After the data merge is completed, the merge information is uploaded to the log server so that the shared device can synchronize data according to the merge information in the log server to ensure data consistency.

[0030] Optionally, determining the storage block to be merged according to the storage identifier includes: taking the data corresponding to the storage identifiers with continuous block identifiers and the same partition identifier as the data to be merged. When the partition identifiers are the same, it means that the business logic of the data is the same, so the data corresponding to the storage identifiers with the same partition identifier and continuous block identifiers can be taken as the data to be merged. Determining the data to be merged based on the partition identifier and the block identifier makes the merging of data more reasonable. Considering the data merging requirements, the storage identifier can be concretized as a partition identifier, a block identifier and the number of merges.

[0031] Accordingly, to ensure data consistency, after the data is merged, the shared device monitors the merge information in the log server and synchronizes the data to be merged into the merged data according to the merge information in the log server. Specifically, the write storage identifier of the old data to be merged is deleted, and the merge storage identifier of the new merged data is stored.

[0032] Embodiment 2

[0033] Figure 2a This is a system architecture diagram of a data reading and writing system provided by Embodiment 2 of the present invention. This embodiment provides a preferred embodiment based on the above embodiments.

[0034] like Figure 2a As shown, the bottom layer of the data reading and writing system is the shared storage layer, and all data is stored in the shared storage. One shard in the shared storage corresponds to multiple copies, supporting multiple master writes and reads without conflict. As can be seen from Figure 2, the data reading and writing system in this embodiment supports multiple master writes, and multiple master writes can be executed simultaneously without conflict. Its data storage unit is part, that is, data reading and writing are exchanged through parts. Get_part log is used to synchronize the writing of shared storage message notifications between master devices, and notify other master devices to load part data.

[0035] The data reading and writing system provided in this embodiment performs calculations through the upper-layer multi-shard multi-master system Master, and writes to the shared storage system through the replica device, adopting a structure in which calculation and storage are separated. Among them, the multi-shard multi-host all include independent database instances. Correspondingly, the relationship between multiple hosts can be regarded as the relationship between multi-master multi-shard database instances. Among them, multi-master means that the same shard host transmits data to other masters. The host can transmit data to multiple other hosts at the same time or one by one. Multi-master means that each host has its own copy or multiple copies, which can be regarded as multiple independent shard multi-master structures.

[0036] Optionally, the write storage identifier can be uniquely determined based on the partition identifier, the minimum block identifier, the maximum block identifier, and the number of merges. For example, the partition identifier + the minimum block identifier + the maximum block identifier + the number of merges is used as the write storage identifier. Each write operation will generate a new write storage identifier. The above mechanism can realize concurrent read and write control, and each read operation can obtain consistent part data, realize non-blocking read and write, and concurrent write operations on the same shard.

[0037] In the above process, since the write storage identifier (i.e., part ID) under the same partition shard needs to be unique, the block identifier under the same shard needs to be unique. This embodiment generates a unique block identifier by self-incrementing. Figure 2b This is a schematic diagram of generating a block identifier provided by the second embodiment of the present invention. Figure 2b As shown, the writing device applies for a block identifier (Black ID), generates a self-incrementing identifier based on the applied block identifier, and uses the generated self-incrementing identifier as the block identifier.

[0038] When writing data, the control device (master) divides the data to be written into multiple sub-data according to the number of shards N and the corresponding strategy, and then selects any replica device corresponding to the shard as the writing device, and uses other replica devices corresponding to the shard as shared devices, and sends the data to be written to the writing device (corresponding to Figure 2a ①send part data), after the writing device writes the data to be written into the underlying shared storage, it uploads the written information to the log server (corresponding to Figure 2a ②send get part log), the shared device monitors the written information in the log server (corresponding to Figure 2a ③ receiveget part log), load the data to be written from the shared storage according to the write information in the log server (corresponding to Figure 2a ④load part data).

[0039] Figure 2c FIG. 1 is a flow chart of a data writing method provided by Embodiment 2 of the present invention. Figure 2c The data writing process shown can be:

[0040] 1) Receive data;

[0041] 2) Generate 1, 2, ..., N pieces of data according to the number of shards N and the corresponding strategy;

[0042] 3) The machine receiving the data randomly selects a copy under the shard and sends the corresponding sharded data (i.e. Figure 2a ①);

[0043] 4) The replica receives the data, generates a partID, writes the corresponding data to the underlying shared storage, and sends the GET_PARTlog (i.e. write information) under the shard. Figure 2a ②) to the log server. Otherwise, it returns failure;

[0044] 5) The same shard copy gets GET_PART LOG from the log server (i.e. Figure 2a ③) in the middle, obtain data from the underlying shared storage according to the acquisition log (i.e. Figure 2a ④), load data;

[0045] 6) Return success.

[0046] Since each data write generates a part, the number of parts increases accordingly when the number of write operations increases, and the database performance decreases. Therefore, based on the above, small parts can be merged. Figure 2d FIG. 1 is a flow chart of a data merging method provided by Embodiment 2 of the present invention. Figure 2d As shown, the data writing process can be:

[0047] 1) Initiate a merge operation;

[0048] 2) Select the parts to be merged based on the strategy;

[0049] Just like the IDs corresponding to the shard parts, partition-0-0, partition-1-1, partition-2-2, ..., etc., according to the strategy, only continuous block IDs can be merged. According to the conditions, the parts of partition-0-0, partition-1-1, and partition-2-2 are selected for merging;

[0050] 3) Generate a new part ID based on the selected part;

[0051] For example, the new part ID may be partition-0-2-1;

[0052] 4) Merge and select part data, generate a new part submission, modify the part status to old, and delete the part that has been merged and has the status of old;

[0053] 5) After the merge is complete, GET_MERGE LOG (merge information) is sent to the log server. The log contains the new partID and the old partID. Success or failure is returned;

[0054] 6) The replica of the same shard listens to GET_MERGE LOG, loads the new part from the underlying shared storage, and deletes the old part from the memory.

[0055] In this embodiment, the new part ID is partition-0-2-1, and the old part IDs are partition-0-0, partition-1-1, and partition-2-2.

[0056] The data reading and writing system provided by the embodiment of the present invention realizes simultaneous writing operations of multiple shards and multiple masters and simultaneous writing operations of different master devices, and does not require data transmission between master devices, thereby ensuring data consistency and avoiding data transmission between replicas, decoupling computing and storage, and improving the fault tolerance of the system.

[0057] Embodiment 3

[0058] Figure 3 1 is a flow chart of a data writing method provided by Embodiment 3 of the present invention. The method provided by this embodiment is executed by a control device in a data reading and writing system provided by any embodiment of the present invention. Figure 3 As shown, the method includes:

[0059] S310. In response to a detected data write request, determine the write device and the shared device associated with the data write request.

[0060] In this embodiment, the control device will perform a computing operation of data writing, determine a writing device and a shared device corresponding to the data writing request, and perform the data writing operation through the writing device.

[0061] In one embodiment of the present invention, the determination of the write device and the shared device associated with the data write request includes: determining the write slice of the data to be written in the data write request, and determining the write device and the shared device based on the replica device associated with the write slice. Optionally, the method for determining the write device and the shared device can refer to the above embodiment, which will not be repeated here.

[0062] S320, sending the data to be written to the writing device, so that the writing device writes the data to be written into the shared storage, and uploads the writing information to the log server, and the shared device performs data synchronization according to the writing information in the log server.

[0063] The embodiment of the present invention determines the writing device and the shared device associated with the data writing request in response to a detected data writing request; sends the data to be written to the writing device so that the writing device writes the data to be written into the shared storage and uploads the writing information to the log server, and the shared device performs data synchronization according to the writing information in the log server, thereby separating computing from storage, allowing multiple data writing operations to be executed simultaneously, avoiding data transmission between devices, and improving the fault tolerance of the system.

[0064] Embodiment 4

[0065] Figure 4 1 is a schematic diagram of the structure of a data writing device provided by the fourth embodiment of the present invention. The data writing device can be implemented in software and / or hardware, for example, the data writing device can be configured in a computer device. Figure 4 As shown, the apparatus includes a device determination module 410 and a data forwarding module 420, wherein:

[0066] The device determination module 410 is used to determine the writing device and the shared device associated with the data writing request in response to the detected data writing request;

[0067] The data forwarding module 420 is used to send the data to be written to the writing device, so that the writing device writes the data to be written into the shared storage and uploads the writing information to the log server, and the shared device synchronizes data according to the writing information in the log server.

[0068] The embodiment of the present invention determines the writing device and the shared device associated with the data writing request in response to the detected data writing request through a device determination module; the data forwarding module sends the data to be written to the writing device, so that the writing device writes the data to be written into the shared storage and uploads the writing information to the log server, and the shared device synchronizes data according to the writing information in the log server, thereby realizing the separation of computing and storage, allowing multiple data writing operations to be executed simultaneously, avoiding data transmission between devices, and improving the fault tolerance of the system.

[0069] Optionally, based on the above solution, the device determination module 410 is specifically used to:

[0070] A write slice of data to be written in a data write request is determined, and the write device and the shared device are determined based on a replica device associated with the write slice.

[0071] The data writing device provided in the embodiment of the present invention can execute the data writing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0072] Embodiment 5

[0073] Figure 5 It is a structural diagram of a computer device provided in Embodiment 5 of the present invention. Figure 5 A block diagram of an exemplary computer device 512 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The computer device 512 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0074] like Figure 5 As shown, the computer device 512 is in the form of a general-purpose computing device. The components of the computer device 512 may include, but are not limited to: one or more processors 516, a system memory 528, and a bus 518 that connects various system components (including the system memory 528 and the processor 516).

[0075] Bus 518 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a local bus to processor 516, or a bus structure using any of a variety of bus architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0076] The computer device 512 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 512, including volatile and non-volatile media, removable and non-removable media.

[0077] System memory 528 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 530 and / or cache memory 532. Computer device 512 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage device 534 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 518 via one or more data medium interfaces. The memory 528 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0078] A program / utility 540 having a set (at least one) of program modules 542 may be stored, for example, in the memory 528, such program modules 542 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 542 generally perform the functions and / or methods of the embodiments described herein.

[0079] The computer device 512 may also communicate with one or more external devices 514 (e.g., keyboard, pointing device, display 524, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 512, and / or communicate with any device that enables the computer device 512 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 522. In addition, the computer device 512 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via a network adapter 520. As shown, the network adapter 520 communicates with other modules of the computer device 512 via a bus 518. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 512, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0080] The processor 516 executes various functional applications and data processing by running the program stored in the system memory 528, for example, implementing the data writing method provided in the embodiment of the present invention, which includes:

[0081] In response to the detected data write request, determining the write device and the shared device associated with the data write request;

[0082] The data to be written is sent to the writing device, so that the writing device writes the data to be written into the shared storage, and uploads the writing information to the log server, and the shared device performs data synchronization according to the writing information in the log server.

[0083] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data writing method provided by any embodiment of the present invention.

[0084] Embodiment 6

[0085] Embodiment 6 of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the data writing method provided in the embodiment of the present invention is implemented. The method includes:

[0086] In response to the detected data write request, determining the write device and the shared device associated with the data write request;

[0087] The data to be written is sent to the writing device, so that the writing device writes the data to be written into the shared storage, and uploads the writing information to the log server, and the shared device performs data synchronization according to the writing information in the log server.

[0088] Of course, the computer program stored on a computer-readable storage medium provided by an embodiment of the present invention is not limited to the above method operations, but can also execute related operations of the data writing method provided by any embodiment of the present invention.

[0089] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0090] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0091] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0092] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0093] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data reading and writing system, characterized in that: include: Control devices, write devices, shared devices, shared storage and log servers, including: The control device is used to determine the writing device and the shared device associated with the data writing request in response to the detected data writing request, and send the data to be written to the writing device; The writing device is used to write the data to be written into the shared storage and upload the writing information to the log server; The shared device is used to obtain the write information in the log server and synchronize the data to be written according to the write information; Wherein, determining the write device and the shared device associated with the data write request includes: determining the write slice of the data to be written in the data write request, and determining the write device and the shared device based on the replica device associated with the write slice.

2. The system according to claim 1, characterized in that The determining the write device and the shared device based on the replica device associated with the write shard includes: Acquire multiple replica devices associated with the write slice, select one of the replica devices as the write device, and use the replica devices other than the write device as the shared devices.

3. The system according to claim 1, characterized in that The step of writing the data to be written into the shared storage and uploading the writing information to the log server includes: The data to be written is written into the shared storage, a write storage identifier corresponding to the data to be written is generated based on the data write path, and the write information is generated based on the write storage identifier and uploaded to the log server.

4. The system according to claim 3, characterized in that The write storage identifier includes a partition identifier corresponding to the data to be written and a block identifier of the data to be written.

5. The system according to claim 4, characterized in that The writing device is also used for: Determining the data to be merged according to the storage identifier; Merging the data to be merged to obtain merged data, and determining a merge storage identifier corresponding to the merged data; Upload the merged information to the log server.

6. The system according to claim 5, characterized in that The determining the data to be merged according to the storage identifier includes: The data corresponding to the storage identifiers having continuous block identifiers and the same partition identifier are used as the data to be merged.

7. The system according to claim 5, characterized in that The shared device is also used for: The merge information in the log server is obtained, and the merge data is synchronized according to the merge information.

8. A data writing method, characterized in that: include: In response to the detected data write request, determining a write device and a shared device associated with the data write request; Sending the data to be written to the writing device, so that the writing device writes the data to be written into the shared storage, and uploads the writing information to the log server, and the shared device performs data synchronization according to the writing information in the log server; Wherein, the determining the writing device and the shared device associated with the data writing request includes: A write slice of data to be written in a data write request is determined, and the write device and the shared device are determined based on a replica device associated with the write slice.

Citation Information

Patent Citations

  • Method and device for synchronizing data consistency

    CN107426265A

  • Method and device of synchronizing resource region of file sharing system after node fault

    CN108959390A

  • Database processing method, device and system

    CN110019066A

  • Data multiplexing method and device, electronic equipment and storage medium

    CN110413568A

  • Shared data synchronization method and device

    CN110895545A