Data reading method and device, equipment, storage medium and program product
By storing the copied data and newly written data into different containers and directly querying the data in the corresponding container during reading, the problem of inefficient reading in incremental replication is solved, and more efficient data reading is achieved.
Patent Information
- Application Number
- CN202311811275.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-26
- Publication Date
- 2025-06-27
AI Technical Summary
During the data replication process, the validity of the target data needs to be frequently judged during incremental replication, resulting in low reading efficiency.
The copied data and newly written data are stored in different containers, and the data is directly read without judging the validity of the data by querying the newly written data in the first container and querying the copied data in the second container.
The steps of reading data are simplified, the data reading efficiency is improved, and the reading performance of the target end remains optimal after copying and disconnection.
Smart Images
Figure CN120215804A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of data cloning and replication, and particularly to a method, apparatus, device, storage medium, and program product for reading data. Background Art
[0002] Replication is a technology that copies a set of data from a source end to one or more target ends. The newer replication technologies allow the data at the target end to be immediately available during the data replication process.
[0003] Currently, during the data replication process, it includes full - volume data replication and / or multiple rounds of incremental data replication. For incremental replication, when starting a new round of incremental replication each time, since the newly written data must be constructed based on the replicated data, the data previously written at the target end must all be "discarded". Therefore, when reading the data at the target end, it is necessary to pay attention to the validity of each version and read it in an orderly manner.
[0004] However, the above - mentioned method has the problem of low reading efficiency. Summary of the Invention
[0005] Based on this, in view of the above - mentioned technical problems, it is necessary to provide a method, apparatus, device, storage medium, and program product for reading data that can improve the reading efficiency.
[0006] In a first aspect, this application provides a method for reading data, including:
[0007] If a read request is received, query the target data corresponding to the read request in the first container; the first container includes the newly written data of the first target end;
[0008] If the target data does not exist in the first container, read the target data in the second container; the second container includes the data copied from the source end to the target end.
[0009] The above - mentioned method realizes a new method for reading the data at the target end. Compared with the existing method that needs to first determine the validity of the data at the target end and then read the valid data in the target end, the above - mentioned method stores the replicated data and the newly written data in different containers. When reading data based on this storage method, there is no need to perform the operation of judging the data validity, and directly read the data in the first container and / or the second container, which simplifies the steps of reading data and thus improves the data reading efficiency to a certain extent.
[0010] In one of the embodiments, the above - mentioned method further includes:
[0011] Copy new data from the source end to the second container;
[0012] Write the newly written data of the second target end into the third container;
[0013] Invalidate the newly written data at the first target end in the original first container, and use the third container as the new first container.
[0014] The data reading method provided by the embodiments of the present application, after copying new data from the source end, by updating the new first container and storing the newly copied data in the second container, realizes a method of reading data when copying new data, that is, without judging the validity of the data, directly read the data in container three or in both container three and container two, which improves the data reading efficiency to a certain extent.
[0015] In one embodiment, the above first container further includes a copy of the index of each data in the second container. If the target data does not exist in the first container, reading the target data in the second container includes:
[0016] If the target data does not exist in the first container, determine whether the target data exists in the second container according to the index of each data in the second container copied to the first container;
[0017] If the target data exists in the second container, read the target data according to the index of the target data.
[0018] The data reading method provided by the embodiments of the present application, based on copying the index of each data in the second container to the first container, when querying the target data, only need to query the index of each data in the second container in the first container, without having to query in the second container again, which improves the data query and reading efficiency to a certain extent. In addition, it ensures that the reading performance at the target end remains optimal after the replication is disconnected.
[0019] In one embodiment, the above method further includes:
[0020] During the process of copying new data from the source end to the second container, when a read request is received, query the target data corresponding to the read request from the first container and the second container. If the target data corresponding to the read request does not exist in the first container and the second container, read the target data corresponding to the read request from the source end.
[0021] The data reading method provided by the embodiments of the present application, based on forwarding the read request to the source end for processing, can receive the read request at the target end during the replication process, realizes that data can be read and newly written data can be read at the target end while the data is being copied, thus realizing the immediate availability of the replication target end.
[0022] In one embodiment, if the above data includes multiple incremental data, reading the target data in the second container includes:
[0023] Determine whether there is target data in the second container;
[0024] If there is no target data in the second container, read the target data from the fourth container; the fourth container includes the second full amount of data copied from the source end to the target end.
[0025] The data reading method provided by the embodiments of the present application queries each incremental data in sequence within a single container, which improves the data reading efficiency to a certain extent.
[0026] In one embodiment, the above data includes the third full amount of data, and the method further includes:
[0027] If a read request is received, read the target data corresponding to the read request from the fifth container; the fifth container includes the third full amount of data and the newly written data at the third target end.
[0028] The data reading method provided by the embodiments of the present application queries and reads the newly written data at the target end and the full amount of copied data in sequence within a single container in the case where the replication process only includes full amount replication, which improves the data reading efficiency to a certain extent.
[0029] In a second aspect, the present application further provides a data reading device, including:
[0030] A query module, configured to query the target data corresponding to the read request in the first container if a read request is received; the first container includes the newly written data at the first target end;
[0031] A reading module, configured to read the target data in the second container if the target data does not exist in the first container; the second container includes the data copied from the source end to the target end.
[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0033] If a read request is received, query the target data corresponding to the read request in the first container; the first container includes the newly written data at the first target end;
[0034] If the target data does not exist in the first container, read the target data in the second container; the second container includes the data copied from the source end to the target end.
[0035] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0036] If a read request is received, query the target data corresponding to the read request in the first container; the first container includes newly written data of the first target end;
[0037] If the target data does not exist in the first container, read the target data in the second container; the second container includes data copied from the source end to the target end.
[0038] In a fifth aspect, the present application also provides a computer program product, including a computer program, which when executed by a processor implements the following steps:
[0039] If a read request is received, query the target data corresponding to the read request in the first container; the first container includes newly written data of the first target end;
[0040] If the target data does not exist in the first container, read the target data in the second container; the second container includes data copied from the source end to the target end.
[0041] The above data reading method, device, equipment, storage medium and program product, the above method realizes a new method for reading data at the target end. Compared with the existing method that needs to first determine the validity of the data at the target end and then read the valid data at the target end, the above method stores the copied data and the newly written data in different containers, so that when reading data based on this storage method, there is no need to perform data validity judgment operations, and directly read the data in the first container and / or the second container, which simplifies the steps of reading data, and thus improves the data reading efficiency to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0043] Figure 1 It is a schematic diagram of the principle of immediate availability of the target end in an embodiment;
[0044] Figure 2 It is a schematic diagram of the distribution of copied data at the target end in an embodiment;
[0045] Figure 3 It is an application environment diagram of the data reading method in an embodiment;
[0046] Figure 4 It is a flowchart of the data reading method in an embodiment;
[0047] Figure 5 Schematic flow chart of the data reading method in another embodiment;
[0048] Figure 6 Schematic distribution diagram of the target - side data placed separately in one embodiment;
[0049] Figure 7 Schematic flow chart of the data reading method in another embodiment;
[0050] Figure 8 Schematic diagram of the "index separation" process in one embodiment;
[0051] Figure 9 Schematic distribution diagram of the target - side after the "index separation" is completed in one embodiment;
[0052] Figure 10 Schematic flow chart of the data reading method in another embodiment;
[0053] Figure 11 Schematic distribution diagram of the target - side data placed separately in one embodiment;
[0054] Figure 12 Schematic distribution diagram of the target - side data in one embodiment;
[0055] Figure 13 Schematic flow chart of the data reading method in another embodiment;
[0056] Figure 14 Schematic block diagram of the data reading device in one embodiment. Detailed implementation manners
[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.
[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above - mentioned drawings are intended to cover non - exclusive inclusion.
[0059] In the description of the embodiments of the present application, technical terms such as "first" and "second" are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity, specific order or primary-secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality" is more than two, unless otherwise specifically defined.
[0060] Reference to "embodiment" in this text means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0061] Master-slave replication means copying the data of the master node (master / leader) to the slave node (slave / follower). Usually, in traditional replication, the data of the slave node cannot be used during the replication process. Only after the replication is completed can the data of the slave node be read and written. Newer replication technologies allow the data at the replication target end (i.e., the slave node in the above background) to be immediately available during full or incremental replication, that is, the target end can be read and written normally immediately after each round of replication starts. To achieve immediate availability, the replication target end usually needs to perform the following processing (as Figure 1 shown, a schematic diagram of reading and writing the data of the target end is provided):
[0062] (1) Distinguish the data copied from the replication source end (i.e., the master node in the above background) and the data newly written by the user to the target end (i.e., Figure 1 the newly written data in it), usually distinguished by version numbers;
[0063] (2) When the user reads the data of the target end, first read the data newly written by the user. If the data needed by the user does not exist in the newly written data, then check the copied data; if the latest round of replication has not been completed (i.e., Figure 1 the data being replicated in it), then read the data that has been copied in the latest replication round. If the data needed by the user still cannot be found in the data copied in the latest replication round, then read the data from the source end (i.e., Figure 1 the source data in it); if the latest round of replication has been completed, then only query the data of each round that has been copied at the target end. If the data needed by the user still cannot be queried in the data of each round that has been copied, it means that the data needed by the user does not exist.
[0064] For incremental replication, each time a new round of incremental replication is started, since the newly written data must be constructed based on the replicated data, the data previously written at the target end must be "discarded", as follows Figure 2 shown, a data distribution diagram of the target end after 1 round of full replication and 3 rounds of incremental replication is provided. In the figure, the newly written data 2 represents the data newly written at the target end after the start of the second round of incremental replication and before the start of the next round of incremental replication. Since the newly written data must be constructed based on the replicated data, only the newly written data in the last round is valid. After the latest round of replication is completed, when reading data from the replication target end, there is no need to access the source end. Just query the metadata sequentially from the high replication version to the low replication version at the target end (for example, Figure 2 when reading the data at the target end in [the example], the query order is as follows: newly written data 3, the incremental data of the third round, the incremental data of the second round, the incremental data of the first round, and the initial full data). However, during the query process, the newly written data of each discarded old version must be skipped. Therefore, when reading the data at the target end, it is necessary to pay attention to the validity of each version and read the valid data. However, the above method has the problem of low reading efficiency. This application aims to solve this problem. After introducing the background technology of the data reading method provided by the embodiments of this application above, below, the implementation environment related to the data reading method provided by the embodiments of this application will be briefly described.
[0065] The data reading method provided by the embodiments of this application can be applied to, for example Figure 3In the computer device shown. It should be noted that the computer device may be the replication destination in the above background introduction. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a cache management method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.
[0066] Those skilled in the art can understand that Figure 3 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.
[0067] After introducing the application scenario of the data reading method provided in the embodiments of this application above, the data reading method described in this application will be introduced in detail below.
[0068] In one embodiment, as Figure 4 shown, a data reading method is provided. Taking the method applied to the computer device (destination) in Figure 1 as an example, the method includes the following steps:
[0069] S201. If a read request is received, query the target data corresponding to the read request in the first container.
[0070] Among them, the read request can be a request for reading certain data triggered by the user at the target end, or it can also be a request for reading certain data automatically initiated by the target end within a preset time. The first container includes newly written data of the first target end. The newly written data of the first target end refers to the newly written data of the target end after the start of the latest round of replication. The first container can be implemented using a writable snapshot. For example, if the data copied by the target end from the source end is in sequence: full amount of data, first-round incremental data, second-round incremental data, and third-round incremental data, then what is stored in the first container is the newly written data of the target end after the start of the replication of the third-round incremental data. The target data corresponding to the read request refers to the data requested by the read request.
[0071] In the example of this application, when reading the data that needs to be read, a read request for the data can be first sent to the target end, or the target end can automatically send a read request for the data. And when the computer device receives the read request, first query the target data corresponding to the read request in the first container.
[0072] S202. If the target data does not exist in the first container, read the target data in the second container.
[0073] Among them, the second container includes the data copied from the source end to the target end. For example, the second container includes full amount of data, first-round incremental data, second-round incremental data, and third-round incremental data.
[0074] In the embodiment of this application, after querying the data stored in the first container and determining that the target data corresponding to the read request does not exist in the first container, the target data corresponding to the read request can be continuously queried in the second container. And when the target data exists in the second container, read the target data from the second container. And when the target data does not exist in the second container, it means that the target data corresponding to the read request does not exist. It should be noted that the order of reading the replicated data in the second container is the reverse order of the data copied by the target end from the source end. For example, if the order of the data copied by the target end from the source end is in sequence: full amount of data, first-round incremental data, second-round incremental data, and third-round incremental data, then the order of reading the replicated data in the second container is third-round incremental data, second-round incremental data, first-round incremental data, and full amount of data.
[0075] The data reading method provided by the embodiments of the present application queries for target data corresponding to a read request in a first container if a read request is received; the first container includes newly written data of a first target end; if the target data does not exist in the first container, the target data is read in a second container; the second container includes data copied from a source end to the target end. The above method realizes a new method for reading data at the target end. Compared with the existing method that needs to first determine the validity of the data at the target end and then read the valid data at the target end, the above method stores the copied data and the newly written data in different containers, so that when reading data based on this storage method, there is no need to perform a data validity judgment operation, and the data in the first container and / or the second container can be directly read, which simplifies the steps of reading data and thus improves the data reading efficiency to a certain extent.
[0076] In one embodiment, based on Figure 4 the embodiment shown, as Figure 5 shown, the above method includes:
[0077] S203. Copy new data from the source end to the second container.
[0078] Among them, the new data refers to the data at the source end except for the data in the above second container, and the new data can be a new round of incremental data. As Figure 6 shown, the new data can be Figure 2 the 3rd round of incremental data in
[0079] In the embodiments of the present application, after the above-mentioned copying of the latest data from the source end to the target end is completed, the target end can also copy new data from the source end to the second container at the target end. Continuing with the above example, the second container includes full amount data, the first round of incremental data, the second round of incremental data, the third round of incremental data, and the fourth round of data.
[0080] S204. Write the newly written data of the second target end into a third container.
[0081] Among them, the newly written data of the second target end refers to the newly written data constructed by the target end based on the newly copied data, and the newly written data can be saved in a writable snapshot object. As Figure 6 shown, after the new copied data (the 3rd round of incremental data) starts to be copied, the newly written data at the target end is the newly written data 3, the third container is obj5, and the newly written data in the third container in the figure is valid, while the newly written data stored in other containers (obj4, obj3, and obj2) is invalid.
[0082] In an embodiment of the present application, after obtaining the newly copied data from the source end, the target end can write the newly written data copied from the source end to the second target end and store the newly written data of the second target end in the third container.
[0083] S205. Invalidate the newly written data of the first target end in the original first container, and use the third container as the new first container.
[0084] Among them, the invalidation process means that at the beginning of a new round of replication, the newly written data corresponding to the previous replication data becomes invalid. For example, Figure 6 the newly written data 2 in the obj4 container, the newly written data 1 in the obj3 container, and the newly written data 0 in the obj2 container shown are all invalid data.
[0085] In an embodiment of the present application, after writing the newly written data of the second target end into the third container, the newly written data of the first target end stored in the first container can be invalidated, and the third container can be used as the new first container. When a read request is received again (for example, Figure 6 the entry object in can be a plug-in in the computer device, and the entry object can initiate a request to read data from the computer device), the target data corresponding to the read request can be queried in the new first container. If the target data does not exist in the new first container, the target data corresponding to the read request can be queried in the second container. If the target data exists in the second container, the target data can be read from the second container. If the target data does not exist in the second container, it means that the target data corresponding to the read request does not exist. It should be noted that the order of reading the replicated data in the second container is the reverse order of the target end copying data from the source end. For example, if the order of the target end copying data from the source end is: full amount of data, first round of incremental data, second round of incremental data, third round of incremental data, and fourth incremental data, then the order of reading the replicated data in the second container is fourth incremental data, third round of incremental data, second round of incremental data, first round of incremental data, and full amount of data.
[0086] The data reading method provided by the embodiment of the present application, after copying new data from the source end, realizes a method of reading data when copying new data by updating the new first container and storing the new data in the second container, that is, without judging the validity of the data, directly reading the data in the third container or the data in the third container and the second container, which improves the data reading efficiency to a certain extent.
[0087] In one embodiment, based on Figure 5 the embodiment shown, the process of reading the target data in the second container can be described, as Figure 7As shown in the figure, the above S202 "if the target data does not exist in the first container, read the target data in the second container" includes:
[0088] S301. If the target data does not exist in the first container, determine whether the target data exists in the second container according to the indexes of the data in the second container copied to the first container.
[0089] Among them, the first container includes the newly written data at the target end after the start of the latest round of replication, and also includes a copy of the indexes of the data in the second container. As Figure 8 shown, a schematic diagram of copying the indexes of the replicated data in the second container obj1 into the first container obj5 is provided. The "index separation" version refers to saving a copy of the indexes of the dependent data in the second container on this version to distinguish them from the indexes of the newly written data at the target end in the first container.
[0090] In the embodiment of the present application, after a round of replication ends, the indexes of the data in the second container can be copied (copied) into the first container. At this time, the version of the newly written data in the first container is higher than the version of the index copies copied into the first container. That is to say, when a read request is received, first look for the target data corresponding to the read request in the newly written data in the first container. If the target data corresponding to the read request does not exist in the newly written data, continue to query the index of the target data according to the copies of the indexes of the data in the second container copied into the first container. If the index of the target data is found in the copies of the indexes of the data in the second container in the first container, it means that the target data exists in the second container. If the index of the target data is not found in the copies of the indexes of the data in the second container stored in the first container, it means that the target data does not exist in the second container. At this time, there is no need to continue querying the target data in the second container. As Figure 9 shown, a schematic diagram of the states of the second container obj1 and the first container obj5 after "index separation" is provided. Among them, the indexes of the replicated data in the second container obj1 have been completely copied to the "index separation" version in the first container obj5. When querying the target data for a read request, only need to query whether the target data exists in the first container obj5. If the index of the target data is found, it may be newly written data or data in the second container obj1. In either case, directly read the target data according to the index. If the index of the target data is not found in the first container obj5, it means that the target data does not exist in the second container obj1. At this time, there is no need to read the target data on the second container obj1 anymore. Further, the old indexes of the replicated data saved on the second container Obj1 become invalid if not referenced by other objects and will be gradually cleaned up in the background.
[0091] S302. If there is target data in the second container, read the target data according to the index of the target data.
[0092] In the embodiment of the present application, if the target data is found by querying the indexes of the data in the second container stored in the first container, it means that there is target data in the second container, and the target data can be directly read according to the index. Further, if the target data is not found by querying the indexes of the data in the second container stored in the first container, it means that there is no target data in the second container, and there is no need to query the target data in the second container.
[0093] The data reading method provided by the embodiment of the present application is based on storing the indexes of the dependent data in the second container in the first container. When querying the target data, only the data indexes need to be queried in the first container to determine whether there is target data in the second container, without having to query in the second container again, which improves the data query and reading efficiency to a certain extent. In addition, it ensures that the reading performance at the target end remains optimal after the replication is disconnected.
[0094] In one embodiment, on the basis of the embodiment shown in Figure 4 and Figure 5 the above method further includes:
[0095] During the process of copying new data from the source end to the second container, when a read request is received, query the target data corresponding to the read request in the first container and the second container. If the target data corresponding to the read request does not exist in the first container and the second container, read the target data corresponding to the read request from the source end.
[0096] In the embodiment of the present application, during the process of copying new data from the source end to the second container, when a read request is received, query the target data corresponding to the read request in the first container and the second container. If the target data corresponding to the read request does not exist in the first container and the second container, read the target data corresponding to the read request from the source end.
[0097] The data reading method provided by the embodiment of the present application is based on forwarding the read request to the source end for processing. During the replication process, the read request can be received at the target end, realizing that data can be read and newly written at the target end while the data is being replicated, thus realizing the immediate availability of the data at the target end.
[0098] In one embodiment, on the basis of the embodiment shown in Figure 4 and Figure 5 the process of reading the target data in the second container can be described. As shown in Figure 10 above, S202 "reading the target data in the second container" includes:
[0099] S501. Determine whether there is target data in the second container.
[0100] Among them, the second container includes multiple incremental data copied from the source end to the target end. As Figure 11 shown, a target - end data arrangement mainly based on full - volume copying is provided. Among them, multiple incremental data copied from the source end to the target end (that is, Figure 11 the third - round incremental data, the second - round incremental data, and the first - round incremental data in are stored in the second container obj2, and the newly written data corresponding to the third - round incremental data is stored in the container obj5, the newly written data corresponding to the second - round incremental data is stored in the container obj4, and the newly written data corresponding to the first - round incremental data is stored in the container obj3.
[0101] In the embodiments of the present application, for the data including multiple incremental data. In the case where it is determined that there is no target data in the first container, query the target data in each incremental data in the second container in sequence, and determine whether there is target data in the second container. For example, the order in which the target end copies data from the source end is: the first - round incremental data, the second - round incremental data, the third - round incremental data, and the fourth incremental data. Then the order of querying the target data in the second container in sequence is the fourth incremental data, the third incremental data, the second incremental data, and the first incremental data.
[0102] S502. If there is no target data in the second container, read the target data in the fourth container.
[0103] Among them, the fourth container includes the second full - volume data copied from the source end to the target end. As Figure 11 shown, the initial full - volume data copied from the source end to the target end and the newly written data corresponding to the full - volume data are stored in the fourth container obj1.
[0104] In the embodiments of the present application, if the target data is not found in any of the incremental data in the second container, continue to query the target data in the fourth container. When the target data is found in the fourth container, read the target data in the fourth container. When the target data is not found in the fourth container, it means that there is no target data in the fourth container.
[0105] The data reading method provided by the embodiments of the present application queries each incremental data and full - volume data in sequence, and reads after the target data is found, realizing the sequential query of multiple incremental data in a single container, and improving the data reading efficiency to a certain extent.
[0106] In one embodiment, on the basis of the embodiments shown in Figure 4 and Figure 5 as shown in the figure, the above - mentioned method further includes:
[0107] If a read request is received, the target data corresponding to the read request is read from the fifth container.
[0108] Among them, the fifth container includes the third full amount of data and the newly written data of the third target end. The newly written data of the third target end refers to the newly written data constructed by the target end based on the newly copied third full amount of data.
[0109] In the example of this application, if the entire replication process only includes full amount replication, the third full amount of data and the newly written data of the third target end are stored in the fifth container. When the computer device receives a read request, the target data corresponding to the read request is queried from the third full amount of data and the newly written data of the third target end stored in the fifth container, and read after the target data corresponding to the read request is queried. If the target data corresponding to the read request is not queried in the fifth container, it means that there is no target data corresponding to the read request. As Figure 12 shown, a scenario of only performing full amount replication is given, that is, the fifth container obj1 includes the third full amount of data (that is, Figure 12 the initial full amount of data in Figure 12 and the newly written data 0 of the third target end written in
[0110] The data reading method provided by the embodiments of this application, in the case where the replication process only includes full amount replication, only needs to query and read the newly written data corresponding to the full amount replication and the full amount replication data in a single container in sequence, which improves the data reading efficiency to a certain extent.
[0111] In one embodiment, on the basis of the embodiment shown in Figure 4 or Figure 5 shown, the process of reading the target data in the second container can be described. As Figure 13 shown, the above S202 "read the target data in the second container" includes:
[0112] S401. Query the target data in each incremental data in the second container in sequence according to the replication time order. If it is queried, go to S402; if it is not queried, go to S403.
[0113] Among them, the replication time order refers to the replication order of each data copied from the source end to the target end. For example, the order in which the target end copies data from the source end is: full amount of data, first round of incremental data, second round of incremental data, third round of incremental data, and fourth incremental data. The second container includes the first full amount of data and multiple incremental data copied from the source end to the target end.
[0114] In the embodiment of the present application, in the case where it is determined that the target data does not exist in the first container, the target data is sequentially queried in each incremental data in the second container, and it is determined whether the target data exists in the second container. For example, the order in which the target end copies data from the source end is: full amount data, first round incremental data, second round incremental data, third round incremental data, and fourth incremental data. Then the order of sequentially querying the target data in the second container is fourth incremental data, third incremental data, second incremental data, and first incremental data.
[0115] S402. Read the target data from the queried incremental data.
[0116] In the embodiment of the present application, if the target data is queried in any incremental data in the second container, the target metadata is directly read.
[0117] S403. Read the target data from the full amount data.
[0118] In the embodiment of the present application, if the target data is not queried in all the incremental data in the second container, continue to query the target data in the full amount data stored in the second container. And in the case where the target data is queried in the full amount data, directly read the target data from the full amount data; if the target data is not queried in the full amount data, it means that the target data does not exist in the second container.
[0119] The data reading method provided by the embodiment of the present application reads the target data in each incremental data and full amount data in sequence based on the replication time sequence, realizes the ordered reading of data, and ensures the data correctness.
[0120] It should be understood that although each step in the flowcharts involved in the above-described embodiments is displayed sequentially according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.
[0121] Based on the same inventive concept, an embodiment of the present application further provides a data reading device for implementing the data reading method involved above. The solution provided by this device for solving problems is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the data reading device provided below can refer to the limitations on the data reading method in the above text, and will not be repeated here.
[0122] In an exemplary embodiment, as Figure 14 shown, a data reading device is provided, including: a query module 10 and a reading module 11, where:
[0123] The query module 10 is configured to query, in a first container, target data corresponding to a read request if a read request is received; the first container includes newly written data of a first target end.
[0124] The reading module 11 is configured to read target data in a second container if the target data does not exist in the first container; the second container includes data copied from a source end to a target end.
[0125] In an exemplary embodiment, the above device further includes: a copying module, a writing module, and a defining module, where:
[0126] The copying module is configured to copy new data from a source end to the second container;
[0127] The writing module is configured to write newly written data of a second target end into a third container;
[0128] The defining module is configured to invalidate the newly written data of the first target end in the original first container and use the third container as the new first container.
[0129] In an exemplary embodiment, the above first container further includes indexes of each data in the second container. The reading module 11 includes: a first determining unit and a first reading unit, where:
[0130] The first determining unit is specifically configured to determine whether the target data exists in the second container according to the indexes of each data in the second container copied into the first container if the target data does not exist in the first container;
[0131] The first reading unit is specifically configured to read the target data according to the index of the target data if the target data exists in the second container.
[0132] In an exemplary embodiment, the above device further includes:
[0133] A reading module, which is used to query the target data corresponding to a read request from the first container and the second container when a read request is received during the process of copying new data from the source end to the second container. If the target data corresponding to the read request does not exist in the first container and the second container, the target data corresponding to the read request is read from the source end.
[0134] In an exemplary embodiment, the above data includes a plurality of incremental data. The above reading module 11 further includes: a second determination unit and a fourth reading unit, where:
[0135] The second determination unit is specifically configured to determine whether target data exists in the second container;
[0136] The fourth reading unit is specifically configured to, if the target data does not exist in the second container, read the target data in the fourth container; the fourth container includes the second full amount of data copied from the source end to the target end.
[0137] In an exemplary embodiment, the above data includes the third full amount of data. The above device further includes: a reading module, which is used to, if a read request is received, read the target data corresponding to the read request in the fifth container; the fifth container includes the third full amount of data and the newly added data at the third target end.
[0138] Each module in the above data reading device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above respective modules.
[0139] In an exemplary embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 3 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store replication data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a data reading method.
[0140] Those skilled in the art can understand that Figure 3 the structure shown in Figure 3 is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component arrangement.
[0141] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:
[0142] If a read request is received, query the target data corresponding to the read request in the first container; the first container includes newly written data at the first target end;
[0143] If the target data does not exist in the first container, read the target data in the second container; the second container includes data copied from the source end to the target end.
[0144] In one embodiment, when the processor executes the computer program, the following steps are further implemented:
[0145] Copy new data from the source end to the second container;
[0146] Write the newly written data at the second target end into the third container;
[0147] Invalidate the newly written data at the first target end in the original first container, and use the third container as the new first container.
[0148] In one embodiment, when the processor executes the computer program, the following steps are further implemented:
[0149] If the target data does not exist in the first container, determine whether the target data exists in the second container according to the indexes of the data in the second container copied to the first container;
[0150] If the target data exists in the second container, read the target data according to the index of the target data.
[0151] In one embodiment, when the processor executes the computer program, the following steps are further implemented:
[0152] During the process of copying new data from the source end to the second container, when a read request is received, query the target data corresponding to the read request in the first container and the second container. If the target data corresponding to the read request does not exist in the first container and the second container, read the target data corresponding to the read request from the source end.
[0153] In one embodiment, when the processor executes the computer program, the following steps are further implemented:
[0154] Determine whether there is target data in the second container;
[0155] If there is no target data in the second container, read the target data from the fourth container; the fourth container includes the second full amount of data copied from the source end to the target end.
[0156] In one embodiment, when the processor executes the computer program, the following steps are further implemented:
[0157] If a read request is received, read the target data corresponding to the read request from the fifth container; the fifth container includes the third full amount of data and the third target end write data.
[0158] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:
[0159] If a read request is received, query the target data corresponding to the read request in the first container; the first container includes the first target end newly written data;
[0160] If the target data does not exist in the first container, read the target data from the second container; the second container includes the data copied from the source end to the target end.
[0161] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:
[0162] Copy new data from the source end to the second container;
[0163] Write the second target end newly written data into the third container;
[0164] Invalidate the first target end newly written data in the original first container, and use the third container as the new first container.
[0165] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:
[0166] If the target data does not exist in the first container, determine whether there is target data in the second container according to the indexes of the data in the second container copied into the first container;
[0167] If there is target data in the second container, read the target data according to the index of the target data.
[0168] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:
[0169] During the process of copying new data from the source end to the second container, when a read request is received, query the target data corresponding to the read request from the first container and the second container. If the target data corresponding to the read request does not exist in the first container and the second container, read the target data corresponding to the read request from the source end.
[0170] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0171] Determine whether the target data exists in the second container;
[0172] If the target data does not exist in the second container, read the target data in the fourth container; the fourth container includes the second full amount of data copied from the source end to the target end.
[0173] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0174] If a read request is received, read the target data corresponding to the read request in the fifth container; the fifth container includes the third full amount of data and the third target end write data.
[0175] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0176] If a read request is received, query the target data corresponding to the read request in the first container; the first container includes the first target end newly written data;
[0177] If the target data does not exist in the first container, read the target data in the second container; the second container includes the data copied from the source end to the target end.
[0178] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0179] Copy new data from the source end to the second container;
[0180] Write the second target end newly written data into the third container;
[0181] Invalidate the first target end newly written data in the original first container, and use the third container as the new first container.
[0182] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0183] If the target data does not exist in the first container, determine whether the target data exists in the second container according to the indexes of the data in the second container copied into the first container;
[0184] If there is target data in the second container, read the target data according to the index of the target data.
[0185] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0186] During the process of copying new data from the source end to the second container, when a read request is received, query the target data corresponding to the read request from the first container and the second container. If the target data corresponding to the read request does not exist in the first container and the second container, read the target data corresponding to the read request from the source end.
[0187] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0188] Determine whether there is target data in the second container;
[0189] If there is no target data in the second container, read the target data in the fourth container; the fourth container includes the second full amount of data copied from the source end to the target end.
[0190] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:
[0191] If a read request is received, read the target data corresponding to the read request in the fifth container; the fifth container includes the third full amount of data and the third newly written data at the target end.
[0192] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memories can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.
[0193] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.
[0194] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.
Claims
1. A method for reading data, characterized in that, The method includes: If a read request is received, query for the target data corresponding to the read request in a first container; the first container includes newly written data of a first target end. If the target data does not exist in the first container, read the target data in a second container; the second container includes data copied from a source end to the target end.
2. The method according to claim 1, wherein The method further includes: Copy new data from the source end to the second container; Write newly written data of a second target end into a third container; Invalidate the newly written data of the first target end in the original first container, and use the third container as the new first container.
3. The method according to claim 1 or 2, characterized in that, The first container further includes indexes of each data in the second container, and if the target data does not exist in the first container, reading the target data in the second container includes: If the target data does not exist in the first container, determine whether the target data exists in the second container according to the indexes of each data in the second container copied into the first container; If the target data exists in the second container, read the target data according to the index of the target data.
4. The method according to claim 1 or 2, characterized in that, The method further includes: During the process of copying new data from the source end to the second container, when a read request is received, query for the target data corresponding to the read request in the first container and the second container. If the target data corresponding to the read request does not exist in the first container and the second container, read the target data corresponding to the read request from the source end.
5. The method according to claim 1 or 2, characterized in that, If the data includes a plurality of incremental data, reading the target data in the second container includes: Determine whether the target data exists in the second container; If the target data does not exist in the second container, read the target data in a fourth container; the fourth container includes second full amount data copied from the source end to the target end.
6. The method according to claim 1 or 2, characterized in that, If the data includes third full amount data, the method further includes: If a read request is received, read the target data corresponding to the read request in a fifth container; the fifth container includes the third full amount data and newly written data of a third target end.
7. A data reading device, characterized in that, The device includes: A query module, configured to query for the target data corresponding to the read request in a first container if a read request is received; the first container includes newly written data of a first target end. A reading module, configured to read the target data in a second container if the target data does not exist in the first container; the second container includes data copied from a source end to the target end.
8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.