A data multiplexing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a data multiplexing method and device, electronic equipment and storage medium. The method comprises: storing a write request sent by a user through a synchronization log module; wherein the write request comprises: current write data and a number of the current write data; storing the current write data into a current data file of a storage engine module, and hard chaining a current shadow file corresponding to the current data file; establishing first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; and establishing second index information of the current write data in the storage engine module after the current write data is submitted. Embodiments of the present application can multiplex one piece of data of the synchronization log module and the storage engine module, reduce the disk write pressure under the same per-second write QPS, improve the write performance of the distributed storage system, and prolong the service life of the storage device.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer technology, and particularly relate to a data multiplexing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Distributed storage system is to store data on multiple independent devices. The traditional network storage system uses centralized storage server to store all data, and the storage server becomes the bottleneck of system performance and the focus of reliability and security, which cannot meet the needs of large-scale storage applications. Distributed network storage system uses scalable system structure, uses multiple storage servers to share storage load, and uses location server to locate storage information. It not only improves the reliability, availability and access efficiency of the system, but also is easy to expand.

[0003] In the distributed storage system, a synchronization log module and a storage engine module are usually involved. The synchronization log module is used to ensure the consistency of multiple copies of data, and the storage engine module provides persistent storage and read-write services for the copy data. At present, in most distributed storage systems, the synchronization log module and the storage engine module are stored separately. The write request needs to be sent to the synchronization log module and the storage engine module respectively, and the data needs to be written to the disk twice, which leads to the fact that the disk IO often becomes the bottleneck of the entire distributed storage system in the case of large data volume, and the continuous large disk writing pressure seriously affects the service life of the disk.

[0004] In order to solve the problem of twice disk IO, some products use the method of caching write data in memory, only keeping the latest fixed entries of write data. If the write data requested from the copy is not in the cache, a full synchronization will be triggered again. By caching write data in memory, the problem of twice disk IO for read-write request data can be solved, but it also brings more memory overhead. At the same time, if the master-slave copy network is partitioned, the slave copy machine is down, or other abnormal conditions occur, a full synchronization will be triggered to ensure the data consistency between the master and the slave, which brings additional network bandwidth overhead. SUMMARY

[0005] Therefore, embodiments of the present application provide a data multiplexing method, device, electronic equipment and storage medium, which can realize the multiplexing of one copy of data for the synchronization log module and the storage engine module, reduce the disk writing pressure under the same per-second write QPS, improve the write performance of the distributed storage system, and prolong the service life of the storage device.

[0006] In a first aspect, embodiments of the present application provide a data multiplexing method, which comprises:

[0007] store the write request sent by the user through the synchronization log module; wherein the write request comprises current write data and a number of the current write data;

[0008] store the current write data into a current data file of the storage engine module, and hard link a current shadow file corresponding to the current data file for the current data file;

[0009] establish first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; wherein the first index information is used to read the current write data according to the number of the current write data;

[0010] establish second index information of the current write data in the storage engine module after the current write data is committed; wherein the second index information is used to provide query service of the current write data.

[0011] In the above embodiment, the method further comprises:

[0012] determine whether each historical write data is valid write data according to version information of each historical write data predetermined in the storage engine module and a memory lookup table acquired in advance;

[0013] if each historical write data is valid write data, switch each historical write data and second index information of each historical write data from an original directory of the storage engine module to a target directory of the storage engine module.

[0014] In the above embodiment, the switching of each historical write data and second index information of each historical write data from the original directory of the storage engine module to the target directory of the storage engine module comprises:

[0015] if each historical write data is committed write data, store each historical write data and second index information of each historical write data in the target directory of the storage engine module, and delete each historical write data and second index information of each historical write data in the original directory of the storage engine module;

[0016] if each historical write data is uncommitted write data, hard link each historical data file in the original directory to the target directory of the storage engine module, and update the mapping relationship between each shadow file and each historical data file.

[0017] In the above embodiment, the method further comprises:

[0018] store the read request sent by the user through the synchronization log module; wherein the read request comprises the number of the current write data;

[0019] According to the number of the current write data, the first index information of the current write data is obtained in the synchronization log module;

[0020] According to the first index information of the current write data, the shadow file name and the corresponding offset and length of the current write data in the storage engine module are determined;

[0021] According to the shadow file name and the corresponding offset and length, the current write data is read in the storage engine module.

[0022] In the above embodiment, the first index information comprises at least one start number start_id; each start_id comprises at least one log file di_file; each di_file comprises at least one log entry di_entry and the log entry index index of all di_entry; each di_entry comprises a mark magic_num, a type type, a log entry data data, and a check value crc; the data comprises a request header and index information; the index information comprises a shadow file name, an offset, and a length; the second index information comprises at least one storage unit; each storage unit comprises an original directory and a target directory; the original directory and the target directory respectively comprise 0-64 index check indexcrc, 0-64 index index, and 0-63 data data; each data is correspondingly provided with a shadow file; the shadow file comprises a shadow file name filename, an offset offset, and a length len.

[0023] In a second aspect, an embodiment of the present application provides a data multiplexing device, which comprises a first storage module, a second storage module, a first establishing module, and a second establishing module; wherein,

[0024] The first storage module is configured to store a write request sent by a user through a synchronization log module; wherein the write request comprises current write data and the number of the current write data;

[0025] The second storage module is configured to store the current write data into a current data file of a storage engine module, and hard chain a current shadow file corresponding to the current data file for the current data file;

[0026] The first establishing module is configured to establish first index information of the current write data according to the number of the current write data and the current shadow file in the synchronization log module; wherein the first index information is used to read the current write data according to the number of the current write data.

[0027] The second establishing module is configured to establish second index information of the current write data in the storage engine module after the current write data is submitted; wherein the second index information is used to provide query service of the current write data.

[0028] In the above embodiment, the device further comprises a switching module configured to determine whether each historical write data is valid write data according to version information of each historical write data in the storage engine module and a pre-acquired memory lookup table; if each historical write data is valid write data, switch each historical write data and the second index information of each historical write data from an original directory of the storage engine module to a target directory of the storage engine module.

[0029] In the above embodiment, the switching module is specifically configured to, if each historical write data is submitted write data, store each historical write data and the second index information of each historical write data in the target directory of the storage engine module; delete each historical write data and the second index information of each historical write data in the original directory of the storage engine module; if each historical write data is unsubmitted write data, hard link each historical data file in the original directory in the target directory of the storage engine module, and update the mapping relationship between each shadow file and each historical data file.

[0030] In the above embodiment, the device further comprises a first reading module and a second reading module; wherein,

[0031] The first storage module is configured to store the read request sent by the user through the synchronization log module; wherein the read request comprises the number of the current write data.

[0032] The first reading module is configured to acquire the first index information of the current write data in the synchronization log module according to the number of the current write data; and determine the shadow file name of the current write data in the storage engine module and the corresponding offset and length according to the first index information of the current write data.

[0033] The second reading module is configured to read the current write data in the storage engine module according to the shadow file name and the corresponding offset and length.

[0034] In the above embodiments, the first index information comprises at least one start number start_id; each start_id comprises at least one log file di_file; each di_file comprises at least one log entry di_entry and a log entry index index of all di_entry; each di_entry comprises a mark magic_num, a type type, log entry data data, and a check value crc; the data data comprises a request header and index information; the index information comprises a shadow file name, an offset, and a length; the second index information comprises at least one storage unit; each storage unit comprises an original directory and a target directory; the original directory and the target directory respectively comprise 0-64 index check index crc, 0-64 index index, and 0-63 data data; each data data is correspondingly provided with a shadow file; and the shadow file comprises a shadow file name filename, an offset offset, and a length len.

[0035] In a third aspect, an electronic device is provided, comprising:

[0036] one or more processors;

[0037] a memory for storing one or more programs,

[0038] when the one or more programs are executed by the one or more processors, the one or more processors implement the data multiplexing method according to any of the embodiments of the present application.

[0039] In a fourth aspect, a storage medium is provided, and the storage medium has stored thereon a computer program, which, when executed by a processor, implements the data multiplexing method according to any of the embodiments of the present application.

[0040] This invention proposes a data reuse method, apparatus, electronic device, and storage medium. First, a synchronization log module stores write requests sent by the user. Then, the currently written data is stored in the current data file of the storage engine module, and a corresponding current shadow file is hard-linked to the current data file. Next, a first index of the currently written data is established in the synchronization log module based on the number of the currently written data and the current shadow file. After the currently written data is submitted, a second index is established in the storage engine module. In other words, in this invention, the synchronization log module and the storage engine module can reuse a single set of data; the synchronization log module stores the first index information for reading the currently written data based on its number; and the storage engine module stores the second index information for providing query services for the currently written data. In existing data reuse methods, most distributed storage systems store the synchronization log module and storage engine module separately. Write requests need to be sent to both modules separately, requiring two disk I / O operations, which severely impacts disk lifespan. Some products cache write data in memory, retaining only the latest fixed entries. If the write data requested from the replica is not in the cache, a full synchronization is triggered again, increasing memory overhead. Therefore, compared with existing technologies, the data reuse method, apparatus, electronic device, and storage medium proposed in this invention can enable the synchronization log module and storage engine module to reuse a single set of data, reducing disk write pressure at the same QPS, improving the write performance of the distributed storage system, and extending the lifespan of the storage device. Furthermore, the technical solution of this invention is simple and convenient to implement, easy to popularize, and has a wider range of applications. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the data reuse method provided in Embodiment 1 of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of the synchronization log module provided in Embodiment 1 of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of a log file provided in Embodiment 1 of the present invention;

[0044] Figure 4 This is a schematic diagram of the structure of the storage engine module provided in Embodiment 1 of the present invention;

[0045] Figure 5 This is a flowchart illustrating the data reuse method provided in Embodiment 2 of the present invention;

[0046] Figure 6(a) is a schematic diagram of the storage engine module provided in Embodiment 2 of the present invention before data switching;

[0047] Fig. 6(b) is a structural schematic diagram of the storage engine module provided by the second embodiment of the present application after data switching;

[0048] Figure 7 Fig. 1 is a first structural schematic diagram of the data multiplexing device provided by the third embodiment of the present application;

[0049] Figure 8 Fig. 2 is a second structural schematic diagram of the data multiplexing device provided by the third embodiment of the present application;

[0050] Figure 9 Fig. 3 is a structural schematic diagram of the electronic device provided by the fourth embodiment of the present application. DETAILED DESCRIPTION

[0051] The present application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the parts.

[0052] Embodiment 1

[0053] Figure 1 Fig. 1 is a flowchart of the data multiplexing method provided by the first embodiment of the present application, which can be executed by the data multiplexing device or the electronic device, which can be realized by software and / or hardware, and which can be integrated in any smart device with network communication function. As shown in Fig. 1, the data multiplexing method can include the following steps: Figure 1

[0054] S101, storing the write request sent by the user through the synchronization log module; wherein the write request includes the current write data and the number of the current write data.

[0055] ​In the embodiment of the present application, the electronic device can store the write request sent by the user through the synchronization log module; wherein the write request comprises: current write data and the number of the current write data. Specifically, the distributed storage system can comprise a plurality of electronic devices, respectively: electronic device 1, electronic device 2, … electronic device N; wherein N is a natural number greater than 1. Each electronic device can comprise: a synchronization log module and a storage engine module; for example, electronic device 1 can comprise: synchronization log module 1 and storage engine module 1; electronic device 2 can comprise: synchronization log module 2 and storage engine module 2; …; electronic device N can comprise: synchronization log module N and storage engine module N. In this step, the user can send a write request to the synchronization log module of any one of the electronic devices in the distributed storage system; the electronic device can store the write request sent by the user through the synchronization log storage module; wherein the write request comprises: current write data and the number of the current write data.

[0056] S102, store the current write data into the current data file of the storage engine module, and hard chain the current shadow file corresponding to the current data file.

[0057] In the embodiment of the present application, the electronic device can store the current write data into the current data file of the storage engine module, and hard chain the current shadow file corresponding to the current data file. Specifically, in the distributed storage system, the electronic device can establish first index information in the synchronization log module; wherein the first index information comprises at least one start_id; each start_id comprises at least one di_file; each di_file comprises at least one di_entry and the index of all di_entry; each di_entry comprises: magic, type, data, crc; data comprises: request header and index information; index information comprises: shadow file name, offset and length. The electronic device can also establish second index information in the storage engine module; wherein the second index information comprises: at least one storage unit; each storage unit comprises: original directory and target directory; the original directory and the target directory respectively comprise: 0-64 index crc; 0-64 index; 0-63 data; each data is provided with a shadow file.

[0058] Figure 2 The structure diagram of the synchronization log module provided by the first embodiment of the present application is shown in the figure. Figure 2As shown, the synchronization log module can include Y start IDs, respectively: start ID 1 (start_id_1), start ID 2 (start_id_2), …, start ID Y (start_id_Y); wherein Y is a natural number greater than or equal to 1. Start ID 1 includes X log files, respectively: log file 1 (di_file_1), log file 2 (di_file_2), log file 3 (di_file_3), …, log file X (di_file_X); wherein X is a natural number greater than or equal to 1. Start ID 2 includes X log files, respectively: log file 1 (di_file_1), log file 2 (di_file_2), log file 3 (di_file_3), …, log file X (di_file_X); wherein X is a natural number greater than or equal to 1. And so on. In addition, the synchronization log module can also include: a log index table and a file mapping; wherein the log index table is used to find the start ID corresponding to the user's write request; and the file mapping is used to save the correspondence between the file and the shadow file.

[0059] S103, establishing first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; wherein the first index information is used to read the current write data according to the number of the current write data.

[0060] In a specific embodiment of the present application, the electronic device can establish first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; wherein the first index information is used to read the current write data according to the number of the current write data.

[0061] Figure 3 A structure diagram of the log file provided by the first embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the log file includes a log index table and a file mapping. Figure 3As shown, one log file includes K log items and a log item index; the K log items are respectively: log item 1, log item 2, …, log item K; wherein K is a natural number greater than or equal to 1. Each log item includes: a flag, a type, log item data and a check value; the log item data includes: a request header and index information; the index information includes: a shadow file name, an offset and a length. In the synchronization log module, the storage spaces of the start number 1 (start_id_1), the start number 2 (start_id_2), …, the start number Y (start_id_Y) can be pre-set; the storage spaces of the log file 1 (di_file_1), the log file 2 (di_file_2), the log file 3 (di_file_3), …, the log file X (di_file_X) can also be pre-set; the storage spaces of the log item 1, the log item 2, …, the log item K can also be pre-set. In this step, the electronic device can determine the current start code, the current file and the current log item corresponding to the current write data according to the current write data number and the current shadow file name filename, the current offset offset and the current length len in the current shadow file, and therefore the electronic device can establish the first index information of the current write data in the synchronization log module according to the current start code, the current file and the current log item.

[0062] S104, after the current write data is submitted, the second index information of the current write data is established in the storage engine module; wherein the second index information is used to provide the query service of the current write data.

[0063] In the embodiment of the application, the electronic device can establish the second index information of the current write data in the storage engine module after the current write data is submitted; wherein the second index information is used to provide the query service of the current write data. Specifically, the current write data can be first stored to the storage engine module of one electronic device in the distributed storage system; at this time, the current write data is uncommitted write data; then the synchronization log module of the electronic device can synchronize the current write data to the storage engine module of other electronic devices; at this time, the current write data is committed write data. It should be noted that when the current write data is synchronized to the storage engine module of other electronic devices, the current write data can be regarded as committed write data; or when the current write data is synchronized to the storage engine modules of multiple electronic devices, the current write data can be regarded as committed write data. Here, the number of the storage engine modules of other electronic devices to which the current write data is synchronized is not limited.

[0064] Figure 4 The structure schematic diagram of the storage engine module provided by the embodiment one of the application is shown in FIG. 1. Figure 4As shown, the storage engine module can include P groups, respectively: group 1, group 2, … group P; wherein P is a natural number greater than or equal to 1. Each group includes two directories, respectively: the original directory and the target directory. Here, the first directory identifier upddir can point to the target directory, and the second directory identifier curdir can point to the original directory, which can be the same or different. If they are the same, it means that the storage engine module is not currently rewriting, and if they are different, it means that the storage engine module is currently rewriting, and the data and second index information in the storage engine module are being moved from the original directory to the target directory. The original directory and the target directory can respectively include: indexcrc.0-indexcrc.64; index.0-index.64; data.0-data.63; each data corresponds to a shadow file (shadow); each shadow file includes: filename, offset and len. All shadow files are combined into a shadow directory, which can include Q shadow files, respectively: shadow file 1, shadow file 2, … shadow file Q; wherein Q is a natural number greater than or equal to 1.

[0065] The data multiplexing method provided by the embodiment of the present application first stores the write request sent by the user through the synchronization log module; then stores the current write data into the current data file of the storage engine module, and hard chains the current shadow file corresponding to the current data file; then establishes the first index information of the current write data according to the number of the current write data and the current shadow file in the synchronization log module; and after the current write data is submitted, the second index information of the current write data is established in the storage engine module. That is, in the technical solution of the present application, the synchronization log module and the storage engine module can multiplex one piece of data; the synchronization log module saves the first index information, which is used to read the current write data according to the number of the current write data; and the storage engine module saves the second index information, which is used to provide query service for the current write data. In the existing data multiplexing method, most distributed storage systems separately store the synchronization log module and the storage engine module, the write request needs to be sent to the synchronization log module and the storage engine module respectively, and the data needs to be subjected to two disk IOs, which seriously affects the disk life; some products use the method of caching the write data in the memory, only the latest fixed entries of the write data are retained, and if the write data requested from the copy is not in the cache, a full synchronization is triggered again, resulting in increased memory overhead. Therefore, compared with the prior art, the data multiplexing method provided by the embodiment of the present application can multiplex one piece of data of the synchronization log module and the storage engine module, reduce the disk write pressure under the same QPS, improve the write performance of the distributed storage system, and prolong the service life of the storage device; and the technical solution of the embodiment of the present application is simple and convenient to implement, easy to popularize, and has a wider application range.

[0066] Embodiment Two

[0067] Figure 5 A flowchart of a data multiplexing method provided by Embodiment Two of the present application. As shown in the figure, the data multiplexing method can include the following steps: Figure 5

[0068] S501, storing, by a synchronization log module, a write request sent by a user; wherein the write request includes current write data and a number of the current write data.

[0069] In a specific embodiment of the present application, an electronic device can store a write request sent by a user through a synchronization log module; wherein the write request includes current write data and a number of the current write data. Specifically, a distributed storage system can include a plurality of electronic devices, respectively: electronic device 1, electronic device 2, … electronic device N; wherein N is a natural number greater than 1. Each electronic device can include a synchronization log module and a storage engine module; for example, electronic device 1 can include a synchronization log module 1 and a storage engine module 1; electronic device 2 can include a synchronization log module 2 and a storage engine module 2; …; electronic device N can include a synchronization log module N and a storage engine module N. In this step, a user can send a write request to the synchronization log module of any one of the electronic devices in the distributed storage system; the electronic device can store the write request sent by the user through the synchronization log storage module; wherein the write request includes current write data and a number of the current write data.

[0070] S502, storing the current write data into a current data file of a storage engine module, and hard chaining a current shadow file corresponding to the current data file for the current data file.

[0071] ​In the embodiments of the present application, the electronic device can store the current write data into the current data file of the storage engine module and hard chain the current shadow file corresponding to the current data file. Specifically, in the distributed storage system, the electronic device can establish first index information in the synchronization log module; wherein the first index information comprises at least one start_id; each start_id comprises at least one di_file; each di_file comprises at least one di_entry and index of all di_entries; each di_entry comprises: magic, type, data, crc; the data comprises: request header and index information; the index information comprises: shadow file name, offset and length. The electronic device can also establish second index information in the storage engine module; wherein the second index information comprises: at least one storage unit; each storage unit comprises: original directory and target directory; the original directory and the target directory respectively comprise: 0-64 index crc; 0-64 index; 0-63 data; each data is provided with a shadow file; the shadow file comprises: filename, offset and len.

[0072] S503, establishing first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; wherein the first index information is used to read the current write data according to the number of the current write data.

[0073] In the embodiments of the present application, the electronic device can establish first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; wherein the first index information is used to read the current write data according to the number of the current write data.

[0074] S504, establishing second index information of the current write data in the storage engine module after the current write data is submitted; wherein the second index information is used to provide query service of the current write data.

[0075] In the embodiments of the present application, the electronic device can establish second index information of the current write data in the storage engine module after the current write data is submitted; wherein the second index information is used to provide query service of the current write data. Specifically, the current write data can be first stored in the storage engine module of one electronic device in the distributed storage system; at this time, the current write data is uncommitted write data; then the synchronization log module of the electronic device can synchronize the current write data to the storage engine module of other electronic devices; at this time, the current write data is committed write data.

[0076] In the specific embodiments of the present application, since all the data in the storage engine module is implemented based on append write, when there is data update or data deletion, the historical data is not deleted but only new data is appended. In this way, if a key value Key is updated multiple times, only the data of the most recent time is valid, and the historical data becomes garbage data, and the disk needs to be cleaned to release disk space. The cleaning process is referred to as rewriting of the engine. The core idea of rewriting in the present application is to move the data in the storage engine module from the original directory to the target directory. In the moving process, whether each historical write data is valid write data can be judged according to the number of each historical write data in the storage engine module and the memory lookup table obtained in advance. If each historical write data is valid write data, each historical write data and the second index information of each historical write data are switched from the original directory of the storage engine module to the target directory of the storage engine module. If each historical write data is invalid write data, each historical write data and the second index information of each historical write data are directly deleted from the original directory of the storage engine module. Specifically, if each historical write data is valid write data, the electronic device can judge that each historical write data is committed write data. If each historical write data is committed write data, each historical write data and the second index information of each historical write data are stored in the target directory of the storage engine module. Each historical write data and the second index information of each historical write data are deleted from the original directory of the storage engine module. If each historical write data is uncommitted write data, each historical data file in the original directory is hard linked in the target directory of the storage engine module, and the mapping relationship between each shadow file and each historical data file is updated.

[0077] FIG. 6(a) is a structural schematic diagram of the storage engine module before data switching according to the second embodiment of the present application. As shown in FIG. 6(a), the storage engine module can include data 0 (data.0)-data 63 (data.63); index 0 (index.0)-index 64 (index.64). It is assumed that data 0 (data.0)-data 60 (data.60) are historical data before switching, data 61 (data.61) and data 62 (data.62) are newly written uncommitted data before switching; and data 63 (data.63) is newly written committed data before switching. Index 0 (index.0)-index 63 (index.63) are indexes of the historical data before switching; and index 64 (index.64) is an index of the newly written data before switching. Data 61 (data.61) is hard linked to shadow file 1; and data 62 (data.62) is hard linked to shadow file 0.

[0078] Figure 6(b) is a structure diagram of the storage engine module after data switching according to the second embodiment of the present application. As shown in Figure 6(b), the storage engine module can include: data 0 (data.0)-data 63 (data.63); index 0 (index.0)-index 65 (index.65). Assuming that data 0 (data.0)-data 60 (data.60) are the historical data after switching, data 61 (data.61) is the newly written data after switching; data 62 (data.62) and data 63 (data.63) are the data hard linked after switching; index 0 (index.0)-index 63 (index.63) are the indexes of the historical data after switching; index 64 (index.64) is the index of the data hard linked after switching; index 65 (index.65) is the index of the newly written data after switching. Data 62 (data.62) is hard linked to shadow file 1; data 63 (data.63) is hard linked to shadow file 0.

[0079] S505, storing the read request sent by the user through the synchronization log module; wherein the read request includes the number of the current write data.

[0080] In the specific embodiments of the present application, the electronic device can store the read request sent by the user through the synchronization log module; wherein the read request includes the number of the current write data.

[0081] S506, obtaining the first index information of the current write data in the synchronization log module according to the number of the current write data.

[0082] In the specific embodiments of the present application, the electronic device can obtain the first index information of the current write data in the synchronization log module according to the number of the current write data. Specifically, the electronic device can determine the current start code, the current file and the current log item corresponding to the current write data in the synchronization log module; and then determine the shadow file name and the corresponding offset corresponding to the current write data in the current log item.

[0083] S507, determining the shadow file name and the corresponding offset of the current write data in the storage engine module according to the first index information of the current write data.

[0084] In the specific embodiments of the present application, the electronic device can determine the shadow file name and the corresponding offset of the current write data in the storage engine module according to the first index information of the current write data. Specifically, the electronic device can determine the current start code, the current file and the current log item corresponding to the current write data in the synchronization log module; and then determine the shadow file name and the corresponding offset corresponding to the current write data in the current log item.

[0085] S508: Read the currently written data from the storage engine module based on the shadow file name and corresponding offset.

[0086] In a specific embodiment of the present invention, the electronic device can read the currently written data in the storage engine module based on the shadow file name and the corresponding offset. Specifically, the electronic device can determine the shadow file name, offset, and length based on the index information corresponding to the current log entry; and then read the currently written data in the storage engine module based on the shadow file name and the corresponding offset and length.

[0087] The data reuse method proposed in this invention first stores the write requests sent by the user through a synchronization log module; then, the currently written data is stored in the current data file of the storage engine module, and a corresponding current shadow file is hard-linked to the current data file; next, a first index information for the currently written data is established in the synchronization log module based on the number of the currently written data and the current shadow file; after the currently written data is submitted, a second index information is established in the storage engine module for the currently written data. In other words, in the technical solution of this invention, the synchronization log module and the storage engine module can reuse a single set of data; the synchronization log module stores the first index information for reading the currently written data based on its number; the storage engine module stores the second index information for providing query services for the currently written data. However, in existing data reuse methods, in most distributed storage systems, the synchronization log module and the storage engine module are stored separately. Write requests need to be sent to both the synchronization log module and the storage engine module separately, requiring two disk I / O operations, which severely impacts disk lifespan. Some products use a method of caching write data in memory, retaining only the latest fixed entry of write data. If the write data requested from the replica is not in the cache, a full synchronization is triggered again, leading to increased memory overhead. Therefore, compared with the prior art, the data reuse method proposed in this embodiment of the invention can enable the synchronous log module and the storage engine module to reuse a single piece of data, reduce disk write pressure under the same QPS, improve the write performance of the distributed storage system, and extend the service life of the storage device; moreover, the technical solution of this embodiment of the invention is simple and convenient to implement, easy to popularize, and has a wider range of applications.

[0088] Example 3

[0089] Figure 7 This is a first structural schematic diagram of the data multiplexing device provided in Embodiment 3 of the present invention. Figure 7 As shown, the data multiplexing device described in this embodiment of the invention may include: a first storage module 701, a second storage module 702, a first establishment module 703, and a second establishment module 704; wherein,

[0090] The first storage module 701 is used for storing a write request sent by a user through a synchronization log module; wherein the write request comprises current write data and a number of the current write data;

[0091] The second storage module 702 is used for storing the current write data into a current data file of a storage engine module, and hard chaining a current shadow file corresponding to the current data file for the current data file;

[0092] The first establishment module 703 is used for establishing first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; wherein the first index information is used for reading the current write data according to the number of the current write data;

[0093] The second establishment module 704 is used for establishing second index information of the current write data in the storage engine module after the current write data is submitted; wherein the second index information is used for providing a query service of the current write data.

[0094] Figure 8 A second structural diagram of a data multiplexing device provided by the third embodiment of the present application is shown in Fig. 3. Figure 8 As shown in Fig. 3, the device further comprises a switching module 705, which is used for judging whether each historical write data is valid write data according to version information of each historical write data pre-determined in the storage engine module and a memory lookup table pre-acquired, and if each historical write data is valid write data, switching each historical write data and second index information of each historical write data from an original directory of the storage engine module to a target directory of the storage engine module.

[0095] Further, the switching module 705 is specifically used for, if each historical write data is submitted write data, storing each historical write data and second index information of each historical write data in the target directory of the storage engine module, and deleting each historical write data and second index information of each historical write data in the original directory of the storage engine module; if each historical write data is unsubmitted write data, hard linking each historical data file in the original directory to the target directory of the storage engine module, and updating a mapping relationship between each shadow file and each historical data file.

[0096] Further, the device further comprises a first reading module 706 (not shown in the figure) and a second reading module 707 (not shown in the figure); wherein,

[0097] The first storage module 701 is configured to store the read request sent by the user through the synchronization log module, wherein the read request comprises the number of the current write data.

[0098] The first reading module 706 is configured to acquire first index information of the current write data in the synchronization log module according to the number of the current write data, and determine a shadow file name of the current write data in the storage engine module and corresponding offset and length according to the first index information of the current write data.

[0099] The second reading module 707 is configured to read the current write data in the storage engine module according to the shadow file name and corresponding offset and length.

[0100] Further, the first index information comprises at least one start_id, each start_id comprises at least one di_file, each di_file comprises at least one di_entry and index of all di_entry, each di_entry comprises magic_num, type, data, crc, the data comprises a request header and index information, the index information comprises a shadow file name, an offset and a length, the second index information comprises at least one storage unit, each storage unit comprises an original directory and a target directory, the original directory and the target directory respectively comprise 0-64 index crc, 0-64 index and 0-63 data, each data is provided with a shadow file, and the shadow file comprises a filename, an offset and a len.

[0101] The data multiplexing device described above can execute the method provided by any embodiment of the application, has the corresponding function modules and beneficial effects of executing the method. Technical details not described in detail in the embodiment can be referred to the data multiplexing method provided by any embodiment of the application.

[0102] Embodiment four

[0103] Figure 9 The structure schematic diagram of the electronic device provided by the embodiment four of the application is shown. Figure 9 A block diagram of an exemplary electronic device suitable for use in implementing embodiments of the application is shown. Figure 9 The electronic device 12 shown is merely an example and should not limit the function and use range of the embodiment of the application.

[0104] As Figure 9As shown, the electronic device 12 is in the form of a general- purpose computer. The components of the electronic device 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processing unit 16.

[0105] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics bus (e.g., an Accelerated Graphics Port, or AGP bus) and a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0106] The electronic device 12 typically includes a variety of computer system readable media. Such media can be any available media that is located either internally or externally to the electronic device 12, including both volatile and nonvolatile media, removable and non-removable media.

[0107] The system memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 9 Not shown, a removable / non-removable interface can also be provided and can include at least one drive ultra / low density (ULDM) interface allowing the drive to read from and / or write to a removable, non-removable, and / or Figure 9 In these instances, each drive can also be connected to the bus 18 via one or more data media interfaces. The storage system 28 can include a number of program products including an operating system 38, one or more application programs, other program modules 42, and program data 44. Each of the operating system 38, one or more application programs, other program modules 42, and program data 44 can be stored on the storage system 28, or in some other volatile / non-volatile computer system media, such as a RAM 30 or a cache 32. Such stored computing components represent an example of at least one program product according to some embodiments.

[0108] Program / utility 40 having a set (at least one) of program modules 42 can be stored in, for example, memory 28 by way of example, such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof can include implementation of a network environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments described herein.

[0109] The electronic device 12 can also communicate with one or more external devices 14 such as a keyboard or a pointing device, a display 24, etc.; other devices that enable a user to interact with the electronic device 12; and / or any devices (e.g., a networking module, a Figure 9 Other hardware and / or software modules that can be used in conjunction with the electronic device 12, such as microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. are not shown in FIG. 1.

[0110] The processing unit 16 performs various

[0111] Embodiment Five

[0112] Embodiment Five of the present application provides a computer storage medium.

[0113] The computer readable storage medium of an embodiment of the present application can be any media that can be read by a computer. The computer readable storage medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example but not limited to, an electrical, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0114] A computer readable signal medium can include a propagated data signal with computer executable code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport programming code.

[0115] Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0116] Computer program code for carrying out operations for aspects of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0117] It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead with reference to the appended claims, along with their full scope of equivalents.

Claims

1. A data multiplexing method, characterized by, The method comprises: storing a write request sent by a user through a synchronization log module; wherein the write request comprises current write data and a number of the current write data; storing the current write data into a current data file of a storage engine module, and hard chaining a current shadow file corresponding to the current data file for the current data file; establishing first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; wherein the first index information is established according to a current starting code, a current file and a current log item corresponding to the current write data, and is used to read the current write data according to the number of the current write data; the current starting code, the current file and the current log item corresponding to the current write data are determined according to the number of the current write data and a current shadow file name, a current offset and a current length in the current shadow file; establishing second index information of the current write data in the storage engine module after the current write data is committed; wherein the second index information is established according to at least one storage unit and original directories and target directories included in the storage unit, and is used to provide query services of the current write data.

2. The method of claim 1, wherein, The method further comprises: judging whether each historical write data is valid write data according to version information of each historical write data predetermined in the storage engine module and a memory lookup table acquired in advance; if each historical write data is valid write data, switching each historical write data and second index information of each historical write data from an original directory to a target directory of the storage engine module.

3. The method of claim 2, wherein, The switching of each historical write data and second index information of each historical write data from the original directory to the target directory of the storage engine module comprises: if each historical write data is committed write data, storing each historical write data and second index information of each historical write data in the target directory of the storage engine module, and deleting each historical write data and second index information of each historical write data in the original directory of the storage engine module; if each historical write data is uncommitted write data, hard linking each historical data file in the original directory to the target directory of the storage engine module, and updating a mapping relationship between each shadow file and each historical data file.

4. The method of claim 1, wherein, The method further comprises: storing a read request sent by the user through the synchronization log module; wherein the read request comprises the number of the current write data; acquiring first index information of the current write data in the synchronization log module according to the number of the current write data; determining a shadow file name and corresponding offset and length of the current write data in the storage engine module according to the first index information of the current write data; reading the current write data in the storage engine module according to the shadow file name and corresponding offset and length.

5. The method of claim 1, wherein, The first index information comprises at least one start number start_id; each start_id comprises at least one log file di_file; each di_file comprises at least one log entry di_entry and a log entry index index of all di_entries; Each di_entry comprises a mark magic_num, a type type, log entry data data, and a check value crc; the data comprises a request header and index information; the index information comprises a shadow file name, an offset, and a length; the second index information comprises at least one storage unit; each storage unit comprises an original directory and a target directory; the original directory and the target directory respectively comprise 0-64 index check values indexcrc, 0-64 indexes index, and 0-63 data data; each data corresponds to a shadow file; the shadow file comprises a shadow file name filename, an offset offset, and a length len.

6. A data multiplexing apparatus characterized by comprising: The device comprises a first storage module, a second storage module, a first establishment module, and a second establishment module; wherein, The first storage module is configured to store a write request sent by a user through a synchronization log module; wherein the write request comprises current write data and a number of the current write data; The second storage module is configured to store the current write data into a current data file of a storage engine module and hard chain a current shadow file corresponding to the current data file for the current data file; The first establishment module is configured to establish first index information of the current write data in the synchronization log module according to the number of the current write data and the current shadow file; wherein the first index information is established according to a current start code, a current file, and a current log entry corresponding to the current write data, is used to read the current write data according to the number of the current write data, and the current start code, the current file, and the current log entry corresponding to the current write data are determined according to the number of the current write data and a current shadow file name, a current offset, and a current length in the current shadow file; The second establishment module is configured to establish second index information of the current write data in the storage engine module after the current write data is submitted; wherein the second index information is established according to at least one storage unit and an original directory and a target directory included in the storage unit, and is used to provide a query service of the current write data.

7. The apparatus of claim 6, wherein, The device further comprises a switching module configured to determine whether each historical write data is valid write data according to version information of each historical write data in the storage engine module and a memory lookup table acquired in advance, and switch each historical write data and second index information of each historical write data from an original directory of the storage engine module to a target directory of the storage engine module if each historical write data is valid write data.

8. The apparatus of claim 7, wherein: the switching module is specifically configured to, if each historical write data is committed write data, store each historical write data and second index information of each historical write data under a target directory of the storage engine module; delete each historical write data and the second index information of each historical write data under an original directory of the storage engine module; and if each historical write data is uncommitted write data, hard link each historical data file under the original directory to a target directory of the storage engine module, and update a mapping relationship between each shadow file and each historical data file.

9. The apparatus of claim 6, wherein, The apparatus further comprises a first reading module and a second reading module; wherein, the first storage module is configured to store the read request sent by the user through the synchronization log module; wherein the read request comprises a number of the current write data; the first reading module is configured to acquire first index information of the current write data in the synchronization log module according to the number of the current write data, and determine a shadow file name and corresponding offset and length of the current write data in the storage engine module according to the first index information of the current write data; the second reading module is configured to read the current write data in the storage engine module according to the shadow file name and corresponding offset and length.

10. The apparatus of claim 6, wherein, The first index information comprises at least one start number start_id; each start_id comprises at least one log file di_file; each di_file comprises at least one log entry di_entry and a log entry index index of all di_entry; each di_entry comprises a mark magic_num, a type type, log entry data data, and a check value crc; the data comprises a request header and index information; the index information comprises a shadow file name, an offset, and a length; the second index information comprises at least one storage unit; each storage unit comprises an original directory and a target directory; the original directory and the target directory respectively comprise 0-64 index check indexcrc, 0-64 index index, and 0-63 data data; each data is correspondingly provided with a shadow file; the shadow file comprises a shadow file name filename, an offset offset, and a length len.

11. An electronic device, comprising: comprise: one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data multiplexing method of any one of claims 1 to 5.

12. A storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the data multiplexing method of any one of claims 1 to 5.

Images