Data Disk Writing Method of Database System and Storage Medium

Abstract

The present invention provides a data disk writing method and a storage medium for a database system. The database system includes a sending end and a receiving end. A data transmission channel is established between the sending end and the receiving end according to a preset transmission protocol. The transmission protocol includes a memory alignment size. The data disk writing method includes: the receiving end receives a binary stream file from the sending end, and the binary stream file carries the transmission protocol; parsing the binary stream file to obtain the carried transmission protocol; decoding the binary stream file according to the transmission protocol to obtain disk writing data; and writing the disk writing data into the disk through a buffer of the operating system. By establishing a data transmission channel between the sending end and the receiving end according to the memory alignment size, when the sending end imports data to the receiving end, it only needs to pass through the cache of the operating system, reducing the occupancy rate of the processor during the data disk writing process.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and particularly to a method for data disk writing of a database system and a storage medium. Background Art

[0002] During the use of a database system, data import operations are often performed on the database, and the imported data needs to be written to a disk for persistent storage. Due to the huge difference in processing speeds between the processor and the disk, during data import, when writing data to the disk, it needs to be buffered first through the shared buffer of the database, then through the buffer of the target operating system, and finally written to the disk, so as to improve the data writing speed.

[0003] Therefore, before the data falls onto the disk, it needs to pass through the shared buffer of the database and also through the cache of the operating system itself. That is to say, there are two caches, which increases the processor occupancy rate. Summary of the Invention

[0004] An object of the present invention is to provide a method for data disk writing of a database system and a storage medium that can solve any of the above problems.

[0005] In particular, the present invention provides a method for data disk writing of a database system. The database system includes a sending end and a receiving end, and a data transmission channel is established between the sending end and the receiving end according to a preset transmission protocol. The transmission protocol includes a memory alignment size. The data disk writing method includes:

[0006] The receiving end receives a binary stream file from the sending end, and the binary stream file carries the transmission protocol;

[0007] Parse the binary stream file to obtain the transmission protocol carried therein;

[0008] Decode the binary stream file according to the transmission protocol to obtain disk-written data;

[0009] Write the disk-written data into the disk through the buffer of the operating system.

[0010] Optionally, the transmission protocol includes: a disk writing method flag;

[0011] Before the step of decoding the binary stream file according to the transmission protocol to obtain disk-written data, it includes:

[0012] Judge whether the disk writing method flag in the transmission protocol meets a preset condition. If so, execute the step of decoding the binary stream file according to the transmission protocol to obtain disk-written data.

[0013] Optionally, when the receiving end is a server, the step of decoding the binary stream file according to the transmission protocol to obtain disk-written data includes:

[0014] Decoding the binary stream file into a row data format, so as to obtain disk-written data stored in a row storage manner.

[0015] Optionally, when the receiving end is a client, the step of decoding the binary stream file according to the transmission protocol to obtain disk-written data includes:

[0016] Decoding the binary stream file into a text format, so as to obtain disk-written data stored in a text manner.

[0017] Optionally, the step of writing the disk-written data into a disk through a buffer of an operating system includes:

[0018] Writing the disk-written data into a disk through a buffer of an operating system by using a pre-configured direct memory access channel.

[0019] Optionally, before the step that the receiving end receives the binary stream file from the sending end includes:

[0020] The sending end reads data to be imported;

[0021] Encoding and compressing the data to be imported into a binary stream file;

[0022] Sending the binary stream file to the receiving end.

[0023] Optionally, the step of reading the data to be imported includes:

[0024] The sending end maps the data to be imported from a disk to memory;

[0025] Reading the data to be imported from the memory.

[0026] Optionally, the step of encoding and compressing the data to be imported into a binary stream file includes:

[0027] Obtaining a transmission protocol between the sending end and the receiving end;

[0028] Encoding and compressing the data to be imported according to the transmission protocol.

[0029] Optionally, the step of sending the binary stream file to the receiving end includes:

[0030] Storing the binary stream file into a preset transmission queue between the sending end and the receiving end.

[0031] According to another aspect of the present invention, there is also provided a machine-readable storage medium, on which a machine-executable program is stored. When the machine-executable program is executed by a processor, it implements the data disk dropping method of the database system according to any one of the above.

[0032] The data disk dropping method of the database system of the present invention establishes a data transmission channel between the sending end and the receiving end according to the memory alignment size. When the sending end imports data to the receiving end, the sending end can convert the data to be imported to the receiving end into a binary stream file according to the pre-negotiated memory alignment size, and then the receiving end decodes the binary stream file according to the negotiated memory alignment size, so that the data file can be written into the disk of the receiving end by using the direct memory access method, that is, the data file does not need to pass through the processor. Furthermore, it only needs to pass through the cache of the operating system and does not need to pass through the shared buffer of the database, reducing the occupancy rate of the processor during the data disk dropping process.

[0033] Those skilled in the art will understand the above and other objects, advantages and features of the present invention more clearly according to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Some specific embodiments of the present invention will be described in detail hereinafter with reference to the accompanying drawings in an exemplary but not restrictive manner. The same reference numerals in the drawings denote the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0035] Figure 1 is a schematic diagram of a database system according to an embodiment of the present invention;

[0036] Figure 2 is a schematic flowchart of the data disk dropping method of the database system according to an embodiment of the present invention;

[0037] Figure 3 is a schematic flowchart of the data disk dropping method of the database system according to another embodiment of the present invention;

[0038] Figure 4 is a schematic flowchart of the data disk dropping method of the database system according to yet another embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of a machine-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] Those skilled in the art should understand that the embodiments described below are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention. This part of the embodiments is intended to explain the technical principles of the present invention, rather than to limit the protection scope of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts should still fall within the protection scope of the present invention.

[0041] As Figure 1 and Figure 2 shown, in one embodiment, the database system includes a sending end 100 and a receiving end 200. A data transmission channel is established between the sending end 100 and the receiving end 200 according to a preset transmission protocol, and the transmission protocol includes a memory alignment size. That is to say, the sending end 100 and the receiving end 200 store data according to the same memory alignment size.

[0042] The sending end 100 is configured with a sending end codec cluster 110 and a sending end disk 120. The sending end codec cluster 110 is used to perform encoding conversion on data according to the preset transmission protocol between the sending end 100 and the receiving end 200, and the sending end disk 120 is used for persistent storage of data. The receiving end 200 is configured with a receiving end codec cluster 210 and a receiving end disk 220. The receiving end codec cluster 210 is used to decode the received data file according to the preset transmission protocol between the sending end 100 and the receiving end 200, and the receiving end disk 220 is used for persistent storage of data.

[0043] As Figure 2 shown, in this embodiment, the data disk-down method of the database system generally includes:

[0044] Step S201, the receiving end receives the binary stream file from the sending end, and the binary stream file carries the transmission protocol. Specifically, the sending end 100 uses the sending end codec cluster 110 to convert the data to be sent to the receiving end 200 into a binary stream file, and the transmission protocol between the sending end 100 and the receiving end 200, that is, the memory alignment size information of the sending end 100 and the receiving end 200, is carried in the binary stream file.

[0045] Step S202, parse the binary stream file to obtain the carried transmission protocol. Specifically, after the receiving end 200 receives the binary stream file sent by the sending end 100, it parses the format header of the binary stream file to obtain the memory alignment size therein.

[0046] Step S203, decode the binary stream file according to the transmission protocol to obtain the disk-down data. Specifically, that is to decode the binary stream file into a data format that can be written into the receiving end disk 220 according to the memory alignment size.

[0047] Specifically, when the receiving end 200 is a server, the steps of decoding the binary stream file according to the transmission protocol to obtain the disk-written data include:

[0048] Decode the binary stream file into a row data format, so as to obtain the disk-written data with a row storage method. That is to say, the data is stored in a row format in the server database, so the binary stream file is decoded into a row data format, so that the decoded data can be written into the disk in a row storage manner.

[0049] In addition, when the receiving end is a client, the steps of decoding the binary stream file according to the transmission protocol to obtain the disk-written data include:

[0050] Decode the binary stream file into a text format, so as to obtain the disk-written data with a text storage method. When the receiving end 200 is a client, decoding the binary stream file into a text format facilitates direct reading or application by the user.

[0051] Step S204, write the disk-written data into the disk through the buffer of the operating system. Specifically, use a pre-configured direct memory access channel to write the disk-written data into the disk through the buffer of the operating system. A direct memory access channel, that is, DMA (Direct Memory Access), is configured between the receiving end codec cluster 210 and the receiving end disk 220. The disk-written data directly passes through the direct memory access channel and is written into the receiving end disk 220 after passing through the memory of the operating system, completing the disk-writing work.

[0052] It should be noted that the sending end 100 can be a client and the receiving end 200 can be a server. Or the sending end 100 can be a server and the receiving end 200 can be a client.

[0053] In the solution of this embodiment, by establishing a data transmission channel according to the memory alignment size between the sending end 100 and the receiving end 200, when the sending end 100 imports data to the receiving end 200, the sending end 100 can convert the data to be imported to the receiving end 200 into a binary stream file according to the pre-negotiated memory alignment size, and then the receiving end 200 decodes the binary stream file according to the negotiated memory alignment size, so that the data file can be written into the disk of the receiving end 200 by using the direct memory access method, that is, the data file does not need to be processed by the processor. Furthermore, it only needs to pass through the cache of the operating system and does not need to pass through the shared buffer of the database, reducing the occupancy rate of the processor during the data disk-writing process.

[0054] As Figure 3 shown, in one embodiment, the transmission protocol further includes a disk-writing method flag.

[0055] The data disk writing method of the database system includes:

[0056] Step S301, the receiving end receives the binary stream file from the sending end, and the binary stream file carries a transmission protocol.

[0057] Step S302, parse the binary stream file to obtain the transmission protocol carried therein.

[0058] Step S303, determine whether the disk writing method flag in the transmission protocol meets the preset conditions. If so, execute Step S304; if not, execute Step S306. Specifically, by stipulating the disk writing method flag in the transmission protocol, when the sending end 100 transmits a data file to the receiving end 200, the disk writing method flag can be added to the compressed data file. After the receiving end 200 receives the binary stream file, parse the format header of the data file to obtain the disk writing method flag, and determine whether to adopt the data disk writing method that only passes through the buffer of the operating system according to the disk writing method flag.

[0059] Only when the disk writing method flag meets the preset conditions, the data disk writing method that only passes through the buffer of the operating system is adopted. Otherwise, the ordinary disk writing method is adopted for data disk writing, that is, the double-buffer method passing through the shared buffer of the database and the buffer of the operating system. The preset conditions can be specified key values, or there is a mark at the disk writing method flag. If there is no mark, it means that the preset conditions are not met.

[0060] Step S304, decode the binary stream file according to the transmission protocol to obtain the disk writing data.

[0061] Step S305, write the disk writing data into the disk through the buffer of the operating system.

[0062] Step S306, adopt the ordinary disk writing method for data disk writing. That is, the double-buffer method passing through the shared buffer of the database and the buffer of the operating system.

[0063] In the solution of this embodiment, by adding a disk writing method flag in the transmission protocol, the receiving end 200 can determine whether to adopt the data disk writing method that only passes through the buffer of the operating system according to whether the parsed disk writing method flag meets the preset conditions, so that the selection of the data disk writing method is more flexible.

[0064] As Figure 4 shown, in one embodiment, the data disk writing method of the database system generally includes: Before the step of the receiving end receiving the binary stream file from the sending end, it includes:

[0065] Step S401, the sending end reads the data to be imported. Specifically, the sending end 100 reads the data that needs to be transmitted to the receiving end 200 from the sending end disk 120. Specifically, this step includes the sending end mapping the data to be imported from the disk to the memory and reading the data to be imported from the memory. That is to say, first map the data to be imported in the sending end disk 120 to the memory, and then read the data to be imported from the memory.

[0066] Step S402, encode and compress the data to be imported into a binary stream file. Specifically, obtain the transmission protocol between the sending end and the receiving end, and encode and compress the data to be imported according to the transmission protocol. That is to say, after reading the data to be imported, the sending end 100 checks the transmission protocol with the receiving end 200 to obtain the memory alignment size, and then encodes and compresses the data into a binary stream file according to the transmission protocol.

[0067] Step S403, send the binary stream file to the receiving end. Specifically, store the binary stream file in a preset transmission queue between the sending end and the receiving end. Refer to Figure 1 As shown, a transmission queue 300 is established between the sending end 100 and the receiving end 200. Each time the sending end 100 completes a binary stream file, it can put the binary stream file into the transmission queue 300, so that the receiving end 200 can sequentially obtain the binary stream files from the transmission queue 300 to ensure that the data is written to disk in an orderly manner.

[0068] This embodiment also provides a machine-readable storage medium and a computer device. Figure 5 It is a schematic diagram of a machine-readable storage medium 10 according to an embodiment of the present invention. A machine-executable program 11 is stored on the machine-readable storage medium 10. When the machine-executable program 11 is executed by a processor, it implements the data disk writing method of the database system in any of the above embodiments.

[0069] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any machine-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices.

[0070] For the description of this embodiment, the machine-readable storage medium 10 can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection part (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the machine-readable storage medium 10 can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or, if necessary, other suitable processing, and then storing it in a computer memory.

[0071] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0072] The flowcharts provided in this application are not intended to indicate that the operations of the method will be executed in any specific order, or that all operations of the method are included in every case. Additionally, the method can include additional operations. Within the scope of the technical concept provided by the method of this embodiment, additional changes can be made to the above method.

[0073] Up to this point, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived from the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and determined to cover all these other variations or modifications.

Claims

1. A data disk writing method for a database system, the database system including a sending end and a receiving end, a data transmission channel being established between the sending end and the receiving end according to a preset transmission protocol, the transmission protocol including a memory alignment size, the data disk writing method including: The receiving end receives a binary stream file from the sending end, the binary stream file carrying the transmission protocol; Parse the binary stream file to obtain the carried transmission protocol; Decode the binary stream file according to the transmission protocol to obtain disk writing data; Write the disk writing data into a disk through a buffer of an operating system.

2. The data disk writing method of the database system according to claim 1, wherein, The transmission protocol includes: a disk writing method flag; Before the step of decoding the binary stream file according to the transmission protocol to obtain disk writing data, it includes: Judge whether the disk writing method flag in the transmission protocol meets a preset condition, and if so, execute the step of decoding the binary stream file according to the transmission protocol to obtain disk writing data.

3. The data disk writing method of the database system according to claim 1, wherein, When the receiving end is a server, the step of decoding the binary stream file according to the transmission protocol to obtain disk writing data includes: Decode the binary stream file into a row data format, thereby obtaining disk writing data with a row storage method.

4. The data disk writing method of the database system according to claim 1, wherein, When the receiving end is a client, the step of decoding the binary stream file according to the transmission protocol to obtain disk writing data includes: Decode the binary stream file into a text format, thereby obtaining disk writing data with a text storage method.

5. The data disk writing method of the database system according to claim 1, wherein, The step of writing the disk writing data into a disk through a buffer of an operating system includes: Write the disk writing data into a disk through a buffer of an operating system by using a pre-configured direct memory access channel.

6. The data disk writing method of the database system according to claim 1, wherein, Before the step of the receiving end receiving a binary stream file from the sending end, it includes: The sending end reads data to be imported; Encode and compress the data to be imported into a binary stream file; Send the binary stream file to the receiving end.

7. The data disk writing method of the database system according to claim 6, wherein, The step of reading data to be imported includes: The sending end maps the data to be imported from a disk to memory; Read the data to be imported from the memory.

8. The data disk writing method of the database system according to claim 6, wherein, The step of encoding and compressing the data to be imported into a binary stream file includes: Obtain the transmission protocol between the sending end and the receiving end; Encode and compress the data to be imported according to the transmission protocol.

9. The data disk writing method of the database system according to claim 6, wherein, The step of sending the binary stream file to the receiving end includes: Store the binary stream file into a preset transmission queue between the sending end and the receiving end.

10. A machine-readable storage medium, on which a machine-executable program is stored, and when the machine-executable program is executed by a processor, it implements the data disk writing method for a database system according to any one of claims 1 to 9.

Images