Database input method, device, medium and product based on stream reading

Abstract

The application provides a database input method based on stream reading, equipment, medium and product. The database input method based on stream reading comprises the following steps: receiving a call to a pre-built stream object; under the condition that the call is the first call to the pre-built stream object, creating at least one logical block group by using a callback function, each logical block group comprising a plurality of block numbers, and the callback function being used to obtain a block number to be read; after the creation of the logical block group is completed, reading corresponding data blocks from a disk to a buffer managed by the pre-built stream object in units of the first logical block group in the at least one logical block group; and returning a buffer corresponding to the current block number to be read. The number of interactions with the disk can be reduced, the consumption of running resources can be reduced, and the database input performance can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a database input method, device, medium, and product based on streaming read. Background Technology

[0002] To compensate for the significant speed difference between disk and memory access (disk I / O latency is typically several orders of magnitude slower than memory access), database systems commonly employ caching mechanisms. By pre-reading disk data blocks and storing them in the cache, when subsequent queries request the same data, the database can directly retrieve the data from the cache, avoiding repeated costly disk I / O operations. This significantly improves query response speed and reduces overall system I / O load. Currently, databases can only perform single-block reads when accessing relational data, leading to frequent system I / O calls and impacting performance. Summary of the Invention

[0003] One object of the present invention is to provide a database input method, device, medium and product based on streaming read that can solve any of the above problems.

[0004] Specifically, the present invention provides a database input method based on streaming reads, comprising: A call to the pre-built stream object was received; Under the condition that the call is the first call to the pre-built stream object, at least one logical block group is created using a callback function. Each logical block group contains multiple block numbers, and the callback function is used to obtain the block number to be read. After the logical block group is created, the corresponding data block is read from the disk into the buffer managed by the pre-built stream object, taking the first logical block group in at least one logical block group as the unit. Returns the buffer corresponding to the current block number to be read.

[0005] Optionally, the step of creating at least one logical block group using a callback function includes: Check if the number of fixed buffers plus the number of unprocessed block numbers in the pre-built stream object is less than the total number of buffers in the pre-built stream object. The unprocessed block number is the number of the block to be read that the pre-built stream object has obtained but whose buffer has not yet been fixed. If so, use the callback function to obtain the block number to be read; Determine whether the preset conditions have been met. If so, create a logical block group with a sequence number using the unprocessed block number, and fix the corresponding buffer for the created logical block group. Return to the step of checking whether the number of fixed buffers plus the number of unprocessed block numbers in the pre-built stream object is less than the total number of buffers in the pre-built stream object.

[0006] Optionally, after the step of fixing the corresponding buffer for the created logical block group, the following steps are included: Determine whether the currently created logical block group needs a prefetch operation. If so, perform a prefetch operation on the currently created logical block group.

[0007] Optionally, the steps of creating a logical block group with a sequence number using the unprocessed block number include: Determine if the remaining number of fixed buffers in the entire database meets the requirements. If yes, create a logical block group with a sequence number using all unprocessed block numbers. If no, create a logical block group with a sequence number using unprocessed block numbers that do not exceed the remaining number of fixed buffers in the entire database. Fix the corresponding buffer for the created logical block group and end the logical block group creation operation.

[0008] Optionally, the preset conditions include that the block number to be read currently obtained by the callback function is not consecutive with all unprocessed block numbers, or that the number of unprocessed block numbers reaches a set number.

[0009] Optionally, if the call is not the first call to the pre-built stream object, an unprocessed logical block group is detected in the pre-built stream object, and the data corresponding to the first logical block group in the unprocessed logical block group is read into the buffer managed by the pre-built stream object.

[0010] Optionally, the step of receiving a call to a pre-built stream object includes: Determine whether a fast scan mode needs to be performed on the pre-built stream object. If so, use a callback function to obtain the block number to be read. Check if the data block corresponding to the block number to be read obtained by the callback function exists in the buffer. If it does, return the buffer corresponding to the block number to be read obtained by the callback function. If not, fix the buffer for the block number to be read obtained by the callback function, turn off the fast scan mode, read the data block corresponding to the block number to be read obtained by the callback function from the disk into the buffer, and return the buffer corresponding to the block number to be read obtained by the callback function.

[0011] According to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer executable program stored in the memory and running on the processor, wherein the processor, when executing the computer executable program, implements the database input method based on streaming read according to any of the preceding claims.

[0012] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer-executable program is stored, which, when executed by a processor, implements the database input method based on streaming read according to any of the preceding claims.

[0013] According to another aspect of the present invention, a computer program product is also provided, comprising a computer executable program that, when executed by a processor, implements the database input method based on streaming read according to any of the preceding claims.

[0014] The database input method based on streaming read of this invention creates at least one logical block group using a callback function after receiving the first call to a pre-built stream object. Each logical block group contains multiple block numbers, and the callback function is used to obtain the block number to be read. After multiple logical block groups are created, the first logical block group contains the current block number to be read. Data blocks corresponding to the first logical block group are read from the disk into the buffer managed by the pre-built stream object, and the buffer corresponding to the current block number is returned. On the one hand, this scheme reads data blocks corresponding to multiple block numbers into the buffer at once, reducing the number of interactions with the disk and helping to reduce runtime resource consumption. On the other hand, this scheme allows the pre-built stream object to obtain all the block numbers to be read through the callback function, thereby creating logical block groups and preparing the corresponding buffers in advance, completing the preparation for reading data blocks from the disk to the buffer, thus effectively improving database input performance.

[0015] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0016] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic flowchart of a database input method based on streaming reading according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of creating a logical block group in a database input method based on streaming reading according to an embodiment of the present invention; Figure 3 This is a partial schematic flowchart of a database input method based on streaming reading according to an embodiment of the present invention; Figure 4 This is a partial schematic flowchart of a database input method based on streaming reading according to another embodiment of the present invention; Figure 5 This is a partial schematic flowchart of a database input method based on streaming reading according to yet another embodiment of the present invention; Figure 6This is a schematic flowchart of a database input method based on streaming reading according to another embodiment of the present invention; Figure 7 This is a schematic flowchart of creating a logical block group in a database input method based on streaming reading according to another embodiment of the present invention; Figure 8 This is a schematic diagram of a database architecture according to an embodiment of the present invention; Figure 9 This is a schematic diagram of a computer device according to an embodiment of the present invention; Figure 10 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; Figure 11 This is a schematic diagram of a computer program product according to an embodiment of the present invention. Detailed Implementation

[0017] Those skilled in the art should understand that the embodiments described below are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. These partial embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0018] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0019] The flowcharts provided in this invention are not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0020] like Figure 1 As shown, in one embodiment, the database input method based on streaming reads generally includes: Step S101: A call to the pre-built stream object is received.

[0021] Specifically, the pre-built stream object is used to manage the work of reading data blocks from disk into the buffer, and the pre-built stream object provides an interface. When the database executor needs to obtain a data block, it calls the pre-built stream object by calling the interface of the pre-built stream object. The pre-built stream object determines whether the data block exists in the buffer, that is, whether it can directly return the buffer containing the data block that the executor needs to the executor and perform subsequent actions.

[0022] Under the condition that the call is the first call to a pre-built stream object: Execute step S102, using a callback function to create at least one logical block group, each logical block group containing multiple block numbers, and the callback function is used to obtain the block number to be read.

[0023] Specifically, the pre-built stream object can obtain the block numbers to be read by the executor, both currently and subsequently. The callback function is used to obtain the block numbers to be read in the order required by the executor. In other words, the pre-built stream object can use the callback function to sequentially obtain all the block numbers to be read required by the executor under this operation request event, in the order required by the executor.

[0024] After multiple logical block groups are created: Execute step S103, read the corresponding data block from the disk into the buffer managed by the pre-built stream object, taking the first logical block group in at least one logical block group as the unit.

[0025] Specifically, the pre-built stream object manages multiple buffers, each storing a data block. The first logical block group is the logical block group with the earliest sequence number, so it contains multiple block numbers, including the first block number to be returned to the executor. Data blocks are read from disk into the buffer managed by the pre-built stream object, grouped by the first logical block group. In other words, all data blocks corresponding to the block numbers in the first logical block group are read from disk into the buffer managed by the pre-built stream object in a single disk interaction. Afterward, the pre-built stream object's buffer contains the data block corresponding to the currently pending block number.

[0026] Step S104: Return the buffer corresponding to the current block number to be read.

[0027] The current block number to be read is the first block number among all the data blocks that the executor has not yet acquired, in the required order. If the data block corresponding to the current block number exists in the buffer of the pre-built stream object, the buffer containing the data block corresponding to the current block number can be returned to the executor.

[0028] like Figure 2 As shown, specifically, step S102 includes: Step S201: Check if the number of fixed buffers plus the number of unprocessed block numbers in the pre-built stream object is less than the total number of buffers in the pre-built stream object. The unprocessed block number is the block number to be read that the pre-built stream object has acquired but whose buffer has not yet been fixed. If yes, proceed to step S202; otherwise, proceed to step S206.

[0029] Specifically, once the buffer determines the data blocks that need to be allocated or the data blocks that have already been stored, it enters a fixed state and becomes a fixed buffer.

[0030] Step S202: Use a callback function to obtain the block number to be read.

[0031] Specifically, this means using callback functions to sequentially obtain the block numbers to be read by the executor in the required order of the executor block numbers.

[0032] Step S203: Determine whether the preset conditions have been met. If yes, proceed to step S204; otherwise, proceed to step S201. For each block number to be read, determine whether the preset conditions have been met.

[0033] Specifically, the preset conditions include: the block number to be read currently obtained by the callback function is not consecutive with all unprocessed block numbers; that is, the preset condition is met when the block number to be read currently obtained by the callback function is not consecutive with all unprocessed block numbers. Alternatively, the number of unprocessed block numbers reaches a set number; that is, the preset condition is met when the number of unprocessed block numbers reaches a set number.

[0034] Alternatively, the preset condition is that either the block number to be read currently obtained by the callback function is not consecutive with all unprocessed block numbers, or the number of unprocessed block numbers reaches a set number. In other words, the preset condition is met when the block number to be read currently obtained by the callback function is not consecutive with all unprocessed block numbers. The preset condition is also met when the number of unprocessed block numbers reaches the set number.

[0035] Consecutive block numbers can improve read efficiency. Setting a limit can prevent the number of block numbers in each logical block group from being too large, which would cause the read time from the disk to the buffer to be too long and affect the execution efficiency of the executor.

[0036] Step S204: Create a logical block group with a sequence number using the unprocessed block number.

[0037] Specifically, when the preset conditions are met, a logical block group with assigned sequence numbers is generated, and all currently unprocessed block numbers are placed into this logical block group.

[0038] Step S205: Fix the corresponding buffer for the created logical block group.

[0039] Specifically, after creating a logical block group, a corresponding buffer is allocated for each block number within the logical block group. The data blocks corresponding to each subsequent block number are then stored in the allocated buffers.

[0040] After fixing the corresponding buffer for the created logical block group, return to step S201.

[0041] Step S206: Determine if there are any unprocessed block numbers within the pre-built flow object. If yes, proceed to step S204. If no, end the logical block group creation operation.

[0042] In this embodiment, after receiving the first call to the pre-built stream object, at least one logical block group is created using a callback function. Each logical block group contains multiple block numbers, and the callback function is used to obtain the block number to be read. After multiple logical block groups are created, the first logical block group contains the current block number to be read. Data blocks corresponding to the first logical block group are read from the disk into the buffer managed by the pre-built stream object, and the buffer corresponding to the current block number is returned. On the one hand, this solution reads data blocks corresponding to multiple block numbers into the buffer at once, reducing the number of interactions with the disk and helping to reduce runtime resource consumption. On the other hand, this solution allows the pre-built stream object to obtain all the block numbers to be read through the callback function, thereby creating logical block groups and preparing the corresponding buffers in advance, completing the preparation for reading data blocks from the disk to the buffer, thus effectively improving database input performance.

[0043] like Figure 3 As shown, in one embodiment, after the step of fixing the corresponding buffer for the created logical block group, the following is included: Step S301: Determine whether the currently created logical block group needs a prefetch operation. If yes, proceed to step S302; otherwise, proceed to step S201.

[0044] Step S302: Perform a prefetch operation on the currently created logical block group. Additionally, while performing the prefetch operation, step S201 can continue to be executed.

[0045] Specifically, the prefetch operation involves reading all data blocks corresponding to the block numbers within a logical block group from the disk into the database's shared buffer. When data blocks are subsequently needed to be read into the buffer managed by the pre-built stream object, they can be directly read from the shared buffer without going through the disk. Furthermore, the prefetch operation can be performed in parallel with the operation of directly reading other logical block groups from the disk into the buffer managed by the pre-built stream object. This effectively improves data input efficiency.

[0046] like Figure 4 As shown, step S204 includes: Step S401: Determine whether the number of remaining fixed buffers in the entire database meets the requirements. If yes, proceed to step S402; otherwise, proceed to step S403.

[0047] Specifically, besides the operation request corresponding to this pre-built stream object, the database also has other operation requests that will read data blocks from the disk into the buffer. The number of remaining fixable buffers in the entire database is also limited. Therefore, even if the buffer managed by this pre-built stream object meets the requirement for the number of unprocessed block numbers, the total number of remaining fixable buffers in the entire database may not meet the requirement. Therefore, it is necessary to determine whether the total number of remaining fixable buffers in the entire database is greater than or equal to the number of unprocessed block numbers.

[0048] Step S402: Create a logical block group with a sequence number using all unprocessed block numbers.

[0049] If the remaining number of fixable buffers in the entire database meets the requirements, create a logical block group with a sequence number using all unprocessed block numbers. Then, fix the corresponding buffers for the created logical block group and return to the step of checking whether the number of fixed buffers plus the number of unprocessed block numbers in the pre-built stream object is less than the total number of buffers in the pre-built stream object.

[0050] Step S403: Create a logical block group with a sequence number using the unprocessed block numbers that do not exceed the number of remaining fixed buffers in the entire database.

[0051] If the remaining number of fixable buffers in the entire database is insufficient to meet the demand, create a logical block group with a sequence number using unprocessed block numbers that do not exceed the remaining number of fixable buffers in the entire database. Alternatively, if the remaining number of fixable buffers in the entire database is zero, terminate the logical block group creation operation.

[0052] Step S404: Fix the corresponding buffer for the created logical block group and end the logical block group creation operation.

[0053] In this embodiment, by additionally determining whether the number of remaining fixed buffers in the entire database meets the requirements, it helps to prevent the overall operating resources of the database from exceeding the limit.

[0054] like Figure 5 As shown, in one embodiment, the database input method based on streaming reads generally includes: Step S501: A call to the pre-built stream object is received.

[0055] If the call is not the first call to the pre-built stream object, an unprocessed group of logical blocks is detected within the pre-built stream object: Step S502: Read the data block corresponding to the first logical block group in the unprocessed logical block group into the buffer managed by the pre-built stream object. The first logical block group is the logical block group with the earliest sequence number. Unprocessed logical block groups are those whose data blocks corresponding to the included block numbers have not yet been stored in the buffer.

[0056] In other words, each time a pre-built stream object is called, only one logical block group is read, which helps to quickly end the call and ensure the efficiency of the executor.

[0057] Additionally, if the call is not the first call to the pre-built stream object, and it is found that there are still unprocessed block numbers within the pre-built stream object, or the block numbers to be read can still be obtained through the callback function, at least one logical block group is created. The steps for creating a logical block group are as follows: Figure 2 As shown.

[0058] Additionally, the steps following receiving a call to a pre-built stream object include: Determine whether a fast scan mode needs to be performed on the pre-built stream object. If so, use a callback function to obtain the block number to be read. Check if the data block corresponding to the block number to be read obtained by the callback function exists in the buffer. If it does, return the buffer corresponding to the block number to be read obtained by the callback function. If not, fix the buffer for the block number to be read obtained by the callback function, turn off the fast scan mode, read the data block corresponding to the block number to be read obtained by the callback function from the disk into the buffer, and return the buffer corresponding to the block number to be read obtained by the callback function.

[0059] By setting an on / off fast scan mode, fast scanning can be enabled even when there are already many cached data blocks in the pre-built stream object, thereby greatly reducing the time to call the pre-built stream object.

[0060] like Figure 6 As shown, in one embodiment, the database input method based on streaming reads generally includes: Step S601: A call to the pre-built stream object is received.

[0061] Step S602: Determine whether a fast scan mode needs to be performed on the pre-built stream object. If yes, proceed to step S603; otherwise, proceed to step S604.

[0062] Step S603: Use a callback function to obtain the block number to be read.

[0063] Specifically, the callback function is used to obtain the block number that the executor currently needs to read.

[0064] Step S6031: Check if the data block corresponding to the block number to be read obtained by the callback function exists in the buffer. If yes, proceed to step S605; otherwise, proceed to step S6032.

[0065] Check if the data block corresponding to the block number to be read obtained from the callback function exists in the buffer managed by the pre-built stream object.

[0066] Step S6032 is to fix the buffer for the block number to be read obtained by the callback function.

[0067] If the data block corresponding to the block number to be read does not exist in the buffer managed by the pre-built stream object, it needs to be retrieved from disk. First, fix the buffer for the block number to be read obtained by the callback function.

[0068] Step S6033: Disable fast scan mode and read the data block corresponding to the block number to be read obtained from the callback function from the disk into the buffer.

[0069] Because the data block corresponding to the block number to be read does not exist in the buffer managed by the pre-built stream object, the fast scan mode is turned off, and the data block corresponding to the current block number to be read is read from the disk into the buffer. Then, step S605 is executed.

[0070] Step S605: Return the buffer corresponding to the current block number to be read.

[0071] If the data block corresponding to the block number to be read exists in the buffer managed by the pre-built stream object, the buffer can be returned directly.

[0072] The current block number to be read is the first block number in the required order among all the data blocks that the executor has not yet obtained, or the block number corresponding to the data block that needs to be returned in this call.

[0073] Step S604: Determine if this is the first call to the pre-built stream object. If yes, proceed to step S6041. If no, proceed to step S6042.

[0074] In other words, it determines whether a data block already exists in the multiple buffers managed by the pre-built stream object; if not, it is the first call.

[0075] It should be noted that the fast scan mode can also be omitted, in which case it will directly determine whether it is the first call.

[0076] Step S6041: Create a logic block group using a callback function. Then proceed to step S6042.

[0077] Step S6042: Attempt to obtain the buffer corresponding to the current block number to be read.

[0078] Specifically, it checks whether the buffer managed by the pre-built stream object contains the data block corresponding to the current block number to be read. If it does, the buffer corresponding to the current block number to be read can be returned directly. If not, the corresponding data block needs to be read from the disk into the buffer managed by the pre-built stream object in units of the first logical block group.

[0079] Step S6043: Check if there are any unprocessed logical block groups in the pre-built flow object. If yes, proceed to step S6044; otherwise, proceed to step S605.

[0080] Unprocessed logical block groups are those whose data blocks corresponding to the included block numbers have not yet been stored in the buffer.

[0081] Specifically, the buffer may contain the data block corresponding to the currently pending block number only in cases other than the initial call. The initial call occurs when the block number of the data block to be returned catches up with an unprocessed logical block group, meaning the block number of the data block to be returned exists in the unprocessed logical block group, indicating that the buffer does not contain the data block. Regardless of whether the buffer containing the data block corresponding to the currently pending block number can be directly returned, if an unprocessed logical block group exists in this call, step S6044 is executed once. That is, if the corresponding buffer can be obtained in step S6042, while returning the corresponding buffer, the step of checking whether an unprocessed logical block group exists within the pre-built stream object is also executed simultaneously.

[0082] If there are no unprocessed logical block groups, it means that the data blocks required by the executor for this event have been read into the buffer. There will be no more operations to create logical block groups or read from the disk. Therefore, it can definitely obtain the buffer where the required data blocks are located and directly return the buffer corresponding to the data blocks.

[0083] It should be noted that the existence of unprocessed logical block groups can be determined by setting a counter. Specifically, the counter is incremented by 1 each time a logical block group is generated, and decremented by 1 each time the data block corresponding to a logical block group is stored in the buffer. Thus, the existence of unprocessed logical block groups can be determined by checking if the counter is 0; if it is 0, they do not exist; otherwise, they do exist.

[0084] Step S6044: Read the corresponding data blocks from the disk into the buffer managed by the pre-built stream object, in units of the first logical block group.

[0085] The data blocks are read from the disk into the buffer managed by the pre-built stream object, with the logical block group whose sequence number is the first.

[0086] Additionally, if a counter for an unprocessed logic block group is set, this step will be followed by decrementing the counter of the unprocessed logic block group by 1.

[0087] Step S6045: Expand the number of buffers managed by the pre-built stream object.

[0088] For example, it can be expanded exponentially.

[0089] Then, since the corresponding data blocks have been read from the disk into the buffer managed by the pre-built stream object in units of the first logical block group, the buffer now contains the data block corresponding to the current block number to be read, so the buffer corresponding to the current block number to be read can be returned.

[0090] Additionally, if the call is not the first call to the pre-built stream object, and it is found that there are still unprocessed block numbers within the pre-built stream object, or the block number to be read can still be obtained through the callback function, then step S6041, which uses the callback function to create a logical block group, is executed. Then step S6042 is executed. If there are no unprocessed block numbers and the block number to be read cannot be obtained through the callback function, step S6042 is executed directly.

[0091] like Figure 7 As shown, step S6041 includes: Step S701: Check whether the number of fixed buffers plus the number of unprocessed block numbers in the pre-built stream object is less than the total number of buffers in the pre-built stream object. If yes, proceed to step S702; otherwise, proceed to step S711.

[0092] Step S702: Use a callback function to obtain the block number to be read.

[0093] Step S703: Determine whether the preset conditions have been met. If yes, proceed to step S704; otherwise, proceed to step S701. For each block number to be read, determine whether the preset conditions have been met.

[0094] The preset conditions are as described above.

[0095] It should be noted that a counter for unprocessed block numbers can be set, and the unprocessed block number will be incremented by 1 if the preset conditions are not met.

[0096] Step S704: Calculate the number of remaining fixed buffers in the entire database.

[0097] Step S705: Determine whether the number of remaining fixed buffers in the entire database meets the requirements. If yes, proceed to step S706; otherwise, proceed to step S707.

[0098] Step S706: Create a logical block group with a sequence number using all unprocessed block numbers. Execute step S708.

[0099] Step S707: Create a logical block group with a sequence number using the unprocessed block numbers that do not exceed the remaining fixed buffer size of the entire database. Execute step S7071.

[0100] The steps to create a logical block group with a sequence number include: assigning a logical block group sequence number and placing unprocessed block numbers into the logical block group.

[0101] Step S7071: Fix the corresponding buffer for the created logical block group and end the work of creating the logical block group.

[0102] Step S708: Fix the corresponding buffer for the created logical block group.

[0103] It should be noted that when a counter for unprocessed logic block groups is set, after creating a logic block group with a sequence number, the counter for the unprocessed logic block group is incremented by 1.

[0104] Additionally, after fixing the corresponding buffer for the created logical block group, the operation of updating the fixed number of buffers can be performed.

[0105] It should be noted that if a counter for unprocessed block numbers is set, this step includes resetting the counter for unprocessed block numbers.

[0106] Step S709: Determine whether the currently created logical block group needs a prefetch operation. If yes, proceed to step S710; otherwise, proceed to step S201.

[0107] In step S710, a prefetch operation is performed on the currently created logical block group. Additionally, while performing the prefetch operation, step S201 can continue to be executed.

[0108] Step S711: Determine if there are any unprocessed block numbers within the pre-built flow object. If yes, proceed to step S704. If no, end the logical block group creation operation.

[0109] In this embodiment, a pre-built stream object is configured to manage data reading from the disk to the buffer. When the executor needs a data block, it calls the pre-built stream object. Internally, the pre-built stream object reads data from the disk into the buffer by creating logical block groups. This approach reads multiple data blocks corresponding to different block numbers into the buffer at once, reducing the number of interactions with the disk and helping to reduce runtime resource consumption. Furthermore, by dividing the data into multiple logical block groups, the reading of a single logical block group is prevented from being too time-consuming, allowing the executor's execution and the pre-built stream object's reading to be interleaved, improving both executor efficiency and pre-built stream object reading efficiency. Moreover, it prevents the number of buffers managed by the pre-built stream object from increasing too rapidly, thus avoiding excessively long creation times for logical block groups.

[0110] Therefore, in some implementations, the initial number of buffers managed by the pre-built stream object is set to be less than a preset threshold.

[0111] Reference Figure 8 As shown, the database architecture corresponding to this embodiment generally includes an executor 100, a pre-built stream object 200, a buffer 300, a callback function 400, and a disk 500. The executor 100 processes database data according to operation requests sent to the database by the client, and the database data is stored and accessed in units of data blocks. Therefore, the executor 100 determines the block number of the data block involved based on the operation request.

[0112] Pre-built stream object 200 manages multiple buffers 300. Pre-built stream object 200 is used to manage the work of reading data blocks from disk into the buffers. Therefore, executor 100 needs to obtain data blocks by calling pre-built stream object 200. After receiving the call, pre-built stream object 200 internally calls callback function 400.

[0113] Callback function 400 can retrieve block numbers in the order required by executor 100 and then return them to pre-built stream object 200. Then, pre-built stream object 200 internally creates logical block groups, reads data blocks from disk 500 in units of logical block groups, and then returns the buffer containing the data blocks currently required by executor 100.

[0114] In other words, when executor 100 calls pre-built stream object 200 once, pre-built stream object 200 will only return the buffer corresponding to one data block, which is the buffer corresponding to the first data block that executor 100 needs in the required order. However, pre-built stream object 200 has already built the block groups corresponding to the subsequent required data blocks and fixed the buffers internally, so it can be read directly from disk 500 later, reducing preparation work.

[0115] This embodiment also provides a computer device and a computer-readable storage medium. Figure 9 This is a schematic diagram of a computer device 10 according to an embodiment of the present invention. Figure 10 This is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention.

[0116] Computer device 10 may include memory 110, processor 120, and computer executable program 11 stored on memory 110 and running on processor 120. When processor 120 executes computer executable program 11, it implements the database input method based on streaming read of any of the above embodiments.

[0117] The computer-readable storage medium 20 stores a computer-executable program 11 thereon, which, when executed by a processor, implements the database input method based on streaming read of any of the above embodiments.

[0118] This embodiment also provides a computer program product. Figure 11 This is a schematic diagram of a computer program product 30 according to an embodiment of the present invention. The computer program product 30 includes a computer executable program 11, which, when executed by a processor 120, implements any of the database input methods based on streaming reads described above.

[0119] Specifically, the computer executable program 11 used to perform the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, computer instructions, computer-related instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages.

[0120] For the purposes of this embodiment, the computer-readable storage medium 20 can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, the computer-readable storage medium 20 can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0121] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0122] Computer device 10 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 10 can be a cloud acquisition node. Computer device 10 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 10 can be implemented in a distributed cloud acquisition environment where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud acquisition environment, program modules can reside on local or remote acquisition system storage media, including storage devices.

[0123] Computer device 10 may include a processor 120 adapted to execute stored instructions and a memory 110 that provides temporary storage space for the operation of said instructions during operation. Processor 120 may be a single-core processor, a multi-core processor, an acquisition cluster, or any other configuration. Memory 110 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0124] The processor 120 can be connected via a system interconnect (e.g., PCI, PCI-Express, etc.) to an I / O interface (input / output interface) suitable for connecting the computer device 10 to one or more I / O devices (input / output devices). I / O devices may include, for example, a keyboard and indicating devices, where indicating devices may include a touchpad or touchscreen, etc. I / O devices may be built into the computer device 10 or may be external devices connected to the acquisition device.

[0125] The processor 120 may also be linked via a system interconnect to a display interface suitable for connecting the computer device 10 to a display device. The display device may include a display screen that is a built-in component of the computer device 10. The display device may also include an external computer monitor, television, or projector connected to the computer device 10. Furthermore, a network interface controller (NIC) may be adapted to connect the computer device 10 to a network via a system interconnect. In some embodiments, the NIC may use any suitable interface or protocol (such as an Internet Minicomputer System Interface) to transmit data. The network may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. Remote devices may connect to the computer device via the network.

[0126] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A database input method based on streaming reading, comprising: A call to the pre-built stream object was received; Under the condition that the call is the first call to the pre-built stream object, at least one logical block group is created using the callback function, each logical block group containing multiple block numbers, and the callback function is used to obtain the block number to be read; After the logical block group is created, the corresponding data block is read from the disk into the buffer managed by the pre-built stream object, taking the first logical block group in the at least one logical block group as the unit. Returns the buffer corresponding to the current block number to be read.

2. The database input method based on streaming reading according to claim 1, wherein... The step of creating at least one logical block group using the callback function includes: Check whether the number of fixed buffers plus the number of unprocessed block numbers in the pre-built stream object is less than the total number of buffers in the pre-built stream object. The unprocessed block number is the block number to be read that the pre-built stream object has obtained but whose buffer has not yet been fixed. If so, use the callback function to obtain the block number to be read; Determine whether the preset conditions are met. If so, create a logical block group with a sequence number using the unprocessed block number, and fix the corresponding buffer for the created logical block group. Return to the step of checking whether the number of fixed buffers plus the number of unprocessed block numbers in the pre-built stream object is less than the total number of buffers in the pre-built stream object.

3. The database input method based on streaming reading according to claim 2, wherein... The step of fixing the corresponding buffer for the created logical block group includes: Determine whether the currently created logical block group needs a prefetch operation. If so, perform a prefetch operation on the currently created logical block group.

4. The database input method based on streaming reading according to claim 2, wherein... The step of creating a logical block group with a sequence number using the unprocessed block number includes: Determine whether the remaining number of fixed buffers in the entire database meets the requirements. If yes, create a logical block group with a sequence number using all unprocessed block numbers. If no, create a logical block group with a sequence number using unprocessed block numbers that do not exceed the remaining number of fixed buffers in the entire database. Fix the corresponding buffer for the created logical block group and end the logical block group creation operation.

5. The database input method based on streaming reading according to claim 2, wherein... The preset conditions include that the block number to be read currently obtained by the callback function is not consecutive with all unprocessed block numbers, or that the number of unprocessed block numbers reaches a set number.

6. The database input method based on streaming reading according to claim 1, wherein... If the call is not the first call to the pre-built stream object, and an unprocessed logical block group is found in the pre-built stream object, the data corresponding to the first logical block group in the unprocessed logical block group is read into the buffer managed by the pre-built stream object.

7. The database input method based on streaming reading according to claim 1, wherein... The step of receiving a call to a pre-built stream object includes: Determine whether a fast scan mode needs to be performed on the pre-built stream object; if so, use the callback function to obtain the block number to be read. Check if the data block corresponding to the block number to be read obtained by the callback function exists in the buffer. If it does, return the buffer corresponding to the block number to be read obtained by the callback function. If not, fix the buffer for the block number to be read obtained by the callback function, turn off the fast scan mode, read the data block corresponding to the block number to be read obtained by the callback function from the disk into the buffer, and return the buffer corresponding to the block number to be read obtained by the callback function.

8. A computer device comprising a memory, a processor, and a computer-executable program stored on the memory and running on the processor, wherein the processor, when executing the computer-executable program, implements the database input method based on streaming according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer-executable program stored thereon, the computer-executable program, when executed by a processor, implementing the database input method based on streaming according to any one of claims 1 to 7.

10. A computer program product comprising a computer executable program, wherein the computer executable program, when executed by a processor, implements the database input method based on streaming read according to any one of claims 1 to 7.

Images