A local database query method based on shared memory

Abstract

The present invention discloses a method for querying a local database based on shared memory, belonging to the technical field of databases. The technical problem to be solved by the present invention is how to eliminate the pain points of shared memory, reduce the number of data accesses and data replication times in memory, and improve the query efficiency. The technical solution adopted is as follows: This method is based on the inter-process communication mechanism of shared memory, and the query iterator object created in the local database process is shared for use in the client process, thereby realizing data query from the client to the database.

Description

Technical Field

[0001] The invention relates to the technical field of databases, in particular to a local database query method based on shared memory. Background Art

[0002] The database system runs as an independent process, and the client is another process. When the client and the database system run on the same server, the communication between them is inter-process communication. The client accesses the database generally using a Socket-based communication mechanism.

[0003] Inter-process communication is the transmission or exchange of information between different processes. The user spaces of processes are independent of each other and generally cannot access each other. The only exception is the shared memory area. In addition, the system space is a "public place" that can be accessed by all processes, so the kernel can also provide such conditions. Commonly used inter-process communication (IPC) methods include: signals, pipes, sockets, semaphores, shared memory, and message queues.

[0004] Shared memory is a mapping of a section of memory that can be accessed by other processes. This section of shared memory is created by one process, but can be accessed by multiple processes. Shared memory is the fastest IPC method, and it is specially designed to address the low efficiency of other inter-process communication methods.

[0005] The shortcomings of shared memory communication are as follows:

[0006] ①The data format in shared memory is predefined between processes;

[0007] ②The data parsing process is cumbersome and may require re-creating the data index;

[0008] ③The shared memory area must be continuous.

[0009] Therefore, how to eliminate the pain points of shared memory, reduce the number of data accesses and data copies in memory, and improve query efficiency is a technical problem that needs to be solved urgently. Summary of the invention

[0010] The technical task of the present invention is to provide a local database query method based on shared memory to solve the problem of how to remove the pain points of shared memory, reduce the number of data accesses and data copies in the memory, and improve query efficiency.

[0011] The technical task of the present invention is achieved in the following way: a local database query method based on shared memory, which is an inter-process communication mechanism based on shared memory, and shares the query iterator object created in the local database process to the client process for use, thereby enabling the client to query the database for data; the details are as follows:

[0012] S1. When the client queries data within any time period, it calls the query API interface of the database SDK;

[0013] S2. The database SDK in the client calls the query function of the database process through the gRPC mechanism;

[0014] S3. The database process applies to the shared memory management module for the space required by the iterator object, and creates a shared memory iterator on the applied shared memory space;

[0015] S4. Find the corresponding memory block object according to the query time range, and increase the reference technology of the memory block object by one;

[0016] S5. Store the memory block object pointer in the ref array of the iterator, and find the position of the first data to be queried through the positioning algorithm;

[0017] S6. Return the iterator pointer to the client's database SDK through the gRPC mechanism;

[0018] S7. After receiving the pointer address, the database SDK of the client converts the space pointed to by the pointer into an iterator object in the client process through a forced conversion instruction;

[0019] S8. The client uses the iterator object to access the data in the database SDK; it calls the value function to obtain the value of the current record, and then calls the next function to locate the next record, and at the same time uses the valid function to determine whether the query is completed;

[0020] S9. After the client completes the query, the end interface is called, and the client's database SDK calls the end query function of the database process through the gRPC mechanism;

[0021] S10, the database process reduces the reference counts of the objects in the iterator's ref array by one, and then deletes the iterator object;

[0022] S11. Return success.

[0023] Preferably, before the client queries the database for data, it executes the following:

[0024] Create shared memory between processes: By allocating a large shared memory space, the shared memory in multiple processes is set to the same starting address, so that the pointers in the shared memory can be used directly between processes, creating a basis for sharing objects between processes;

[0025] Database process manages shared memory: the database process is used to manage and maintain the data in the entire shared memory, and the client process is used to read the memory data; a shared memory management module is set in the database process, and the shared memory management module is used to provide functions for applying for and releasing space.

[0026] As a preferred method, creating inter-process shared memory is as follows:

[0027] When the database process and the client process are started, create and load the shared memory, and set the same starting address of the shared memory;

[0028] Configure the starting address of the shared memory mapping area of ​​the client process and the starting address of the shared memory mapping area of ​​the database process to be the same starting address;

[0029] After the setting is completed, a pointer to any variable stored in the shared memory is created in the shared memory. The pointer can be correctly parsed and the variable pointed to by the pointer can be read in different processes.

[0030] More preferably, the shared memory management module is also used to manage a whole continuous space allocated by the shared memory, and provide an interface for applying for and releasing a memory space of any size in the continuous area of ​​the shared memory.

[0031] Preferably, the memory management method of the shared memory management module adopts a simple and highest-performance fixed-length memory management.

[0032] Preferably, the shared memory management module also has the following functions:

[0033] ①. Initialization of the shared memory management module: The shared memory management module divides the entire shared memory space into fixed-size memory blocks according to the size of 8K, and puts the first address of the memory block into the available queue;

[0034] ②. Apply for space from the shared memory management module: The shared memory management module provides the memory application allocation interface Allocate8K. After calling the interface Allocate8K, the shared memory management module takes out the first address of the available queue head memory block and returns it;

[0035] ③. Release the space requested by the shared memory management module: The shared memory management module provides a memory release interface Release. The shared memory management module puts the first address of the incoming memory block at the end of the available queue.

[0036] More preferably, the shared memory space is used for sharing data and shared objects;

[0037] The database process stores the data and objects required for the query in the shared memory space. The shared memory space creates interrelated class objects and array space for storing data in the same way as normal memory.

[0038] More preferably, the iterator internally includes a list of all memory block objects (ZMetricVector objects) overlapping with the query time range and a pointer to the iterator's current position.

[0039] An electronic device comprising: a memory and at least one processor;

[0040] Wherein, the memory stores a computer program;

[0041] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the above-mentioned local database query method based on shared memory.

[0042] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the above-mentioned local database query method based on shared memory.

[0043] The local database query method based on shared memory of the present invention has the following advantages:

[0044] (I) The present invention addresses the problem that when querying a local database, data needs to be copied multiple times in the memory, resulting in low query performance. By using a shared memory technology, the data stored in the database does not need to be copied at all, which greatly improves performance and has the following advantages:

[0045] ① The data to be queried does not need to be copied;

[0046] ②Using shared object technology, the memory data format is shielded, simplifying the research and development work;

[0047] ③Using the iterator mechanism, the client obtains data one by one on demand without occupying memory space;

[0048] (II) The present invention addresses the pain points of shared memory communication, and based on the inter-process communication mechanism of shared memory, uses the query iterator object created in the local database process, which can be shared with the client process for use, shielding the shared memory data format, and the client does not need to create a new iterator object, thereby improving the query efficiency and simplifying the client's parsing process of shared memory data;

[0049] (iii) The present invention allocates a large shared memory space and sets the shared memory in multiple processes to the same starting address, so that the pointers in the shared memory can be directly used between processes, creating a basis for sharing objects between processes;

[0050] (IV) The present invention reduces the number of data accesses and data copies in the memory by sharing the iterator objects used for query, thereby improving the query performance. At the same time, the logic of storage format parsing of the data to be queried is hidden. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention is further described below in conjunction with the accompanying drawings.

[0052] Attached Figure 1 A schematic diagram for creating shared memory between processes;

[0053] Attached Figure 2 Diagram of managing shared memory for database processes. DETAILED DESCRIPTION

[0054] A local database query method based on shared memory of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0055] Embodiment 1:

[0056] The local database query method based on shared memory of the present invention is an inter-process communication mechanism based on shared memory, which shares the query iterator object created in the local database process to the client process for use, thereby enabling the client to query the database for data; the details are as follows:

[0057] S1. When the client queries data within any time period, it calls the query API interface of the database SDK;

[0058] S2. The database SDK in the client calls the query function of the database process through the gRPC mechanism;

[0059] S3. The database process applies to the shared memory management module for the space required by the iterator object, and creates a shared memory iterator on the applied shared memory space;

[0060] S4. Find the corresponding memory block object according to the query time range, and increase the reference technology of the memory block object by one;

[0061] S5. Store the memory block object pointer in the ref array of the iterator, and find the position of the first data to be queried through the positioning algorithm;

[0062] S6. Return the iterator pointer to the client's database SDK through the gRPC mechanism;

[0063] S7. After receiving the pointer address, the database SDK of the client converts the space pointed to by the pointer into an iterator object in the client process through a forced conversion instruction;

[0064] S8. The client uses the iterator object to access the data in the database SDK; it calls the value function to obtain the value of the current record, and then calls the next function to locate the next record, and at the same time uses the valid function to determine whether the query is completed;

[0065] S9. After the client completes the query, the end interface is called, and the client's database SDK calls the end query function of the database process through the gRPC mechanism;

[0066] S10, the database process reduces the reference counts of the objects in the iterator's ref array by one, and then deletes the iterator object;

[0067] S11. Return success.

[0068] Before the client in this embodiment queries the database for data, it executes the following:

[0069] (1) Create shared memory between processes: By allocating a large shared memory space, the shared memory in multiple processes is set to the same starting address, so that the pointers in the shared memory can be directly used between processes, creating a basis for sharing objects between processes;

[0070] (ii) Database process manages shared memory: the database process is used to manage and maintain the data in the entire shared memory, and the client process is used to read the memory data; a shared memory management module is set in the database process, and the shared memory management module is used to provide functions for applying for and releasing space.

[0071] As attached Figure 1 As shown, the creation of inter-process shared memory in this embodiment is specifically as follows:

[0072] (1) When the database process and the client process are started, create and load the shared memory and set the same starting address of the shared memory; the processes share a large memory space, which can be configured as a parameter, such as 2G.

[0073] (2) Configure the starting address of the shared memory mapping area of ​​the client process and the starting address of the shared memory mapping area of ​​the database process to be the same starting address;

[0074] (3) After the setting is completed, a pointer to any variable stored in the shared memory is created in the shared memory. The pointer can be correctly parsed and the variable pointed to by the pointer can be read in different processes.

[0075] The shared memory management module in this embodiment is also used to manage a whole continuous space allocated by the shared memory, and provides an interface for applying for and releasing any memory size space in the shared memory continuous area. The memory management method of the shared memory management module adopts a simple and high-performance fixed-length memory management. The shared memory management module also has the following functions:

[0076] ①. Initialization of the shared memory management module: The shared memory management module divides the entire shared memory space into fixed-size memory blocks according to the size of 8K, and puts the first address of the memory block into the available queue;

[0077] ②. Apply for space from the shared memory management module: The shared memory management module provides the memory application allocation interface Allocate8K. After calling the interface Allocate8K, the shared memory management module takes out the first address of the available queue head memory block and returns it;

[0078] ③. Release the space requested by the shared memory management module: The shared memory management module provides a memory release interface Release. The shared memory management module puts the first address of the incoming memory block at the end of the available queue.

[0079] The shared memory space in this embodiment is used for sharing data and shared objects;

[0080] The database process stores the data and objects required for the query in the shared memory space. The shared memory space creates interrelated class objects and array space for storing data in the same way as normal memory.

[0081] As attached Figure 2 As shown, the data is divided into many fixed-size (default is 1000) data blocks in units of ZMetricVector. Each ZMetricVector object maintains a list of the first addresses of memory blocks requested from the shared memory management module to store data. When the reserved storage space of the ZMetricVector object is insufficient, it requests space from the shared memory to store records of a fixed size (default is 100). When the amount of data stored in a ZMetricVector object reaches 1000, this object cannot insert data anymore, and the new data needs to be inserted into the newly created ZMetricVector object.

[0082] As data is continuously inserted into the database, the database maintains a list of all ZMetricVector objects.

[0083] MectricIterator refers to the iterator object returned when querying data in a certain time range. The entire iterator includes a list of all memory block objects (ZMetricVector objects) overlapping the query time range and the iterator's current position pointer. When MectricIterator is initialized, the default position is the record position corresponding to the minimum time in the query time range. The Next function moves the position pointer to the next record position in the sequence. The Valid function is used to determine whether the current record is valid. The Value function is used to return the value of this record.

[0084] Embodiment 2:

[0085] This embodiment also provides an electronic device, including: a memory and a processor;

[0086] Wherein, the memory stores computer-executable instructions;

[0087] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the local database query method based on shared memory in any embodiment of the present invention.

[0088] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0089] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0090] Embodiment 3:

[0091] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the local database query method based on shared memory in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0092] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0093] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0094] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0095] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A local database query method based on shared memory, characterized in that, this method is based on the inter - process communication mechanism of shared memory, sharing the query iterator object created in the local database process to the client process for use, thereby realizing data query from the client to the database; specifically as follows: S1. When the client queries data within any time period, it calls the query API interface of the database SDK; S2. The database SDK in the client calls the query function of the database process through the gRPC mechanism; S3. The database process applies for the space required by the iterator object from the shared memory management module, and creates a shared memory iterator on the applied shared memory space; S4. According to the query time range, find the corresponding memory block object and increment the reference count of the memory block object by one; S5. Store the memory block object pointer in the ref array of the iterator, and find the position of the first data to be queried through the positioning algorithm; S6. Return the pointer of the iterator to the database SDK of the client through the gRPC mechanism; S7. After the database SDK of the client receives the pointer address, through a type - casting instruction, convert the space pointed to by the pointer into an iterator object in the client process; S8. The client uses the iterator object to access the data in the database SDK; calls the value function to obtain the value of the current record, then calls the next function to locate the position of the next record, and at the same time judges whether the query is over through the valid function; S9. After the client finishes the query, it calls the end interface, and the database SDK of the client calls the function to end the query of the database process through the gRPC mechanism; S10. The database process decrements the reference count of each object in the ref array of the iterator by one, and then deletes the iterator object; S11. Return success.

2. The local database query method based on shared memory according to claim 1, characterized in that, before the client queries data from the database, the following operations are performed: Create inter - process shared memory: By allocating shared memory space, set the starting addresses of the shared memory in multiple processes to be the same, so that the pointers in the shared memory can be directly used among processes, creating a basis for sharing objects among processes; The database process manages the shared memory: The database process is used to manage and maintain the data in the entire shared memory, and the client process is used to read the memory data; A shared memory management module is set in the database process, and the shared memory management module is used to provide functions for applying for space and releasing space.

3. The local database query method based on shared memory according to claim 2, characterized in that, The creation of inter - process shared memory is specifically as follows: When the database process and the client process are started, create and load the shared memory, and set the same starting address of the shared memory; Configure the starting address of the shared memory mapping area of the client process to be the same as the starting address of the shared memory mapping area of the database process; After the setting is completed, a pointer to any variable stored in the shared memory is created in the shared memory, and this pointer can be correctly resolved in different processes and the variable pointed to by the pointer can be read out.

4. The method for querying a local database based on shared memory according to claim 1 or 2, characterized in that, the shared memory management module is further configured to manage an entire continuous space allocated from the shared memory, and provide interfaces for applying for and releasing spaces of any memory size in the continuous area of the shared memory.

5. The method for querying a local database based on shared memory according to claim 4, characterized in that, the memory management method of the shared memory management module adopts fixed-length memory management.

6. The method for querying a local database based on shared memory according to claim 5, characterized in that, the shared memory management module further has the following functions: ①. Initialization of the shared memory management module: The shared memory management module divides the entire shared memory space into memory blocks of a fixed size according to the size, and puts the head address of the memory block into the available queue; ②. Applying for space from the shared memory management module: The shared memory management module provides a memory application and allocation interface Allocate8K. By calling the interface Allocate8K, the shared memory management module takes out the head address of the memory block at the head of the available queue and returns it; ③. Releasing the space applied by the shared memory management module: The shared memory management module provides a memory release interface Release, and the shared memory management module puts the incoming head address of the memory block at the end of the available queue.

7. The method for querying a local database based on shared memory according to claim 6, characterized in that, the shared memory space is used for sharing data and shared objects; The database process stores the data and objects required for querying in the shared memory space, and the shared memory space creates interrelated class objects and array spaces for storing data in the normal memory usage manner.

8. The method for querying a local database based on shared memory according to claim 7, characterized in that, the iterator internally includes a list of all memory block objects overlapping with the query time range and a pointer to the current position of the iterator.

9. An electronic device, characterized in that, comprising: a memory and at least one processor; wherein, a computer program is stored on the memory; the at least one processor executes the computer program stored in the memory, so that the at least one processor executes the method for querying a local database based on shared memory according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, a computer program is stored in the computer-readable storage medium, and the computer program can be executed by a processor to implement the method for querying a local database based on shared memory according to any one of claims 1 to 8.

Images