Data query system, method and device, electronic equipment, medium and product

Abstract

The invention discloses a data query system, method and device, electronic equipment, a medium and a product, and relates to the technical field of data processing, by setting a communication interface controller, compressed data can be controlled to be directly read from a disk to a processor for decompression processing and filtering processing to obtain calculation data, the processor is preposed, and the data query efficiency is improved. Repeated transmission of compressed data is avoided, so that the data query efficiency is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a data query system, method, device, electronic device, medium and product. Background Art

[0002] In the application scenario of data, by storing data in devices such as disks, the disks and other devices can persistently store data, thereby improving the stability of the data. When the user side needs to use the data, data query processing is performed on the disk to obtain the data required by the user side.

[0003] In the related art, in order to effectively manage the data on the disk, certain processing is performed on the data before the original data is written to the disk, and when performing data query, the original data is obtained through processing and output. The processing process will cause the problem of low data query efficiency. Summary of the Invention

[0004] This application provides a data query system, method, device, electronic device, medium and product to at least solve the problem of low data query efficiency in the related art.

[0005] This application provides a data query system, including: a communication interface controller, a processor, a disk controller, a user side, and a disk. Among them, the user side is used to send a data query request to the communication interface controller; the communication interface controller is used to determine, according to the data query request, compressed data that needs to be decompressed and filtered from the disk by calling the data reading interface of the disk controller, and transmit the compressed data to the processor; the processor is used to perform decompression processing and filtering processing on the compressed data to obtain calculation data, and the processor is also used to send the calculation data to the communication interface controller; the communication interface controller is also used to send the calculation data to the user side.

[0006] This application also provides a data query method, including: according to the data query request sent by the user side, calling the data reading interface of the disk controller to determine compressed data that needs to be decompressed and filtered from the disk, and transmitting the compressed data to the processor, so that the processor performs decompression processing and filtering processing on the compressed data to obtain calculation data; receiving the calculation data sent by the processor, and sending the calculation data to the user side.

[0007] This application also provides a data query device, including: an execution module, which is used to determine, according to the data query request sent by the user side, compressed data that needs to be decompressed and filtered from the disk by calling the data reading interface of the disk controller, and transmit the compressed data to the processor, so that the processor performs decompression processing and filtering processing on the compressed data to obtain calculation data; a sending module, which is used to send the calculation data to the user side.

[0008] The present application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of the above data query method when executing the computer program.

[0009] The present application also provides a computer-readable storage medium storing a computer program therein, wherein the computer program, when executed by a processor, implements the steps of the above data query method.

[0010] The present application also provides a computer program product including a computer program, which, when executed by a processor, implements the steps of the above data query method.

[0011] Through the present application, by providing a communication interface controller, it is possible to control directly reading compressed data from a disk into a processor for decompression processing and filtering processing to obtain calculation data, implementing processor preposition, avoiding repeated transmission of compressed data, and thus improving data query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0013] Figure 1 FIG. is a schematic diagram of an application scenario of a data query system provided by an embodiment of the present application;

[0014] Figure 2 FIG. is a schematic diagram of the structure of a data query system provided by an embodiment of the present application;

[0015] Figure 3 FIG. is a schematic diagram of point-to-point transmission provided by an embodiment of the present application;

[0016] Figure 4 FIG. is a schematic diagram of memory management provided by an embodiment of the present application;

[0017] Figure 5 FIG. is a schematic diagram of radix tree management provided by an embodiment of the present application;

[0018] Figure 6 FIG. is a schematic diagram of directly reading and writing data provided by an embodiment of the present application;

[0019] Figure 7 FIG. is a schematic diagram of a calculation result mapping table provided by an embodiment of the present application;

[0020] Figure 8 FIG. is a schematic diagram of memory access provided by an embodiment of the present application;

[0021] Figure 9 Schematic diagram of memory access provided by an embodiment of this application;

[0022] Figure 10 Schematic diagram of data writing provided by an embodiment of this application;

[0023] Figure 11 Flow schematic diagram of a data query method provided by an embodiment of this application;

[0024] Figure 12 Structural schematic diagram of a data query device provided by an embodiment of this application;

[0025] Figure 13 Structural schematic diagram of an electronic device provided by an embodiment of this application. Detailed implementation manners

[0026] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0027] It should be noted that in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application, etc. of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] Exemplarily, according to the write request of the client, the data that needs to be reused is written into the disk, and the data is stored by the disk. Compared with temporary storage, the disk can store data persistently. Even if the disk is powered off or removed from the device, the data in the disk will not be lost. When the client needs to use the data in the disk, on the premise that the disk has been connected to the device and powered on, by executing the data execution query request of the client, the data specified by the client can be determined from the disk, and the data is read into the client, so as to realize that the client can freely query the data.

[0030] In practical applications, due to the limited capacity of the disk, and the data may include repetitive patterns or repetitive information (such as fixed format heads and tails or tags, etc.), after compressing the original data through a compression algorithm and then storing the data, more valid data can be stored in the disk, which can be effectively applied to scenarios with a large amount of data. The writing speed of the disk is limited by the head seek time. Compressing the data and then reading the data can reduce the head seek time compared with reading the original data from the disk, thereby improving the efficiency of data query. Compressing the data can reduce the amount of data written to the disk, thereby reducing the wear of the disk and improving the lifespan of the disk. When reading the data, according to the read request of the client, the data in the disk is decompressed and filtered to obtain the data required by the client.

[0031] In the related art, the compressed data is sequentially transmitted from the disk to the client, and the decompression process and filtering process are performed on the compressed data at the client to obtain the data required by the client. During the process of transmitting the compressed data, it needs to be transmitted multiple times repeatedly, resulting in the problem of low transmission efficiency, and further resulting in low data query efficiency.

[0032] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0033] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the data query system depends, the specific application environment architecture or specific hardware architecture is described herein. Refer to Figure 1 , Figure 1 As an example diagram of the application scenario of the data query system, the architecture of the data query system includes a processor, a disk controller, a client, and a disk. The disk is used to store data. The processor is used to perform decompression processing and filtering processing. According to the data query request of the client, the disk controller obtains the corresponding compressed data from the disk and sends it to the client. The compressed data is decompressed and filtered by the corresponding processor of the client to obtain the calculation data, and the client obtains the calculation data from the processor to realize data query.

[0034] Illustrated with the scenario example, refer to Figure 1, The compressed data needs to be transferred from the disk to the processor multiple times. The processor is used to achieve hardware acceleration and has high computing performance. By performing decompression processing and filtering processing through the processor, the efficiency of data query can be effectively improved.

[0035] Figure 2 This is a schematic structural diagram of the data query system provided by the embodiments of the present application. As Figure 2 shown, the data query system includes: a communication interface controller, a processor, a disk controller, a user terminal, and a disk. Among them,

[0036] The user terminal is used to send a data query request to the communication interface controller;

[0037] The communication interface controller is used to determine the compressed data that needs to be decompressed and filtered from the disk by calling the data reading interface of the disk controller according to the data query request, and transfer the compressed data to the processor;

[0038] The processor is used to perform decompression processing and filtering processing on the compressed data to obtain calculation data, and the processor is also used to send the calculation data to the communication interface controller;

[0039] The communication interface controller is also used to send the calculation data to the user terminal.

[0040] Exemplarily, the data query request initiated by the user terminal first passes through the communication interface controller. The communication interface controller is used to execute the data query request. The communication interface controller is connected to the processor, and the communication interface controller can control the data transmission with the processor.

[0041] Exemplarily, the processor of the present application is used to achieve hardware acceleration and has high computing performance compared with the central processing unit. The processor of the present application can be a Field-Programmable Gate Array (FPGA for short), which is not limited in the present application.

[0042] Exemplarily, the disk controller is connected to the communication interface controller and the disk. The disk controller includes an interface connected to the disk. The communication interface controller can call the interface of the disk controller to achieve access to the disk. The communication interface controller transfers the disk data from the disk to the processor by means of direct memory access, which can perform decompression processing and filtering processing in advance compared with the related technology, and avoid repeated transmission of compressed data.

[0043] Exemplarily, the decompression processing and filtering processing for the compressed data can include decompression and data filtering. The data obtained through data filtering is the data that meets the conditions of the user terminal, and the data that meets the conditions of the user terminal is the data required by the user terminal. The processor sends the data that meets the conditions of the user terminal to the communication interface controller.

[0044] Exemplarily, the communication interface controller provides the same file operation interface as the disk controller to the client, such as: open, close, read, write, etc. The disk controller can customize the processing method for reading and writing data by the client, so as to make the data processing process transparent to the client. The communication interface controller can call the file operation interface of the disk controller downward to write the processed data to disk or read it.

[0045] Combined with a scenario example, the disk stores data in disk blocks, and the amount of data stored in each disk block is a fixed size. The disk block can include data that meets the conditions of the client and redundant data. By filtering the redundant data through data filtering processing, it is possible to avoid the redundant data occupying transmission resources during repeated data transmissions and affecting the efficiency of data transmission.

[0046] In the related art, when a user writes a file, the disk needs to be formatted into a storage format corresponding to the file first, such as: NTFS / FAT / EXT4, etc. Through the formatting operation, the storage space of the disk is divided to store the metadata and data of the file. The metadata of the file are the attributes of the file, including: the name of the file, the file size, the file timestamp, and the location information of the file data on the disk. When the client performs a file operation, it first accesses the meta-information of the file, obtains the location of the file data on the disk from the meta-information, and then accesses the data of the file. In addition to the default necessary attributes, the client can add extended attributes by itself. The extended attributes are stored in the file meta-information in the form of keys and values.

[0047] In this application, the communication interface controller extends an extended information interface and a calculation interface. They are respectively used to set the calculation parameter information of the file and the operation method of the file on the disk. The extended information interface directly calls the extended attribute interface of the disk controller through the key-value of the extended information set by the communication interface controller, and stores the extended attribute key-value in the disk, which is used to mark static information of the file such as whether the data needs to be compressed and stored, the storage format information of the data, etc. The calculation interface can directly call all interfaces in the disk controller, obtain static information according to the extended information, transfer filtering conditions and other information, and call the read / write interface to obtain data or write the data that needs to be saved after calculation to disk. The implementation methods of the interfaces include, but are not limited to: extending by using the ioctl interface of the file, adding new file system calls, extending through soft links, and implementing through file system property interfaces such as sysfs. The calculation data, intermediate cache, and calculation results required for the calculation operation executed by the communication interface controller are uniformly temporarily stored in the cache of the communication interface controller. The read / write operations of the client can also choose to bypass the cache of the communication interface controller and directly call the read / write interface of the disk controller.

[0048] The data query system provided by the embodiments of the present application can control the direct transmission of compressed data in the disk to the processor for decompression processing and filtering processing to obtain calculation data by setting a communication interface controller, realizing the front placement of the processor, avoiding the redundant data in the compressed data from occupying the transmission resources during the repeated transmission process, and thus improving the data query efficiency.

[0049] In a feasible implementation, the data query request includes data offset information; the communication interface controller is used to determine the first disk position according to the data offset information and the first radix tree of the host, and the first radix tree is used to manage the disk positions of the host; the communication interface controller is also used to call the data reading interface of the disk controller to read the compressed data and the extended information corresponding to the compressed data from the disk according to the first disk position.

[0050] Exemplarily, the host side includes a communication interface controller, a disk controller, a user side, and a disk. The communication interface controller maintains the first radix tree, and the first radix tree manages the data in the disk on the host side. The data offset information is used to locate the position of the compressed data.

[0051] Optionally, the data offset information can be a logical address encoding. The offset position of the compressed data relative to the reference position is identified through the data offset information, and the decoupling of the logical layer and the disk can be realized through the data offset information, avoiding the direct exposure of the disk address.

[0052] Exemplarily, the extended information corresponds to the compressed data, and the extended information can include the compression attributes of the compressed data, the storage format of the compressed data, etc.

[0053] Optionally, the extended information is stored in the disk in the form of key-value pairs, and the extended information can be safely and quickly queried through the key-value pairs.

[0054] Optionally, the extended information can be implemented by encoding in the data stream. The data stream encoded with the extended data can be directly bound to the compressed data and the extended information, avoiding the loss of the extended information.

[0055] In this feasible implementation, the type of the compressed data for the processor can be clearly indicated through the extended information, thereby improving the reliability of the data query.

[0056] In a feasible implementation, the data query request further includes a filtering condition, and the extended information includes the compression attributes of the compressed data; the communication interface controller is used to send the compressed data, the extended information, and the filtering condition to the processor in a point-to-point transmission manner; the processor is used to decompress the compressed data according to the compression attributes to obtain decompressed data, and filter the decompressed data according to the filtering condition to obtain the calculation data communication interface controller.

[0057] Exemplarily, the filtering condition indicates the requirements of the client for data. For example, the filtering condition is a data identifier, a data type, or a data query statement, etc. The processor filters out the data irrelevant to the requirements of the client according to the filtering condition.

[0058] Exemplarily, through the point-to-point transmission method, the data in the disk can be directly transmitted to the processor for decompression processing and filtering processing, coupling the read operation and the calculation operation, avoiding intermediate forwarding, and thus improving the efficiency of data transmission.

[0059] Next, Figure 3 the point-to-point transmission will be described.

[0060] Figure 3 is a schematic diagram of the point-to-point transmission provided by the embodiment of the present application. As Figure 3 shown, the data is directly transmitted to the processor for decompression processing and filtering processing by calling the interface of the disk controller through the communication interface controller, without intermediate forwarding. The calculated data obtained by the processor performing decompression processing and filtering processing can be directly sent to the communication interface controller, thereby realizing the coupling of the read operation and the calculation operation and improving the efficiency of data layer query.

[0061] Combined with the scenario example, the compressed data is stored in the disk in the form of disk blocks. The compressed data includes a plurality of data irrelevant to the filtering condition. Transmitting the calculated data has higher transmission performance than transmitting the compressed data. By coupling the read operation and the calculation operation, the repeated transmission of the compressed data can be avoided to improve the data transmission performance.

[0062] In this feasible implementation manner, through the point-to-point transmission method, the data reading and data calculation can be sequentially and continuously executed, avoiding the low data transmission efficiency caused by intermediate forwarding, and thus improving the data query efficiency.

[0063] A feasible implementation manner is that the communication interface controller is used to determine a plurality of index values from the structure page number array of the processor; the communication interface controller is further used to determine a target memory block from the processor according to the filtering condition, the extended information, and the plurality of index values, and send the filtering condition to the target memory block. The target memory block is used to implement the filtering processing corresponding to the filtering condition and the decompression processing corresponding to the extended information; the communication interface controller is further used to determine the first bus address corresponding to the target memory block through the structure page number array; the communication interface controller is further used to establish a bus connection between the disk and the target memory block through the first bus address; the disk sends the compressed data and the extended information to the target memory block through the bus connection.

[0064] Among them, the processor is used to implement hardware acceleration. The processor includes a plurality of memory blocks, and each memory block is used to process the corresponding type of calculation.

[0065] Exemplarily, the filtering condition corresponds to the type of filtering operation, and the extended information corresponds to the type of compressed data. The target memory block determined by the filtering condition and the extended information can accurately adapt to the decompression process and the filtering process corresponding to the compressed data, thereby improving the accuracy of data query.

[0066] Exemplarily, the information corresponding to each memory block is stored in the structure page number array of the processor. The information includes index value, bus address, adapted calculation type, etc. Determine the corresponding calculation type according to the filtering condition and the extended information, determine the target index value from multiple index values according to the calculation type, and determine the memory block corresponding to the target index value as the target memory block.

[0067] Exemplarily, convert the physical address to the first bus address, establish a bus connection between the disk and the target memory block of the processor through the first bus address, and let the disk initiate direct memory transfer. Since both the disk and the target memory block of the processor are mounted on the bus, at this time, the disk can directly send the compressed data to the target memory block of the processor through direct memory transfer, and the compressed data does not need to be forwarded by the central processor of the host, thereby reducing the transmission delay.

[0068] Optionally, through the structure page number array structure, implement the API function for address transformation in the linux kernel, complete the conversion of the page to the page frame number, the page frame number to the physical address, and the physical address to the bus address.

[0069] Illustrated with a scenario example, the processor internally uses a dedicated hardware circuit to execute the operation logic of a fixed algorithm. Each memory block of the processor can only execute a fixed calculation logic. Screen to determine the memory block adapted to the specified calculation. By screening the adapted memory block, the accuracy of the decompression process and the filtering process can be improved.

[0070] Next, in combination with Figure 4 describe the memory management.

[0071] Figure 4 It is a schematic diagram of the memory management provided by the embodiment of the present application. As Figure 4As shown, for example, 64 GB of memory is installed on the host side. The central processing unit allocates the range of 0 - 64 GB in the host-side address space, and it is addressed as the host-side memory address space. The central processing unit can access data in any range of 0 - 64 GB, run the computing program specified by the client, and perform general computing. The processor is mounted on the host through a bus interface. The processor contains 4 8-GB memories that can be accessed in parallel, and through mapping, it is addressed as the host-side address space range of 64 - 96 GB. The processor contains 4 computing units that access the memory controller through an interconnection bus. Among them, computing unit 0 and computing unit 1 can only access the memories corresponding to memory controller 0 and memory controller 1, corresponding to the range of 64 GB - 80 GB on the host side; computing unit 2 and computing unit 3 can only access the memories corresponding to memory controller 2 and memory controller 3, corresponding to the range of 80 GB - 96 GB on the host side. The memory allocated from the perspective of the central processing unit is different for the computing operations that the processor can support, thus avoiding conflicts between different computing units of the processor.

[0072] In this feasible implementation, through the filtering condition and the extended information, the target memory block to be adapted and the corresponding first bus address can be accurately determined, so as to accurately send the compressed data, the filtering condition, and the extended information to the target memory block, thereby improving the accuracy of data query.

[0073] In a feasible implementation, the communication interface controller is used to insert the target memory block into the second radix tree of the processor, and the second radix tree is used to manage the memory of the processor; the communication interface controller is also used to establish the pointer address between the first radix tree and the second radix tree; the communication interface controller and the processor search for data with each other through the pointer address.

[0074] Next, in combination with Figure 5 the radix tree management will be described.

[0075] Figure 5 This is a schematic diagram of the radix tree management provided by the embodiment of the present application. As Figure 5As shown, taking the example of data query with the disk block size in units of 2 KB, the maximum bandwidth for data transfer can reach 1 MB within the processor, and the host side uses 4 KB to manage memory pages. When the user side needs to perform filtering calculations on a database table file with a size of 4 MB, the process is as follows: ① Apply for a large continuous processor page. When the communication interface controller calls the interface and finds extended information, it allocates 3 segments of memory from the processor memory address space, including a compressed page, a decompressed page, and a filtering page respectively. ② Insert the allocated processor pages into the second radix tree. The pages are managed through a structure page array, and each structure page array manages a range of 1 MB. Find the memory area that meets the calculation conditions according to the mapping field in the structure page array. Use the _mapcount field in the structure page array to concatenate the compressed page, the decompressed page, and the filtering page. The _mappcount of the filtering page is set to -1, indicating the last page. ③ The communication interface controller obtains the data offset information of the file in the disk by calling the interface of the disk controller. ④ Obtain the processor pages from the second radix tree, and obtain the bus addresses corresponding to these pages according to the structure page array structure corresponding to the pages. ⑤ The disk memory transfer engine transfers the disk block data to the processor memory according to the data offset information and the bus address of the target memory page. ⑥ The processor starts the calculation and obtains the results after decompression and filtering calculations. ⑦ Transmit the calculation results to the host side. ⑧ According to the calculation results, for the page containing the filtering results, the host side calls the filemap_get_pages function, finds that the file has extended information and this segment of the page has been allocated processor-side memory. At this time, allocate memory from the host side, create a corresponding 4 KB memory page, and transfer the corresponding 4 KB memory page extracted from the processor decompressed page to the host side.

[0076] Exemplarily, through the pointer address, mutual lookup between the first radix tree and the second radix tree can be achieved, and after the processor completes the decompression process and the filtering process, the communication interface controller can accurately obtain the calculation data.

[0077] Illustrated with a scenario example, taking the maximum data transfer bandwidth of 1MB as an example, the process of transferring compressed data to the processor is described. The read range on the host side is extended and aligned to 1MB, corresponding to a processor memory block of 1MB in size. The bus address of the processor memory block is used as the target address for direct memory transfer. Through the structure page number array, the physical address of the processor memory block on the host side is obtained, and then the physical address is converted into the first bus address, and the disk is instructed to initiate direct memory transfer. Since both the disk and the processor are mounted on the bus, at this time, the target bus address of the disk direct memory transfer is under the same bus controller as the disk, and the bus controller will directly send the data to the processor. After receiving the bus message, the processor uses the internal bus bridge to convert it into an access to the processor memory. Based on the above implementation, the compressed data does not need to be forwarded by the host-side processor, realizing point-to-point transfer between the disk and the processor, greatly reducing the transmission delay. According to the calculated data, a 4KB memory page is allocated on the host side, data is copied from the processor to the target page, and the page is inserted into the first radix tree. It is determined which pages in the processor-side memory contain row data that meet the conditions, 4KB memory is applied for these pages in batches, and the data of these pages is transferred from the processor to the host side.

[0078] In this feasible implementation, in the related art, data is transferred according to the disk block size (for example, 2KB), and only 2KB of data can be transferred each time, requiring a relatively large number of data transfer times. In this application, through the collaborative work of the first radix tree and the second radix tree, the data transfer between the disk and the processor can be managed according to the maximum data transfer bandwidth (for example, 1MB), reducing the number of data transfer times, and thus improving the data query efficiency.

[0079] In a feasible implementation, if the data query request does not include a filtering condition and the extended information does not include a compression attribute, the communication interface controller is used to call the data reading interface of the disk controller to send the data in the disk to the user side.

[0080] Next, Figure 6 the direct reading and writing of data will be described.

[0081] Figure 6 is a schematic diagram of the direct reading and writing of data provided by the embodiment of this application. As Figure 6 shown, for non-database table files with a low compression ratio, such as image files like MP4 and JEPG, non-database table files do not need to perform filtering operations such as query statements, so data can be directly read and written from the disk. For non-database table files with a high compression ratio, such as text files like TXT and DOC, only compression or decompression processing is performed during the data reading and writing process. For database table files with a high compression ratio, only compression or decompression processing and filtering operations are performed during the data reading and writing process.

[0082] In the related art, all data is compressed without discrimination. However, performing compression operations on data with a low compression ratio consumes high resources while obtaining low storage benefits. Compressing data with a low compression ratio increases overhead, resulting in a problem of low data query efficiency.

[0083] Combined with a scenario example, for a scenario where decompression and filtering are not required, a processor with high computing performance does not need to be used. Instead of using a processor, the communication interface controller can directly read data through the disk controller.

[0084] In this feasible implementation manner, according to the extended attributes, it is possible to accurately determine whether the data is suitable for performing compression or decompression processing, and accurate compression or decompression processing can be performed on the data, thereby improving data query efficiency.

[0085] A feasible implementation manner is that the communication interface controller is used to add a direct read flag bit to a data query request and call the data read interface of the disk controller to send the data in the disk to the user side; the disk controller is used to disable the cache mechanism of the disk controller when the communication interface controller calls the data read interface of the disk controller.

[0086] Exemplarily, the disk controller includes a cache mechanism. The cache mechanism is used to read data from the disk into the cache of the disk controller for processing. There is data copying during the reading process, resulting in additional overhead.

[0087] Exemplarily, for data that does not need to be processed, the direct read flag bit is used to indicate that the cache mechanism of the disk controller does not execute, and the data is directly read into the processor through the interface of the disk controller without copying the data to the cache of the disk controller, thereby reducing overhead.

[0088] In this feasible implementation manner, by using the direct read flag bit, the memory of the processor is prevented from accessing the disk, and the data is directly read into the processor, thereby reducing overhead and improving the performance of data query.

[0089] A feasible implementation method, where the calculation data includes multiple pages and multiple bitmap data. The multiple pages include multiple row data, and the multiple bitmap data indicates whether each row data meets the filtering condition. The communication interface controller is used to filter the multiple pages according to the multiple bitmap data to obtain multiple valid pages, and store the multiple valid pages in the host memory. The bitmap data corresponding to each valid page includes a preset value. The communication interface controller is used to determine multiple valid data from the multiple row data according to the multiple bitmap data. The communication interface controller is further used to generate a calculation result mapping table according to the multiple valid data, and store the calculation result mapping table in the host memory. The communication interface controller is further used to generate target data according to the calculation result mapping table, and send the target data to the user side.

[0090] Exemplarily, the target data is logically continuous but physically discontinuous data.

[0091] Next, Figure 7 the calculation result mapping table will be described.

[0092] Figure 7 It is a schematic diagram of the calculation result mapping table provided by the embodiment of the present application. As Figure 7 shown, the processor executes the decompression processor to obtain the database table page. Taking an 8KB unit as a page as an example, the maximum number of rows in each 8KB page does not exceed 256 rows. Each page is filtered separately to obtain the bitmap corresponding to the page. Multiple positions in the bitmap correspond one-to-one with multiple positions in the page. For example, if the preset value is 1, it means that the data at the corresponding position meets the filtering condition. The data at the position in the page corresponding to the position of bitmap data 1 in the bitmap is the data that meets the filtering condition, and the data at the position in the page corresponding to the position of bitmap data 0 in the bitmap is the data that does not meet the filtering condition. Only apply for the memory on the host side for the valid pages, and the other pages do not include data that meets the filtering condition, so do not apply for the memory on the host side for the other pages. Send the valid pages and bitmap data to the communication interface controller. The communication interface controller determines the positions of the data that meets the filtering condition in the first radix tree according to the bitmap and the bitmap data, constructs a calculation result mapping table according to the positions of the data that meets the filtering condition, and the target data that meets the filtering condition can be accurately determined from the memory through the mapping table.

[0093] Combined with the scenario example, the host memory is for temporary storage. The host memory has higher read and write performance compared to the disk. When using data, the data is read into the memory, and the user side directly uses the data in the memory. The data stored on the disk is discretely stored, and the data that meets the filtering conditions and the data that does not meet the filtering conditions are discretely stored. The data read from the disk includes the data that does not meet the filtering conditions, and the data that does not meet the filtering conditions still occupies the memory, resulting in ineffective occupation of the memory space. The communication interface controller of the present application determines the positions of the data that meet the filtering conditions according to the bitmap and the bitmap data, and performs mapping processing to obtain a calculation result mapping table. The data in the calculation result mapping table is logically continuous target data. The pages that meet the filtering conditions are stored in the memory on the host side, and the pages that do not meet the filtering conditions do not apply for the memory on the host side, which can effectively reduce the occupation of the memory.

[0094] Optionally, in the communication interface controller, the data opened by each process corresponds to a file handle. A calculation result mapping table is added to the file handle, and the data access is remapped according to the result of the filtering calculation, so that the read operation only accesses the data that meets the filtering conditions.

[0095] In this feasible implementation manner, the positions of the data that meet the filtering conditions can be accurately determined through the bitmap data, so as to accurately generate a calculation result mapping table. Logical continuous target data is generated through the calculation result mapping table. Only the pages that meet the filtering conditions are stored in the memory on the host side, and the pages that do not meet the filtering conditions do not apply for the use of the memory on the host side, thereby effectively reducing the occupation of the memory by the data that does not meet the filtering conditions and improving the performance of data query.

[0096] A feasible implementation manner, the communication interface controller is used to determine the first data quantity corresponding to multiple row data, and determine the second data quantity of the valid data; the communication interface controller is also used to determine a first selection ratio according to the second data quantity and the first data quantity; the communication interface controller is also used to if the first selection ratio is greater than or equal to a first preset value, then generate a second calculation result mapping table according to the tree data structure and multiple valid data; the communication interface controller is also used to if the first selection ratio is less than the first preset value, then generate a second calculation result mapping table according to the array data structure and multiple valid data.

[0097] Exemplarily, the first selection ratio being greater than or equal to the first preset value corresponds to high selection ratio data, and there are more valid data that meet the filtering conditions. The first selection ratio being less than the first preset value corresponds to low selection ratio data, and there are fewer valid data that meet the filtering conditions.

[0098] Exemplarily, refer to Figure 7, for scenarios with a relatively high selection ratio (i.e., most data meets the filtering conditions), the calculation result mapping table uses a B+ tree data structure to extract the positions and lengths of the row data that meet the filtering conditions from the database storage pages, and the nodes on the B+ tree point to the row data that meet the filtering conditions. The leaf nodes of the B+ tree form an ordered linked list, supporting fast range traversal. In scenarios with a relatively high selection ratio, range queries have high efficiency. For scenarios with a relatively low selection ratio (i.e., most data does not meet the filtering conditions), the calculation result mapping table uses an array data structure to extract the row data that meet the filtering conditions from the database storage pages and copy and splice them into the array. The array directly locates the data through offsets without hierarchical jumps in the tree structure, and the single-point access speed is high. In scenarios with a relatively low selection ratio, the latency is low and the reading efficiency is high.

[0099] In this feasible implementation, by determining the corresponding data structure based on the selection ratio, the data query efficiency can be effectively improved.

[0100] A feasible implementation, where the communication interface controller is used to send a time window to the processor; the processor is used to reorganize multiple high-frequency pages within the time window to obtain a reorganized page, the selection ratio of each high-frequency page is lower than a second preset value, and each high-frequency page includes high-frequency row data; the processor is also used to send the reorganized page to the communication interface controller.

[0101] Exemplarily, a high-frequency page is a page that appears frequently and has a relatively low selection ratio, that is, a page that is frequently accessed by the user side but has less valid data in the page. High-frequency row data is valid data.

[0102] Combined with a scenario example, for high-frequency pages, frequent appearance and a relatively low selection ratio indicate that it is necessary to extract the page and filter the page multiple times to obtain valid data, resulting in a large consumption of computing resources.

[0103] Exemplarily, the time window is used to determine a time period, and the frequency of page access is judged through the time period.

[0104] In this feasible implementation, reorganizing the high-frequency pages can merge the valid data, reduce the overhead of computing resources, and improve the performance when extracting pages.

[0105] A feasible implementation, where the communication interface controller is used to add the reorganized page to the calculation result mapping table; the communication interface controller is also used to determine multiple redundant data in the calculation result mapping table according to the reorganized page, and the multiple redundant data duplicates the data in the reorganized page; the communication interface controller is also used to delete the multiple redundant data from the calculation result mapping table; the communication interface controller is also used to generate reorganized data based on the calculation result mapping table after deleting the multiple redundant data and send the reorganized data to the user side.

[0106] Exemplarily, for the bitmap formed after filtering and calculating each page, it is accumulated bit by bit in the processor to obtain the accumulated matrix. The row records with larger values in the accumulated matrix can be determined as high-frequency row records. For high-frequency row records, a merging calculation unit can be added in the processor. For the page containing high-frequency row records and with a low selection ratio for this page, the high-frequency row records are merged into a reorganized page, and the original row records are marked invalid. At this time, the processor can output the calculation results in two formats. One is the row record bitmap that meets the filtering conditions. The other is the reorganized page containing high-frequency row records. After receiving the reorganized page, the communication interface controller adds the reorganized page to the end of the table file and starts a process to perform a cleanup operation to clean up redundant data.

[0107] In this feasible implementation, by adding the reorganized page to the calculation result mapping table and deleting redundant data, the number of valid data in the calculation result mapping table can be increased, and the interference of redundant data on data query can be reduced, thereby improving the efficiency of data query.

[0108] A feasible implementation, the processor is used to perform conversion processing on the physical address space of the host through an address converter to obtain a second bus address; the processor is also used to send calculation data to the communication interface controller through the second bus address.

[0109] Among them, the physical address space is used to map the memory of the processor.

[0110] Exemplarily, a bus bridge for converting the bus to an internal interconnection bus is added in the processor to convert the bus address access into an access to the processor memory.

[0111] Next, Figure 8 the memory access will be described.

[0112] Figure 8 This is a schematic diagram of the memory access provided by the embodiment of the present application. As Figure 8 shown, the access between the physical address space and the bus address space is established through the second bus address, and the communication interface controller can directly access the memory of the processor through the bus bridge of the processor.

[0113] Combined with the scenario example, the memory of the processor can be directly accessed through the second bus address without multiple copies of data.

[0114] In this feasible implementation, by directly accessing the memory of the processor, the resource overhead can be reduced and the performance of data query can be improved.

[0115] A feasible implementation method, where the communication interface controller is used to determine the data transfer performance value between the current host and the processor; the communication interface controller is also used to determine the target block size according to the data transfer performance value; the communication interface controller is also used to read the current address space of the processor; the communication interface controller is also used to perform block processing on the current address space according to the target block size to obtain the target address space, and create a structure page number array for the target address space.

[0116] Exemplarily, the data transfer performance value is the optimal performance of direct memory access (DMA). Direct memory access is a hardware mechanism that allows the processor to directly access memory without the byte-by-byte intervention of the central processing unit. Its core goal is to reduce the dependence of data transfer on the central processing unit, thereby improving the overall efficiency of the system.

[0117] Exemplarily, referring to Figure 8 , the memory within the processor memory address range is blocked according to the memory size of the optimal performance of direct memory access, and the corresponding structure page number array is created. Each structure page number array logically corresponds to multiple consecutive 4KB address spaces on the host side. For example: when the size of a single direct memory access is 1MB and the direct memory access performance reaches saturation, the processor side is blocked in 1MB sizes, and each 1MB corresponds to a structure page number array.

[0118] Combined with the scenario example, the performance saturation of direct memory access generally means that the transfer rate has reached the upper limit of the hardware bandwidth and cannot be further improved. When blocking according to the direct memory access size, the direct memory access controller can work with maximum efficiency, reducing the overhead of frequently starting and stopping the transfer.

[0119] In this feasible implementation method, blocking according to the optimal performance of direct memory access can optimize the bus utilization rate and reduce latency, thereby improving the data query efficiency.

[0120] A feasible implementation method, where the communication interface controller is used to add a first field to the structure page number array, and the value of the first field is used to indicate whether the data at the corresponding position between the memory of the processor and the memory of the host is synchronized; the communication interface controller is also used to add a second field to the structure page number array, and the value of the second field is used to indicate the computing units supported by the processor; the communication interface controller is also used to add a third field to the structure page number array, and the value of the third field is used to indicate multiple associated index values corresponding to multiple associated memory blocks in the processor.

[0121] Exemplarily, referring to Figure 8, the flag corresponds to the first field. For example, after a 4KB page on the host side is modified, not only a data inconsistency flag with the disk is added. At the same time, the flag in the page number array of the memory structure on the processor side is modified to indicate that the page data on the current host side is inconsistent with the page data on the processing side. When the page is written to disk or the processor performs calculations on this page, the host-side page needs to be transferred to the processor side. The mapping corresponds to the second field. The mapping field has 32 bits, and each bit represents a computing unit. Each computing unit is used to perform corresponding computing processes, such as decompression processing or filtering processing, and 32 different computing units can be supported. The _mapcount corresponds to the third field. In a single calculation by the processor, 1MB of compressed data memory, 1MB of decompressed data memory, and 1MB of filtered result memory are required. Then the associated page indexes can form a linked list to facilitate quickly finding the pages required for the calculation.

[0122] Combined with the scenario example, the memory of the processor is only used for hardware acceleration, and the types of calculations it can support are fixed and have a relatively single use. For the original uses of some fields in the page number array structure, some fields can be redefined and used.

[0123] In this feasible implementation, by adding different fields to the page number array structure, data query can be assisted from multiple dimensions, thereby improving the efficiency of data query.

[0124] A feasible implementation is that the communication interface controller is used to determine the correspondence between the address range in the host address space and the memory controller; the communication interface controller is also used to create a first memory access table according to the correspondence; the communication interface controller is also used to determine the latency of each computing unit accessing each memory controller in the processor; the communication interface controller is also used to create a second memory access table according to the latency of each computing unit accessing each memory controller; the communication interface controller is also used to send the first memory access table and the second memory access table to the processor, and the processor is used to determine the memory controller corresponding to the extended information according to the first memory access table and the second memory access table; the processor is also used to perform decompression processing and filtering processing on the compressed data through the memory controller to obtain calculation data.

[0125] Next, combined with Figure 9 the memory access will be described.

[0126] Figure 9 is a schematic diagram of the memory access provided by the embodiment of the present application. As Figure 9As shown, 6 computing units are created in the processor: Clean 0, Compress 0, Compress 1, Decompress 0, Filter 0, and Filter 1, and 6 independently accessible memory controllers are included: DDR0, DDR1, DDR2, HBM0, HBM1, and HBM2. Each computing unit can access different memory controllers, and each memory controller controls the access to the corresponding memory, that is, each computing unit has a different memory access range and different access latencies.

[0127] Exemplarily, the first memory access table can be used to accurately determine the address range corresponding to each memory controller. By establishing the second memory access table to count multiple latencies, the memory controller used for decompression processing or filtering processing can be determined according to the latency through the second memory access table.

[0128] Combined with a scenario example for illustration, refer to Figure 8 , the first memory access table records the memory controller numbers corresponding to different address ranges, and the second memory access table records the computing units supported by different address ranges and the latency of the computing unit accessing this memory controller. When it is negative, it means that access is not possible.

[0129] Optionally, the first memory access table and the second memory access table can be loaded into the processor, and through the acceleration operator in the processor, the optimal memory controller can be quickly found.

[0130] In this feasible implementation manner, by combining the first memory access table and the second memory access table, the memory in the processor can be accurately applied, thereby improving the accuracy of data query.

[0131] A feasible implementation manner is that the communication interface controller receives a data write request sent by the user side, and the data write request includes the write data; the communication interface controller is also used to send the write data to the processor according to the data write request; the processor performs cleaning processing on the write data to obtain valid write data; the communication interface controller is also used to call the data write interface of the disk controller through point-to-point transmission to store the valid write data in the disk.

[0132] Next, combined with Figure 10 the data writing will be described.

[0133] Figure 10 This is a schematic diagram of the data writing provided by the embodiment of the present application. As Figure 10As shown, the communication interface controller invokes a write operation according to a write request. The write operation, decompression process, and filtering process are executed synchronously to obtain the pages that need to be written to disk from the user side. Through direct memory transfer, the data is sent from the host side to the processor. In the processor, parallel cleaning calculations are performed on the invalid row records in the page to remove the invalid row records, and the valid row records are merged together to form a cleaned page. The processor starts the compression calculation to compress the cleaned page to obtain a compressed page. The processor sends the compressed page through point-to-point transmission and invokes the interface of the disk controller to perform a write operation to write the compressed page to the disk.

[0134] Combined with a scenario example, it shows that in the operations of adding, deleting, and modifying a database, a large amount of invalid data will be generated. For example, when modifying a database field, the content of the field is not modified in its original position, but the row record at the original position is marked as invalid, and the updated new record is inserted at the end of the file. If the data is directly written to the disk, a large amount of invalid data will gradually accumulate on the disk, and an independent process is required to periodically clean the invalid data, merge the valid data together, and delete the original page. The process of performing the cleaning operation will occupy the resources of the host side and reduce the performance of the host side. In this application, the processor performs cleaning processing and compression processing while the data is being written, which can reduce the resource occupancy of the host side.

[0135] In this feasible implementation, performing cleaning processing and compression processing while the data is being written can reduce the resource occupancy of the host side, thereby improving the performance of the host side.

[0136] Figure 11 It is a schematic flowchart of the data query method provided by the embodiment of this application. As Figure 11 shown, the method includes the following steps:

[0137] S201. According to the data query request sent by the user side, determine the compressed data that needs to be decompressed and filtered from the disk by invoking the data reading interface of the disk controller, and transfer the compressed data to the processor so that the processor performs decompression processing and filtering processing on the compressed data to obtain calculation data.

[0138] Exemplarily, the communication interface controller on the host side is responsible for controlling the execution of the data query request to obtain calculation data.

[0139] Exemplarily, the data query request includes information for locating the position of the compressed data, which can be data offset information, and the data offset information is used to locate the offset position of the compressed data on the disk.

[0140] Exemplarily, the data query request may include a filtering condition. While transmitting the compressed data to the processor, the filtering condition is also sent to the processor. The processor decompresses the compressed data to obtain decompressed data, and the processor filters the decompressed data according to the filtering condition to obtain calculation data that meets the filtering condition.

[0141] Exemplarily, the extended information of the compressed data is stored in the disk, and the compression attributes of the compressed data are stored in the extended information. The extended information is sent to the processor so that the processor can clarify whether decompression processing is required and how to perform decompression processing, thereby improving the accuracy of decompression processing.

[0142] S202. Receive the calculation data sent by the processor and send the calculation data to the user side.

[0143] Exemplarily, perform mapping processing on the calculation data to obtain logically continuous target data, and send the target data to the user side.

[0144] Combined with the scenario example, the host memory is for temporary storage. The host memory has higher read and write performance compared to the disk. When using data, the data is read into the memory, and the user side directly uses the data in the memory. The data stored in the disk is discretely stored, and the data that meets the filtering condition and the data that does not meet the filtering condition are discretely stored. The data read from the disk includes the data that does not meet the filtering condition, and the data that does not meet the filtering condition will still occupy the memory, resulting in ineffective occupation of the memory space. The communication interface controller of the present application determines the position of the data that meets the filtering condition according to the bitmap and bitmap data, and performs mapping processing to obtain a calculation result mapping table. The data in the calculation result mapping table is logically continuous target data, which can effectively reduce the occupation of the memory.

[0145] Based on the above embodiments, through hardware acceleration processing by the processor, the resource occupation on the host side can be reduced. By using a dedicated processor for decompression processing and filtering processing during the data query process, the efficiency of data query is improved.

[0146] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation manner.

[0147] Figure 12 It is a schematic structural diagram of the data query device provided by the embodiment of the present application. As Figure 12 shown, the embodiment of the present application also provides a data query device. The data query device 120 may include: an execution module 121 and a sending module 122, wherein,

[0148] An execution module 121, configured to determine compressed data that needs to be decompressed and filtered from a disk by invoking a data reading interface of a disk controller according to a data query request sent by a client, and transmit the compressed data to a processor, so that the processor performs decompression processing and filtering processing on the compressed data to obtain calculation data.

[0149] A sending module 122, configured to send calculation data to the client.

[0150] Optionally, the execution module 121 may execute Figure 11 S201 in the embodiment.

[0151] Optionally, the sending module 122 may execute Figure 11 S202 in the embodiment.

[0152] It should be noted that the data query device shown in the embodiments of the present application may execute the technical solutions shown in the above method embodiments, and the implementation principles and beneficial effects are similar, and will not be elaborated here.

[0153] Figure 13 It is a schematic structural diagram of an electronic device provided by the present application. As Figure 13 shown, the electronic device 130 provided in this embodiment includes: at least one processor 1301 and a memory 1302. Optionally, the electronic device 130 further includes a communication component 1303. Among them, the processor 1301, the memory 1302, and the communication component 1303 are connected through a bus.

[0154] In a specific implementation process, at least one processor 1301 executes computer execution instructions stored in the memory 1302, so that at least one processor 1301 executes the above data query method embodiment.

[0155] For the specific implementation process of the processor 1301, reference may be made to the above method embodiment, and the implementation principle and technical effect are similar, and will not be elaborated here in this embodiment.

[0156] In the above embodiment, it should be understood that the processor may be a central processing unit (Central Processing Unit, abbreviated as: CPU), and may also be other general-purpose processors, digital signal processors (Digital Signal Processor, abbreviated as: DSP), application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the application may be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0157] The memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0158] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0159] Embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program is configured to execute the steps in any one of the above-described data query method embodiments when running.

[0160] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs, etc., various media that can store computer programs.

[0161] Embodiments of the present application also provide a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any one of the above-described data query method embodiments.

[0162] Embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any one of the above-described data query method embodiments.

[0163] Those skilled in the art may further realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0164] The above has introduced in detail a data query system provided by this application. Specific examples are used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can still be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data query system, characterized in that: include: A communication interface controller, a processor, a disk controller, a user terminal, and a disk, wherein: The user terminal is used to send a data query request to the communication interface controller; The communication interface controller is used to determine the compressed data that needs to be decompressed and filtered from the disk by calling the data reading interface of the disk controller according to the data query request, and transmit the compressed data to the processor; The processor is used to decompress and filter the compressed data to obtain calculated data, and the processor is also used to send the calculated data to the communication interface controller; The communication interface controller is also used to send the calculated data to the user terminal.

2. The data query system according to claim 1, characterized in that: The data query request includes data offset information; The communication interface controller is used to determine a first disk position according to the data offset information and a first radix tree of the host, wherein the first radix tree is used to manage the disk position of the host; The communication interface controller is further configured to call a data reading interface of the disk controller to read the compressed data and the extended information corresponding to the compressed data from the disk according to the first disk position.

3. The data query system according to claim 2, characterized in that: The data query request further includes a filtering condition, and the extended information includes a compression attribute of the compressed data; The communication interface controller is used to send the compressed data, the extended information, and the filtering condition to the processor in a point-to-point transmission manner; The processor is used for decompressing the compressed data according to the compression attribute to obtain decompressed data, and filtering the decompressed data according to the filtering condition to obtain the computing data communication interface controller.

4. The data query system according to claim 3, characterized in that: The communication interface controller is used to determine a plurality of index values ​​from a structure page array of the processor; The communication interface controller is further used to determine a target memory block from the processor according to the filtering condition, the extended information, and the multiple index values, and send the filtering condition to the target memory block, the target memory block is used to implement filtering processing corresponding to the filtering condition and decompression processing corresponding to the extended information; The communication interface controller is further used to determine the first bus address corresponding to the target memory block through the structure page array; The communication interface controller is also used to establish a bus connection between the disk and the target memory block through the first bus address; The disk is connected via the bus to send the compressed data and the extended information to the target memory block.

5. The data query system according to claim 4, characterized in that: The communication interface controller is used to insert the target memory block into a second radix tree of the processor, and the second radix tree is used to manage the memory of the processor; The communication interface controller is further used to establish a pointer address between the first radix tree and the second radix tree; The communication interface controller and the processor search for data with each other through the pointer address.

6. The data query system according to claim 3, characterized in that: If the data query request does not include a filtering condition, and the extended information does not include a compression attribute, the communication interface controller is used to call the data reading interface of the disk controller to send the data in the disk to the user end.

7. The data query system according to claim 6, characterized in that: The communication interface controller is used to add a direct read flag to the data query request, and call the data read interface of the disk controller to send the data in the disk to the user end; The disk controller is used to disable the cache mechanism of the disk controller when the communication interface controller calls the data reading interface of the disk controller.

8. The data query system according to claim 1, characterized in that: The calculation data includes a plurality of pages and a plurality of bitmap data, the plurality of pages include a plurality of row data, and the plurality of bitmap data indicates whether each row data meets the filtering condition; The communication interface controller is used to filter the multiple pages according to the multiple bitmap data to obtain multiple valid pages, and store the multiple valid pages in the host memory, wherein the bitmap data corresponding to each valid page includes a preset value; The communication interface controller is used to determine a plurality of valid data from the row data of the plurality of valid pages according to the plurality of bitmap data; The communication interface controller is also used to generate a calculation result mapping table according to the multiple valid data, and store the calculation result mapping table in the host memory; The communication interface controller is also used to generate target data according to the calculation result mapping table, and send the target data to the user terminal.

9. The data query system according to claim 8, characterized in that: The communication interface controller is used to determine the first data quantity corresponding to the plurality of row data and determine the second data quantity of the valid data; The communication interface controller is further used to determine a first selection ratio according to the second data quantity and the first data quantity; The communication interface controller is further configured to generate a second calculation result mapping table according to the tree data structure and the plurality of valid data if the first selection ratio is greater than or equal to a first preset value; The communication interface controller is further configured to generate the second calculation result mapping table according to the array data structure and the plurality of valid data if the first selection ratio is less than the first preset value.

10. The data query system according to claim 9, characterized in that: The communication interface controller is used to send a time window to the processor; The processor is used to reorganize the multiple high-frequency pages in the time window to obtain reorganized pages, the selection ratio of each high-frequency page is lower than a second preset value, and each high-frequency page includes high-frequency row data; The processor is further configured to send the reorganized page to the communication interface controller.

11. The data query system according to claim 10, characterized in that: The communication interface controller is used to add the reorganized page to the calculation result mapping table; The communication interface controller is further used to determine a plurality of redundant data in the calculation result mapping table according to the reorganized page, wherein the plurality of redundant data overlaps with the data in the reorganized page; The communication interface controller is further used to delete the plurality of redundant data from the calculation result mapping table; The communication interface controller is further configured to generate reorganized data according to a calculation result mapping table for deleting the plurality of redundant data, and send the reorganized data to the user terminal.

12. The data query system according to claim 1, characterized in that: The processor is used to convert the physical address space of the host through the address converter to obtain a second bus address; The processor is further configured to send the calculation data to the communication interface controller via the second bus address.

13. The data query system according to claim 12, characterized in that: The communication interface controller is used to determine the current data transmission performance value between the host and the processor; The communication interface controller is also used to determine a target block size according to the data transmission performance value; The communication interface controller is also used to read the current address space of the processor; The communication interface controller is further used to perform block processing on the current address space according to the target block size to obtain the target address space and create a structure page array of the target address space.

14. The data query system according to claim 13, characterized in that: The communication interface controller is used to add a first field in the structure page array, and the value of the first field is used to indicate whether the data at the corresponding position of the memory of the processor and the memory of the host are synchronized; The communication interface controller is further used to add a second field in the structure page array, the value of the second field is used to indicate the computing unit supported by the processor; The communication interface controller is further used to add a third field in the structure page array, and the value of the third field is used to represent multiple associated index values ​​corresponding to multiple associated memory blocks in the processor.

15. The data query system according to claim 14, characterized in that: The communication interface controller is used to determine the correspondence between the address range in the host address space and the memory controller; The communication interface controller is also used to create a first memory access table according to the corresponding relationship; The communication interface controller is also used to determine the latency of each computing unit in the processor accessing each memory controller; The communication interface controller is further used to create a second memory access table according to the latency of each computing unit accessing each memory controller; The communication interface controller is further configured to send the first memory access table and the second memory access table to the processor; The processor is used for determining the memory controller corresponding to the extended information according to the first memory access table and the second memory access table; The processor is also used to decompress and filter the compressed data through the memory controller to obtain the calculated data.

16. The data query system according to claim 1, characterized in that: The communication interface controller receives a data write request sent by the user end, wherein the data write request includes write data; The communication interface controller is also used to send the write data to the processor according to the data write request; The processor cleans up the written data to obtain valid written data; The communication interface controller is also used to call the data writing interface of the disk controller in a point-to-point transmission mode, and store the valid written data in the processor into the disk.

17. A data query method, characterized in that: include: According to the data query request sent by the user end, the data reading interface of the disk controller is called to determine the compressed data that needs to be decompressed and filtered from the disk, and the compressed data is transmitted to the processor, so that the processor decompresses and filters the compressed data to obtain the calculated data; Receive the calculation data sent by the processor, and send the calculation data to the user end.

18. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the data query method as claimed in claim 17 when executing the computer program.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data query method as claimed in claim 17.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data query method according to claim 17 are implemented.

Images