Data query system, method, device, electronic device, medium and product

Abstract

The present application discloses a data query system, method, device, electronic device, medium and product, which relate to the field of data processing technology. By setting a communication interface controller, it can control the direct reading of compressed data from the disk to the processor for decompression and filtering to obtain calculated data, realize processor pre-positioning, avoid repeated transmission of compressed data, and thus improve data query efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to data query systems, methods, devices, electronic devices, media and products. Background Art

[0002] In data application scenarios, by storing data on devices such as disks, disks and other devices can store data persistently, thereby improving data stability. When the user needs to use the data, the data query processing is performed on the disk to obtain the data required by the user.

[0003] In related technologies, in order to effectively manage data on disk, the original data is processed before being written to the disk. When executing a data query, the original data is obtained through processing and output. This processing process may lead to low data query efficiency. Summary of the Invention

[0004] The present application provides a data query system, method, device, electronic device, medium and product to at least solve the problem of low data query efficiency in related technologies.

[0005] The present application provides a data query system, comprising: a communication interface controller, a processor, a disk controller, a user terminal, and a disk, wherein the user terminal is used to send a data query request to the communication interface controller; the communication interface controller is used to determine, based on the data query request, compressed data that needs to be decompressed and filtered from the disk by calling a data reading interface of the disk controller, and transmit the compressed data to the processor; the processor is used to decompress and filter the compressed data to obtain calculated data, and the processor is also used to send the calculated data to the communication interface controller; the communication interface controller is also used to send the calculated data to the user terminal.

[0006] The present application also provides a data query method, including: based on a data query request sent by a user terminal, calling a data reading interface of a disk controller to determine compressed data that needs to be decompressed and filtered from the disk, and transmitting the compressed data to a processor so that the processor decompresses and filters the compressed data to obtain calculated data; receiving the calculated data sent by the processor, and sending the calculated data to the user terminal.

[0007] The present application also provides a data query device, including: an execution module, which is used to call the data reading interface of the disk controller to determine the compressed data that needs to be decompressed and filtered from the disk according to the data query request sent by the user terminal, and transmit the compressed data to the processor so that the processor decompresses and filters the compressed data to obtain calculated data; a sending module, which is used to send the calculated data to the user terminal.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the above-mentioned data query method when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data query method are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data query method when executed by a processor.

[0011] Through this application, by setting up a communication interface controller, it is possible to control the direct reading of compressed data from the disk to the processor for decompression and filtering to obtain calculated data, realize processor pre-positioning, avoid repeated transmission of compressed data, and thus improve data query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 A schematic diagram of an application scenario of a data query system provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of the structure of a data query system provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of point-to-point transmission provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of memory management provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of radix tree management provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of directly reading and writing data provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of a calculation result mapping table provided in an embodiment of the present application;

[0020] Figure 8 A schematic diagram of memory access provided in an embodiment of the present application;

[0021] Figure 9 A schematic diagram of memory access provided in an embodiment of the present application;

[0022] Figure 10 A schematic diagram of data writing provided in an embodiment of the present application;

[0023] Figure 11 A flowchart of a data query method provided in an embodiment of the present application;

[0024] Figure 12 A schematic diagram of the structure of a data query device provided in an embodiment of the present application;

[0025] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] For example, based on a write request from a user, data that needs to be reused is written to the disk, and the data is stored on the disk. Compared to temporary storage, disks can store data persistently. Even if the disk loses power or is removed from the device, the data on the disk will not be lost. When the user needs to use the data on the disk, provided that the disk is connected to the device and powered on, the user can execute a data query request to determine the data specified by the user from the disk and read the data to the user, allowing the user to freely query the data.

[0030] In practical applications, due to the limited capacity of disks and the fact that data may contain repetitive patterns or information (such as fixed-format headers, footers, or tags), compressing the raw data using a compression algorithm before storing it can store more valid data on the disk, making it effective for scenarios with large data volumes. The write rate of a disk is limited by the head seek time. Compressing the data can reduce the head seek time when reading data compared to reading the raw data from the disk, thereby improving data query efficiency. Data compression reduces the amount of data written to the disk, thereby reducing disk wear and extending its lifespan. When reading data, the data on the disk is decompressed and filtered based on the user's read request to obtain the data required by the user.

[0031] In related technologies, compressed data is transmitted sequentially from a disk to a user end, where it is decompressed and filtered to obtain the data required by the user end. Multiple transmissions are required during the transmission of compressed data, resulting in low transmission efficiency and, in turn, low data query efficiency.

[0032] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0033] In conjunction with the specific application environment architecture or specific hardware architecture that the execution of the data query system depends on, the specific application environment architecture or specific hardware architecture is described here. Figure 1 , Figure 1 This is an example diagram of an application scenario of a data query system. The architecture of the data query system includes a processor, a disk controller, a user terminal, and a disk. The disk is used to store data, and the processor is used to perform decompression and filtering. According to the data query request of the user terminal, the disk controller obtains the corresponding compressed data from the disk and sends it to the user terminal. The compressed data is decompressed and filtered by the corresponding processor of the user terminal to obtain calculated data. The user terminal obtains the calculated data from the processor to realize data query.

[0034] Combined with the scene example, refer to Figure 1, compressed data needs to be transferred repeatedly from the disk to the processor. The processor is used to achieve hardware acceleration and has high computing performance. Decompression and filtering processing by the processor can effectively improve the efficiency of data query.

[0035] Figure 2 A schematic diagram of the structure of the data query system provided in the embodiment of the present application is shown as follows: Figure 2 As shown, the data query system includes: a communication interface controller, a processor, a disk controller, a user terminal, and a disk, wherein:

[0036] The user terminal is used to send a data query request to the communication interface controller;

[0037] The communication interface controller is used to determine the compressed data that needs to be decompressed and filtered from the disk by calling the data reading interface of the disk controller according to the data query request, and transmit the compressed data to the processor;

[0038] The processor is used to decompress and filter the compressed data to obtain calculated data, and the processor is also used to send the calculated data to the communication interface controller;

[0039] The communication interface controller is also used to send calculation data to the user end.

[0040] Exemplarily, a data query request initiated by the user terminal first passes through the communication interface controller, which is used to execute the data query request. The communication interface controller is connected to the processor and can control data transmission with the processor.

[0041] Exemplarily, the processor of the present application is used to implement hardware acceleration and has higher computing performance than a central processing unit. The processor of the present application may be a field programmable gate array (FPGA), which is not limited in the present application.

[0042] Exemplarily, the disk controller is connected to a communication interface controller and a disk. The disk controller includes an interface for connecting to the disk, and the communication interface controller can call the disk controller's interface to access the disk. The communication interface controller transfers disk data from the disk to the processor via direct memory transfer, which allows for pre-processing of decompression and filtering compared to related technologies, thus avoiding repeated transmission of compressed data.

[0043] Exemplarily, the decompression and filtering of compressed data may include decompression and data filtering. The data obtained through data filtering is data that meets the conditions of the user end, and the data that meets the conditions of the user end is data required by the user end. The processor sends the data that meets the conditions of the user end to the communication interface controller.

[0044] For example, the communication interface controller provides the user end with the same file operation interface as the disk controller, such as open, close, read, and write. The disk controller can customize the processing method for user end read and write data, thereby making the data processing process transparent to the user end. The communication interface controller can also call the disk controller's file operation interface to write or read the processed data to the disk.

[0045] Combined with the scenario example, the disk stores data in disk blocks. The amount of data stored in each disk block is a fixed size. The disk block can include data that meets the user's conditions and redundant data. By filtering out redundant data through data filtering, it can be avoided that redundant data occupies transmission resources during repeated data transmission, which affects the efficiency of data transmission.

[0046] In related technologies, when users write files, they need to first format the disk into the storage format corresponding to the file, such as NTFS / FAT / EXT4, etc. Through the formatting operation, the storage space of the disk is divided to store the metadata and data of the file. The metadata of a file is the attribute of the file, which includes: the name of the file, the size of the file, the timestamp of the file, and the location information of the file data on the disk. When the user side performs a file operation, it first accesses the metadata of the file, obtains the location of the file data on the disk from the metadata, and then accesses the file data. In addition to the default necessary attributes of the file attribute information, the user side can add extended attributes by itself. The extended attributes are stored in the file metadata in the form of keys and values.

[0047] In this application, the communication interface controller interface is extended with an extended information interface and a calculation interface. These are used to set file calculation parameters and the file operation method on the disk, respectively. The extended information interface uses the extended information key value set by the communication interface controller to directly call the disk controller's extended attribute interface. The extended attribute key value is stored on the disk, indicating whether the data needs to be compressed and the data storage format, and other static information about the file. The calculation interface can directly call all interfaces in the disk controller, obtain static information based on the extended information, pass information such as filtering conditions, call the read / write interface to obtain data, or write calculated data to the disk. Interface implementation methods include but are not limited to: extending the file ioctl interface, adding new file system calls, extending through soft links, and implementing through file system attribute interfaces such as sysfs. The required calculation data, intermediate cache, and calculation results of the calculation operations performed by the communication interface controller are temporarily stored in the communication interface controller's cache. User-side read and write operations can also choose to bypass the communication interface controller cache and directly call the disk controller's read / write interface.

[0048] The data query system provided in the embodiment of the present application can control the compressed data in the disk to be directly sent from the disk to the processor for decompression and filtering to obtain calculated data by setting a communication interface controller, thereby realizing processor pre-positioning and avoiding redundant data occupying transmission resources during repeated transmission of compressed data, thereby improving data query efficiency.

[0049] A feasible implementation method is that the data query request includes data offset information; the communication interface controller is used to determine the first disk position based on the data offset information and the host's first radix tree, and the first radix tree is used to manage the host's disk position; the communication interface controller is also used to call the disk controller's data reading interface to read compressed data and extended information corresponding to the compressed data from the disk based on the first disk position.

[0050] Exemplarily, the host side includes a communication interface controller, a disk controller, a user terminal, and a disk. The communication interface controller maintains a first radix tree, which manages data in the disk of the host side. The data offset information is used to locate the position of the compressed data.

[0051] Optionally, the data offset information may be a logical address encoding, which identifies the offset position of the compressed data relative to the reference position. The data offset information may be used to decouple the logical layer from the disk, thereby avoiding direct exposure of the disk address.

[0052] Exemplarily, the extended information corresponds to the compressed data, and the extended information may include compression properties of the compressed data, storage format of the compressed data, and the like.

[0053] Optionally, the extended information is stored on disk as key-value pairs, and the extended information can be retrieved safely and quickly through the key-value pairs.

[0054] Optionally, the extended information may be encoded in the data stream. By directly encoding the extended data in the data stream, the compressed data and the extended information may be bound to avoid loss of the extended information.

[0055] In this feasible implementation, the extended information can clearly indicate the type of compressed data to the processor, thereby improving the reliability of data query.

[0056] A feasible implementation method is that the data query request also includes filtering conditions, and the extended information includes compression attributes of the compressed data; the communication interface controller is used to send compressed data, extended information, and filtering conditions to the processor through point-to-point transmission; the processor is used to decompress the compressed data according to the compression attributes to obtain decompressed data, and filter the decompressed data according to the filtering conditions to obtain the calculated data communication interface controller.

[0057] Exemplarily, the filtering condition indicates the user's demand for data. For example, the filtering condition is a data identifier, a data type, or a data query statement, and the processor filters out data irrelevant to the user's demand according to the filtering condition.

[0058] For example, through point-to-point transmission, the data in the disk can be directly transmitted to the processor for decompression and filtering, coupling the read operation and the calculation operation, avoiding intermediate forwarding, and thus improving the efficiency of data transmission.

[0059] Next, combine Figure 3 Point-to-point transmission is described.

[0060] Figure 3 This is a schematic diagram of point-to-point transmission provided in the embodiment of the present application. Figure 3 As shown, the communication interface controller calls the interface of the disk controller to transfer data directly to the processor for decompression and filtering processing without the need for intermediate forwarding. The calculated data obtained by the processor after decompression and filtering processing can be sent directly to the communication interface controller, thereby realizing the coupling of read operations and calculation operations and improving the efficiency of data layer queries.

[0061] Combined with the scenario example, compressed data is stored in the form of disk blocks on the disk. The compressed data includes multiple data that are unrelated to the filtering conditions. Transmitting calculated data has higher transmission performance than transmitting compressed data. By coupling read operations and calculation operations, repeated transmission of compressed data can be avoided to improve data transmission performance.

[0062] In this feasible implementation, data reading and data calculation can be performed continuously in sequence through point-to-point transmission, avoiding low data transmission efficiency caused by intermediate forwarding, thereby improving data query efficiency.

[0063] A feasible implementation method is that the communication interface controller is used to determine multiple index values from the structure page array of the processor; the communication interface controller is also used to determine the target memory block from the processor based on the filtering conditions, extended information, and multiple index values, and send the filtering conditions to the target memory block, and the target memory block is used to implement filtering processing corresponding to the filtering conditions and decompression processing corresponding to the extended information; the communication interface controller is also used to determine the first bus address corresponding to the target memory block through the structure page array; the communication interface controller is also used to establish a bus connection between the disk and the target memory block through the first bus address; the disk sends compressed data and extended information to the target memory block through the bus connection.

[0064] The processor is used to implement hardware acceleration, and the processor includes multiple memory blocks, each of which is used to process a corresponding type of calculation.

[0065] Exemplarily, the filtering condition corresponds to the type of filtering operation, and the extended information corresponds to the type of compressed data. The target memory block determined by the filtering condition and the extended information can accurately adapt to the decompression and filtering processing corresponding to the compressed data, thereby improving the accuracy of data query.

[0066] Exemplarily, the processor's structure page array stores information corresponding to each memory block, including an index value, a bus address, an adapted calculation type, etc. The corresponding calculation type is determined based on the filter condition and the extended information. A target index value is determined from multiple index values based on the calculation type, and the memory block corresponding to the target index value is determined as the target memory block.

[0067] Exemplarily, the physical address is converted to a first bus address, a bus connection is established between the disk and the processor's target memory block via the first bus address, and the disk initiates a direct memory transfer. Because both the disk and the processor's target memory block are mounted on the bus, the disk can send compressed data directly to the processor's target memory block via direct memory transfer, eliminating the need for the compressed data to be forwarded by the host's central processing unit (CPU), thereby reducing transmission latency.

[0068] Optionally, by constructing a page array structure, compatibility with the address conversion API function in the Linux kernel is achieved, completing the conversion of page pages into page frame numbers, page frame numbers into physical addresses, and physical addresses into bus addresses.

[0069] Combined with the scenario example, the processor uses dedicated hardware circuits to execute the calculation logic of a fixed algorithm. The memory block of each processor can only execute fixed calculation logic. Screening is used to determine the memory blocks that are compatible with the specified calculation. By screening the compatible memory blocks, the accuracy of the decompression and filtering processing can be improved.

[0070] Next, combine Figure 4 Explain memory management.

[0071] Figure 4 This is a schematic diagram of memory management provided by the embodiment of this application. Figure 4As shown in the figure, for example, if 64GB of memory is installed on the host side, the CPU is allocated a range of 0-64GB in the host-side address space, which is addressed as the host-side memory address space. The CPU can access data in any range of 0-64GB, run user-specified computing programs, and perform general-purpose computations. The CPU is mounted to the host via a bus interface. The processor contains four 8GB memories that can be accessed in parallel, mapped to the host-side address space of 64-96GB. The processor contains four computing units, which access the memory controller via an interconnect bus. Computing units 0 and 1 can only access the memory corresponding to memory controllers 0 and 1, corresponding to the 64GB-80GB range on the host side. Computing units 2 and 3 can only access the memory corresponding to memory controllers 2 and 3, corresponding to the 80GB-96GB range on the host side. The memory allocated from the CPU's perspective varies depending on the computing operations supported by the processor, thereby avoiding conflicts between the processor's different computing units.

[0072] In this feasible implementation method, through filtering conditions and extended information, the adapted target memory block and the corresponding first bus address can be accurately determined, so that compressed data, filtering conditions, and extended information can be accurately sent to the target memory block, thereby improving the accuracy of data query.

[0073] In one feasible implementation, the communication interface controller is used to insert the target memory block into the second radix tree of the processor, and the second radix tree is used to manage the memory of the processor; the communication interface controller is also used to establish a pointer address between the first radix tree and the second radix tree; the communication interface controller and the processor search for data with each other through the pointer address.

[0074] Next, combine Figure 5 Describe radix tree management.

[0075] Figure 5 This is a schematic diagram of the radix tree management provided in the embodiment of this application. Figure 5As shown in the figure, for example, a data query using a disk block size of 2KB is performed. The processor's 1MB memory size achieves the maximum data transmission bandwidth, and the host uses 4KB memory pages for management. When the user needs to perform filtering calculations on a 4MB database table file, the process is as follows: ① Request a large, contiguous block of processor pages. When the communication interface controller calls the interface and detects extended information, it allocates three memory segments from the processor's memory address space: a compressed page, a decompressed page, and a filtered page. ② Insert the allocated processor pages into the second radix tree. Pages are managed using a structured page array, with each structured page array managing a 1MB range. The mapping field in the structured page array is used to search for memory areas that meet the calculation criteria. The _mapcount field in the structured page array is used to concatenate the compressed, decompressed, and filtered pages. The _mappcount of the filter page is set to -1, indicating the last page. ③ The communication interface controller obtains the file's data offset on disk by calling the disk controller's interface. ④ Retrieve the processor pages from the second radix tree and, based on the corresponding structured page array structure, obtain the bus addresses corresponding to these pages. ⑤ The disk memory transfer engine transfers the disk block data to the processor memory based on the data offset information and the bus address of the target memory page. ⑥ The processor starts the calculation and obtains the results of the decompression and filtering calculations. ⑦ The calculation results are transmitted to the host side. ⑧ Based on the calculation results, the host side calls the filemap_get_pages function for the page containing the filtering results. It finds that the file has extended information and that this page has been allocated to the processor side memory. At this time, memory is allocated from the host side, and the corresponding 4KB memory page is created. The corresponding 4KB memory page is extracted from the processor decompression page and transmitted to the host side.

[0076] Exemplarily, the pointer address can be used to implement mutual search between the first radix tree and the second radix tree, and the communication interface controller can accurately obtain the calculated data after the processor completes the decompression and filtering processes.

[0077] Using a scenario example, the process of transferring compressed data to the processor is explained using a maximum data transfer bandwidth of 1MB. The host's read range is expanded and aligned to 1MB, corresponding to a 1MB processor memory block. The bus address of the processor memory block is used as the target address for direct memory transfer. The physical address of the processor memory block on the host is obtained through the structured page array. This physical address is then converted to a first bus address, which is then passed to the disk to initiate a direct memory transfer. Since both the disk and the processor are mounted on the bus, the target bus address for the disk's direct memory transfer is under the same bus controller as the disk. The bus controller will choose to send the data directly to the processor. After receiving the bus message, the processor uses an internal bus bridge to convert it into an access to the processor's memory. Based on the above implementation, compressed data does not need to be forwarded through the host processor, enabling point-to-point transmission between the disk and the processor, significantly reducing transmission latency. Based on the calculated data, a 4KB memory page on the host is allocated, the data is copied from the processor to the target page, and the page is inserted into the first radix tree. Determine which pages in the processor-side memory contain row data that meets the conditions, apply for 4KB of memory in batches for these pages, and transfer the data of these pages from the processor to the host side.

[0078] In this feasible implementation, in related technologies, data is transferred based on disk block size (e.g., 2KB), and each data transfer can only transfer 2KB of data, requiring a large number of data transfers. This application utilizes the first radix tree and the second radix tree to work together to manage data transfers between the disk and the processor based on the maximum data transfer bandwidth (e.g., 1MB), thereby reducing the number of data transfers and improving data query efficiency.

[0079] In a feasible implementation, if the data query request does not include a filter condition and the extended information does not include a compression attribute, the communication interface controller is used to call the data reading interface of the disk controller to send the data in the disk to the user end.

[0080] Next, combine Figure 6 Describes direct reading and writing of data.

[0081] Figure 6 This is a schematic diagram of directly reading and writing data provided by an embodiment of the present application. Figure 6 As shown, for non-database table files with low compression, such as image files like MP4 and JPEG, no query statements or other filtering operations are required, and data can be read and written directly from disk. For highly compressed non-database table files, such as text files like TXT and DOC, only compression or decompression is performed during data reading and writing. For highly compressed database table files, only compression or decompression and filtering operations are performed during data reading and writing.

[0082] In related technologies, all data are compressed indiscriminately, while compression operations on data with low compression ratios consume high resources in exchange for low storage benefits. Compressing data with low compression ratios increases overhead, leading to low data query efficiency.

[0083] Combined with the scenario example, for scenarios that do not require decompression and filtering, there is no need to use a processor with high computing performance. The processor can be omitted and the communication interface controller can directly read data through the disk controller.

[0084] In this feasible implementation, whether the data is suitable for compression or decompression processing can be accurately determined based on the extended attributes, and the data can be accurately compressed or decompressed, thereby improving data query efficiency.

[0085] A feasible implementation method is that the communication interface controller is used to add a direct read flag to the data query request and call the data read interface of the disk controller to send the data in the disk to the user end; the disk controller is used to disable the cache mechanism of the disk controller when the communication interface controller calls the data read interface of the disk controller.

[0086] Exemplarily, the disk controller includes a cache mechanism, which is used to read data from the disk and store it in the cache of the disk controller for processing. The reading process involves data copying, which results in additional overhead.

[0087] For example, for data that does not need to be processed, the direct read flag is used to instruct the disk controller's cache mechanism not to execute, and the data is directly read to the processor through the disk controller's interface. The data does not need to be copied to the disk controller's cache, thereby reducing overhead.

[0088] In this feasible implementation, the flag bit is directly read to avoid the processor's memory from performing disk fetch operations, and the data is directly read to the processor, thereby reducing overhead and improving data query performance.

[0089] A feasible implementation method is as follows: the calculation data includes multiple pages and multiple bitmap data, the multiple pages include multiple row data, and the multiple bitmap data indicate whether each row data meets the filtering conditions; the communication interface controller is used to filter the multiple pages according to the multiple bitmap data to obtain multiple valid pages, and store the multiple valid pages in the host memory, and the bitmap data corresponding to each valid page includes a preset value; the communication interface controller is used to determine multiple valid data from the multiple row data according to the multiple bitmap data; the communication interface controller is also used to generate a calculation result mapping table according to the multiple valid data, and store the calculation result mapping table in the host memory; the communication interface controller is also used to generate target data according to the calculation result mapping table, and send the target data to the user end.

[0090] Exemplarily, the target data is data that is logically continuous but physically discontinuous.

[0091] Next, combine Figure 7 The calculation result mapping table is described.

[0092] Figure 7 Schematic diagram of the calculation result mapping table provided in the embodiment of this application. Figure 7 As shown, the processor executes the decompression processor to obtain the database table page, taking 8KB units as a page as an example, the maximum number of rows of each 8KB page does not exceed 256 rows, and each page is filtered separately to obtain the bitmap corresponding to the page. Multiple positions in the bitmap correspond one-to-one to multiple positions on the page. For example, the preset value is 1, indicating that the data at the corresponding position meets the filtering conditions. The position of bitmap data 1 in the bitmap corresponds to the position of the page that meets the filtering conditions, and the position of bitmap data 0 in the bitmap corresponds to the position of the page that does not meet the filtering conditions. Only the host side memory is applied for the valid page, and the other pages do not include the data that meets the filtering conditions, and the host side memory is not applied for the other pages. The valid page and the bitmap data are sent to the communication interface controller. The communication interface controller determines the position of the data that meets the filtering conditions in the first radix tree based on the bitmap and the bitmap data, and constructs a calculation result mapping table based on the position of the data that meets the filtering conditions. The mapping table can accurately determine the target data that meets the filtering conditions from the memory.

[0093] Combined with the scenario example, the host memory is a temporary storage. The host memory has higher read and write performance than the disk. When using data, the data is read into the memory, and the user end directly uses the data in the memory. The data stored in the disk is discrete storage, and the data that meets the filter conditions and the data that does not meet the filter conditions are discretely stored. The data read from the disk includes data that does not meet the filter conditions. The data that does not meet the filter conditions will still occupy the memory, resulting in invalid occupation of the memory space. The communication interface controller of this application determines the location of the data that meets the filter conditions based on the bitmap and the bitmap data, and performs mapping processing to obtain a calculation result mapping table. The data in the calculation result mapping table is logically continuous target data. The pages that meet the filter conditions are stored in the host side memory, and the pages that do not meet the filter conditions do not apply for the host side memory, which can effectively reduce memory usage.

[0094] Optionally, in the communication interface controller, each process opens a corresponding file handle. A calculation result mapping table is added to the file handle to remap data access based on the filter calculation result, so that read operations only access data that meets the filter conditions.

[0095] In this feasible implementation method, the location of data that meets the filtering conditions can be accurately determined through bitmap data, so as to accurately generate a calculation result mapping table. Logically continuous target data is generated through the calculation result mapping table. Only pages that meet the filtering conditions are stored in the host side's memory, and pages that do not meet the filtering conditions do not apply for host side memory usage, thereby effectively reducing the memory occupation of data that does not meet the filtering conditions and improving data query performance.

[0096] A feasible implementation method is that the communication interface controller is used to determine the first data quantity corresponding to multiple row data and determine the second data quantity of valid data; the communication interface controller is also used to determine a first selection ratio based on the second data quantity and the first data quantity; the communication interface controller is also used to generate a second calculation result mapping table based on the tree data structure and multiple valid data if the first selection ratio is greater than or equal to a first preset value; the communication interface controller is also used to generate a second calculation result mapping table based on the array data structure and multiple valid data if the first selection ratio is less than the first preset value.

[0097] For example, if the first selection ratio is greater than or equal to the first preset value, corresponding to high selection ratio data, more valid data that meets the filtering conditions will be found. If the first selection ratio is less than the first preset value, corresponding to low selection ratio data, less valid data that meets the filtering conditions will be found.

[0098] Exemplary, reference Figure 7For scenarios with high selection ratios (i.e., most data meets the filtering conditions), the calculation result mapping table uses a B+ tree data structure to extract the location and length of the row data that meets the filtering conditions from the database storage page, and the nodes on the B+ tree point to the row data that meets the filtering conditions. The leaf nodes of the B+ tree form an ordered linked list, which supports fast range traversal. In scenarios with high selection ratios, range queries are highly efficient. For scenarios with low selection ratios (i.e., most data does not meet the filtering conditions), the calculation result mapping table uses an array data structure to extract the row data that meets the filtering conditions from the database storage page, and copy and splice it into the array. The array directly locates data through offsets, without the need for hierarchical jumps in the tree structure. The single-point access speed is high, and in scenarios with low selection ratios, the latency is low and the reading efficiency is high.

[0099] In this feasible implementation, by selecting a ratio to determine the corresponding data structure, the data query efficiency can be effectively improved.

[0100] A feasible implementation method is that the communication interface controller is used to send a time window to the processor; the processor is used to reorganize multiple high-frequency pages within the time window to obtain a reorganized page, the selection ratio of each high-frequency page is lower than a second preset value, and each high-frequency page includes high-frequency row data; the processor is also used to send the reorganized page to the communication interface controller.

[0101] Exemplarily, a high-frequency page is a page that appears frequently and has a low selection ratio, that is, a page that is frequently accessed by the user terminal but contains less valid data. High-frequency row data is valid data.

[0102] Combined with the scenario example, for high-frequency pages, frequent appearance and low selection ratio indicate that it is necessary to repeatedly extract the pages and filter them to obtain valid data, resulting in a large consumption of computing resources.

[0103] Illustratively, the time window is used to determine a time period, and the frequency of page access is determined by the time period.

[0104] In this feasible implementation, high-frequency pages are reorganized to merge valid data, reduce computing resource overhead, and improve performance when extracting pages.

[0105] A feasible implementation method is that the communication interface controller is used to add a reorganized page to a calculation result mapping table; the communication interface controller is also used to determine multiple redundant data in the calculation result mapping table based on the reorganized page, and the multiple redundant data are repeated with the data in the reorganized page; the communication interface controller is also used to delete the multiple redundant data from the calculation result mapping table; the communication interface controller is also used to generate reorganized data based on the calculation result mapping table from which the multiple redundant data are deleted, and send the reorganized data to the user end.

[0106] For example, the bitmap formed after filtering calculations are performed on each page is accumulated bit by bit in the processor to obtain an accumulated matrix. Row records with larger values in the accumulated matrix can be determined as high-frequency row records. For high-frequency row records, a merge calculation unit can be added to the processor. For pages containing high-frequency row records, which have a low selection ratio, the high-frequency row records are merged into a reorganized page, and the original row records are marked invalid. At this time, the processor can output calculation results in two formats, one is a row record bitmap that meets the filtering conditions. The other is a reorganized page containing high-frequency row records. After receiving the reorganized page, the communication interface controller adds the reorganized page to the end of the table file and starts a process to perform a cleanup operation to clean up redundant data.

[0107] In this feasible implementation, by adding the reorganized pages to the calculation result mapping table and deleting redundant data, the amount of valid data in the calculation result mapping table can be increased, and the interference of redundant data on data query can be reduced, thereby improving the efficiency of data query.

[0108] In a feasible implementation, the processor is configured to convert the physical address space of the host through an address converter to obtain a second bus address; the processor is further configured to send calculation data to the communication interface controller through the second bus address.

[0109] Among them, the physical address space is used to map the processor's memory.

[0110] Exemplarily, a bus bridge is added within the processor to convert the bus address access to the internal interconnect bus, so that the bus address access is converted into the access to the processor memory.

[0111] Next, combine Figure 8 Describes memory access.

[0112] Figure 8 This is a schematic diagram of memory access provided by an embodiment of the present application. Figure 8 As shown, access between the physical address space and the bus address space is established through the second bus address, and the communication interface controller can directly access the memory of the processor through the bus bridge of the processor.

[0113] Combined with the scenario example, the processor's memory can be directly accessed through the second bus address without the need for multiple copies of data.

[0114] In this feasible implementation, by directly accessing the processor's memory, resource overhead can be reduced and data query performance can be improved.

[0115] A feasible implementation method is that the communication interface controller is used to determine the data transmission performance value between the current host and the processor; the communication interface controller is also used to determine the target block size based on the data transmission performance value; the communication interface controller is also used to read the current address space of the processor; the communication interface controller is also used to block the current address space according to the target block size to obtain the target address space, and create a structure page array of the target address space.

[0116] Exemplarily, the data transfer performance value is the optimal performance of direct memory transfer. Direct memory transfer is a hardware mechanism that allows the processor to directly access the memory without the need for byte-by-byte intervention by the central processing unit. Its core goal is to reduce the dependence of data transfer on the central processing unit, thereby improving the overall efficiency of the system.

[0117] Exemplary, reference Figure 8 Memory within the processor's memory address range is divided into blocks based on the optimal memory size for direct memory transfers, creating corresponding structure page arrays. Each structure page array logically corresponds to multiple consecutive 4KB address spaces on the host side. For example, if a direct memory transfer is 1MB in size and direct memory transfer performance reaches saturation, the processor will divide the memory into 1MB blocks, with each 1MB corresponding to a structure page array.

[0118] Using example scenarios to illustrate, direct memory transfer performance saturation typically means the transfer rate has reached the upper limit of the hardware bandwidth and cannot be further increased. By partitioning direct memory transfers by size, the direct memory transfer controller can operate at maximum efficiency, reducing the overhead of frequently starting and stopping transfers.

[0119] In this feasible implementation, block division is performed according to the optimal performance of direct memory transfer, which can optimize bus utilization and reduce latency, thereby improving data query efficiency.

[0120] A feasible implementation method is that the communication interface controller is used to add a first field in the structure page array, and the value of the first field is used to indicate whether the data at the corresponding position of the processor's memory and the host's memory are synchronized; the communication interface controller is also used to add a second field in the structure page array, and the value of the second field is used to indicate the computing unit supported by the processor; the communication interface controller is also used to add a third field in the structure page array, and the value of the third field is used to indicate multiple associated index values corresponding to multiple associated memory blocks in the processor.

[0121] Exemplary, reference Figure 8, flag corresponds to the first field. For example, after a 4KB page on the host side is modified, not only is a data inconsistency mark added with the disk, but the flag mark in the memory structure page array on the processor side is also modified, indicating that the current host side page data is inconsistent with the processing side page data. When the page is dropped to the disk or when the processor performs calculations on the page, the host side page needs to be transferred to the processor side. mapping corresponds to the second field. The mapping field has 32 bits, each bit represents a computing unit, and each computing unit is used to perform corresponding computing processing, such as decompression processing or filtering processing, and can support 32 different computing units. _mapcount corresponds to the third field. The processor requires 1MB of compressed data memory, 1MB of decompressed data memory, and 1MB of filtering result memory in one calculation. The associated page index can form a linked list to facilitate quick finding of the pages required for the calculation.

[0122] Combined with the scenario example, the processor's memory is only used for hardware acceleration, and the types of calculations it can support are fixed and its purpose is relatively simple. Some fields in the structure page array structure have original purposes, so some fields can be redefined and used.

[0123] In this feasible implementation, by adding different fields to the structure page array, data query can be assisted from multiple dimensions, thereby improving the efficiency of data query.

[0124] A feasible implementation method is as follows: a communication interface controller is used to determine the correspondence between the address range in the host address space and the memory controller; the communication interface controller is also used to create a first memory access table based on the correspondence; the communication interface controller is also used to determine the delay of each computing unit in the processor accessing each memory controller; the communication interface controller is also used to create a second memory access table based on the delay of each computing unit accessing each memory controller; the communication interface controller is also used to send the first memory access table and the second memory access table to the processor, and the processor is used to determine the memory controller corresponding to the extended information based on the first memory access table and the second memory access table; the processor is also used to decompress and filter the compressed data through the memory controller to obtain the calculated data.

[0125] Next, combine Figure 9 Describes memory access.

[0126] Figure 9 This is a schematic diagram of memory access provided by an embodiment of the present application. Figure 9As shown in the figure, the processor has six compute units: Clean 0, Compression 0, Compression 1, Decompression 0, Filter 0, and Filter 1. These units have six independently accessible memory controllers: DDR0, DDR1, DDR2, HBM0, HBM1, and HBM2. Each compute unit has access to a different memory controller, and each memory controller controls access to its corresponding memory. This means that each compute unit has access to a different memory range and experiences different access latencies.

[0127] For example, the first memory access table can be used to accurately determine the address range corresponding to each memory controller. By establishing a second memory access table to count multiple delays, the memory controller used for decompression or filtering can be determined based on the delays.

[0128] Combined with the scene example, refer to Figure 8 The first memory access table records the memory controller numbers corresponding to different address ranges, and the second memory access table records the computing units supported by different address ranges and the latency of the computing units accessing the memory controller. When it is a negative value, it means that it cannot be accessed.

[0129] Optionally, the first memory access table and the second memory access table may be loaded into a processor, and an acceleration operator in the processor may be used to quickly find an optimal memory controller.

[0130] In this feasible implementation, the first memory access table and the second memory access table can be combined to accurately apply for memory in the processor, thereby improving the accuracy of data query.

[0131] A feasible implementation method is that the communication interface controller receives a data write request sent by the user end, and the data write request includes write data; the communication interface controller is also used to send the write data to the processor according to the data write request; the processor cleans and processes the write data to obtain valid write data; the communication interface controller is also used to call the data write interface of the disk controller through point-to-point transmission to store the valid write data on the disk.

[0132] Next, combine Figure 10 Data writing is explained.

[0133] Figure 10 This is a schematic diagram of data writing provided in the embodiment of the present application. Figure 10As shown, the communication interface controller invokes a write operation based on a write request. This write operation, along with decompression and filtering, is performed simultaneously to retrieve the database page that needs to be written to disk from the user. Data is sent from the host to the processor via direct memory transfer. The processor performs parallel cleanup calculations on invalid rows in the page, removing them and merging valid rows together to form a cleaned page. The processor initiates compression calculations to compress the cleaned page to produce a compressed page. The processor then transmits the compressed page to disk via point-to-point transmission, invoking the disk controller's interface for a write operation.

[0134] Combined with the scenario example, a large amount of invalid data will be generated during the addition, deletion and modification operations of the database. For example, the modification of a database field will not modify the content of the field at the original location, but mark the row record at the original location as invalid, and insert the updated new record at the end of the file. If the data is written directly to the disk, a large amount of invalid data will gradually accumulate on the disk. An independent process is required to regularly clean up the invalid data, merge the valid data together, and delete the original page. The process performing the cleanup operation will occupy the resources on the host side and reduce the performance of the host side. The present application can reduce the resource usage on the host side by performing cleanup and compression processing on the processor while writing the data.

[0135] In this feasible implementation, cleaning and compression are performed while data is being written, which can reduce resource usage on the host side and thus improve the performance of the host side.

[0136] Figure 11 A flow chart of the data query method provided in the embodiment of the present application is shown as follows: Figure 11 As shown, the method includes the following steps:

[0137] S201. Based on a data query request sent by a user terminal, the compressed data that needs to be decompressed and filtered is determined from the disk by calling the data reading interface of the disk controller, and the compressed data is transmitted to the processor so that the processor decompresses and filters the compressed data to obtain calculated data.

[0138] Exemplarily, the communication interface controller on the host side is responsible for controlling the execution of data query requests to obtain calculation data.

[0139] Exemplarily, the data query request includes information for locating the position of the compressed data, which may be data offset information. The data offset information is used to locate the offset position of the compressed data in the disk.

[0140] Exemplarily, the data query request may include a filtering condition. When the compressed data is transmitted to the processor, the filtering condition is also sent to the processor. The processor decompresses the compressed data to obtain decompressed data. The processor filters the decompressed data according to the filtering condition to obtain calculated data that meets the filtering condition.

[0141] Exemplarily, the extended information of the compressed data is stored in the disk, and the compression properties of the compressed data are stored in the extended information. The extended information is sent to the processor so that the processor can clearly understand whether decompression processing is required and how to perform the decompression processing, thereby improving the accuracy of the decompression processing.

[0142] S202: Receive calculation data sent by the processor, and send the calculation data to the user end.

[0143] Exemplarily, the calculated data is mapped to obtain logically continuous target data, and the target data is sent to the user end.

[0144] Combined with the scenario example, the host memory is a temporary storage. The host memory has higher read and write performance than the disk. When using data, the data is read into the memory, and the user end directly uses the data in the memory. The data stored in the disk is discrete storage, and the data that meets the filter conditions and the data that does not meet the filter conditions are stored discretely. The data read from the disk includes data that does not meet the filter conditions. The data that does not meet the filter conditions will still occupy the memory, resulting in invalid occupation of the memory space. The communication interface controller of this application determines the location of the data that meets the filter conditions based on the bitmap and bitmap data, and performs mapping processing to obtain a calculation result mapping table. The data in the calculation result mapping table is logically continuous target data, which can effectively reduce memory usage.

[0145] Based on the above implementation, hardware acceleration processing is performed by the processor, which can reduce the resource usage on the host side, and the decompression and filtering processing during the data query process is performed by a dedicated processor, thereby improving the efficiency of data query.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0147] Figure 12 This is a schematic diagram of the structure of the data query device provided in the embodiment of the present application. Figure 12 As shown, the embodiment of the present application further provides a data query device, the data query device 120 may include: an execution module 121 and a sending module 122, wherein:

[0148] The execution module 121 is used to determine the compressed data that needs to be decompressed and filtered from the disk by calling the data reading interface of the disk controller according to the data query request sent by the user terminal, and transmit the compressed data to the processor so that the processor decompresses and filters the compressed data to obtain calculated data.

[0149] The sending module 122 is used to send the calculated data to the user terminal.

[0150] Optionally, the execution module 121 may execute Figure 11 S201 in the embodiment.

[0151] Optionally, the sending module 122 may execute Figure 11 S202 in the embodiment.

[0152] It should be noted that the data query device shown in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0153] Figure 13 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 13 As shown, the electronic device 130 provided in this embodiment includes: at least one processor 1301 and a memory 1302. Optionally, the electronic device 130 further includes a communication component 1303. The processor 1301, the memory 1302 and the communication component 1303 are connected via a bus.

[0154] During the specific implementation process, at least one processor 1301 executes the computer-executable instructions stored in the memory 1302, so that the at least one processor 1301 executes the above-mentioned data query method embodiment.

[0155] The specific implementation process of the processor 1301 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0156] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0157] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0158] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0159] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned data query method embodiments when running.

[0160] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0161] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above data query method embodiments are implemented.

[0162] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data query method embodiments are implemented.

[0163] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0164] The above is a detailed introduction to a data query system provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data query system, characterized in that: include: Communication interface controller, processor, disk controller, user terminal, and disk, wherein, The user terminal is used to send a data query request to the communication interface controller; The communication interface controller is configured to determine, based on the data query request, compressed data that needs to be decompressed and filtered from the disk by calling the data reading interface of the disk controller, and transmit the compressed data to the memory of the processor; the communication interface controller is further configured to establish a pointer address, the pointer address being used to implement mutual search between a first radix tree and a second radix tree, the first radix tree being used to manage the disk location of the host, and the second radix tree being used to manage the memory of the processor; The processor is configured to decompress and filter the compressed data to obtain calculated data, and the processor is further configured to send the calculated data to the communication interface controller, wherein the calculated data includes a plurality of pages and a plurality of bitmap data, the plurality of pages including a plurality of row data, and the plurality of bitmap data indicating whether each row data meets a filtering condition; The communication interface controller is configured to filter the plurality of pages according to the plurality of bitmap data to obtain a plurality of valid pages, and store the plurality of valid pages in a host memory, wherein the bitmap data corresponding to each valid page includes a preset value; the communication interface controller is configured to determine a plurality of valid data from the row data of the plurality of valid pages according to the plurality of bitmap data; the communication interface controller is further configured to generate a calculation result mapping table according to the positions of the plurality of valid data determined in the first radix tree, and store the calculation result mapping table in the host memory; the communication interface controller is further configured to generate target data according to the calculation result mapping table; The communication interface controller is further configured to send the target data to the user terminal.

2. The data query system according to claim 1, characterized in that: The data query request includes data offset information; The communication interface controller is configured to determine a first disk position according to the data offset information and a first radix tree of the host; The communication interface controller is further configured to call a data reading interface of the disk controller to read the compressed data and the extended information corresponding to the compressed data from the disk according to the first disk position.

3. The data query system according to claim 2, characterized in that: The data query request further includes a filtering condition, and the extended information includes a compression attribute of the compressed data; The communication interface controller is used to send the compressed data, the extended information, and the filtering condition to the processor in a point-to-point transmission manner; The processor is configured to decompress the compressed data according to the compression attribute to obtain decompressed data, and filter the decompressed data according to the filtering condition to obtain the computing data communication interface controller.

4. The data query system according to claim 3, characterized in that: The communication interface controller is configured to determine a plurality of index values from a structure page array of the processor; The communication interface controller is further configured to determine a target memory block from the processor based on the filtering condition, the extended information, and the multiple index values, and send the filtering condition to the target memory block, where the target memory block is configured to implement filtering processing corresponding to the filtering condition and decompression processing corresponding to the extended information; The communication interface controller is further configured to determine a first bus address corresponding to the target memory block through the structure page array; The communication interface controller is further configured to establish a bus connection between the disk and the target memory block via the first bus address; The disk is connected via the bus and sends the compressed data and the extended information to the target memory block.

5. The data query system according to claim 4, characterized in that: The communication interface controller is configured to insert the target memory block into a second radix tree of the processor; The communication interface controller is further configured to establish a pointer address between the first radix tree and the second radix tree; The communication interface controller and the processor search for data with each other through the pointer address.

6. The data query system according to claim 3, characterized in that: If the data query request does not include a filtering condition, and the extended information does not include a compression attribute, the communication interface controller is configured to call a data reading interface of the disk controller to send the data in the disk to the user end.

7. The data query system according to claim 6, characterized in that: The communication interface controller is used to add a direct read flag to the data query request, and call the data read interface of the disk controller to send the data in the disk to the user end; The disk controller is used to disable the cache mechanism of the disk controller when the communication interface controller calls the data reading interface of the disk controller.

8. The data query system according to claim 1, characterized in that: The communication interface controller is used to determine the first data quantity corresponding to the plurality of rows of data and determine the second data quantity of the valid data; The communication interface controller is further configured to determine a first selection ratio based on the second data quantity and the first data quantity; The communication interface controller is further configured to generate a second calculation result mapping table based on the tree data structure and the plurality of valid data if the first selection ratio is greater than or equal to a first preset value; The communication interface controller is further configured to generate the second calculation result mapping table according to the array data structure and the plurality of valid data if the first selection ratio is less than the first preset value.

9. The data query system according to claim 8, characterized in that: The communication interface controller is used to send a time window to the processor; The processor is configured to reorganize the plurality of high-frequency pages within the time window to obtain reorganized pages, wherein the selection ratio of each high-frequency page is lower than a second preset value, and each high-frequency page includes high-frequency row data; The processor is further configured to send the reorganized page to the communication interface controller.

10. The data query system according to claim 9, characterized in that: The communication interface controller is used to add the reorganized page to the calculation result mapping table; The communication interface controller is further configured to determine, based on the reorganized page, a plurality of redundant data in the calculation result mapping table, the plurality of redundant data being duplicated with data in the reorganized page; The communication interface controller is further configured to delete the plurality of redundant data from the calculation result mapping table; The communication interface controller is further configured to generate reorganized data according to a calculation result mapping table of deleting the plurality of redundant data, and send the reorganized data to the user terminal.

11. The data query system according to claim 1, characterized in that: The processor is configured to convert the physical address space of the host through an address converter to obtain a second bus address; The processor is further configured to send the calculated data to the communication interface controller via the second bus address.

12. The data query system according to claim 11, characterized in that: The communication interface controller is used to determine the current data transmission performance value between the host and the processor; The communication interface controller is further configured to determine a target block size according to the data transmission performance value; The communication interface controller is further configured to read the current address space of the processor; The communication interface controller is further configured to perform block processing on the current address space according to the target block size to obtain a target address space, and to create a structure page array of the target address space.

13. The data query system according to claim 12, characterized in that: The communication interface controller is used to add a first field in the structure page array, wherein the value of the first field is used to indicate whether data at corresponding positions of the memory of the processor and the memory of the host are synchronized; The communication interface controller is further configured to add a second field to the structure page array, wherein a value of the second field is used to indicate a computing unit supported by the processor; The communication interface controller is further configured to add a third field to the structure page array, wherein a value of the third field is configured to represent a plurality of associated index values corresponding to a plurality of associated memory blocks in the processor.

14. The data query system according to claim 13, characterized in that: The communication interface controller is used to determine the correspondence between the address range in the host address space and the memory controller; The communication interface controller is further configured to create a first memory access table according to the corresponding relationship; The communication interface controller is further configured to determine a latency for each computing unit in the processor to access each memory controller; The communication interface controller is further configured to create a second memory access table according to a delay of each computing unit accessing each memory controller; The communication interface controller is further configured to send the first memory access table and the second memory access table to the processor; The processor is configured to determine a memory controller corresponding to the extended information according to the first memory access table and the second memory access table; The processor is further configured to perform decompression and filtering processing on the compressed data through the memory controller to obtain the calculated data.

15. The data query system according to claim 1, characterized in that: The communication interface controller receives a data write request sent by the user terminal, wherein the data write request includes write data; The communication interface controller is further configured to send the write data to the processor according to the data write request; The processor cleans up the written data to obtain valid written data; The communication interface controller is further configured to call the data writing interface of the disk controller in a point-to-point transmission manner, and store the valid written data in the processor in the disk.

16. A data query method, characterized in that: include: Based on a data query request sent by a user terminal, calling a data reading interface of a disk controller to determine compressed data that needs to be decompressed and filtered from the disk, and transmitting the compressed data to a memory of a processor, so that the processor decompresses and filters the compressed data to obtain calculated data, the calculated data including a plurality of pages and a plurality of bitmap data, the plurality of pages including a plurality of rows of data, and the plurality of bitmap data indicating whether each row of data meets a filtering condition; Establishing a pointer address, wherein the pointer address is used to implement mutual search between a first radix tree and a second radix tree, wherein the first radix tree is used to manage a disk location of a host, and the second radix tree is used to manage a memory of the processor; receiving the calculation data sent by the processor, filtering the plurality of pages according to the plurality of bitmap data to obtain a plurality of valid pages, and storing the plurality of valid pages in a host memory, wherein the bitmap data corresponding to each valid page includes a preset value; and determining a plurality of valid data from the row data of the plurality of valid pages according to the plurality of bitmap data; generating a calculation result mapping table according to the positions of the plurality of valid data determined in the first radix tree, and storing the calculation result mapping table in the host memory; Generate target data according to the calculation result mapping table, and send the calculated data to the user terminal.

17. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data query method according to claim 16 when executing the computer program.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the data query method according to claim 16 when executed by a processor.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data query method according to claim 16 are implemented.

Images