Data processing method

Abstract

The embodiment of the invention provides a data processing method. The method comprises the steps that a data query request sent by a second data processor is received; according to the data query request, performing data reading and processing on initial data of a data storage unit to obtain target data and a data query table corresponding to the target data; the target data and the data query table are stored in the data storage unit, and a query ending instruction is sent to the second data processor, so that the second data processor obtains the data query table according to the query ending instruction, and obtains the target data according to the data query table. And determining whether to acquire the target data from the data storage unit. Through the method, the second data processor obtains the data query table, so that the data migration amount is effectively reduced, the problem of causing read-write competition is reduced, and the workload of a data migration tool DMA and the read-write competition in a data storage unit are reduced.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular, to a data processing method. Background Art

[0002] In big data scenarios, CPU (Central Processing Unit) is usually used for data processing (such as data query). The CPU has a high operating frequency and flexible design, but the operating mode of a single-core CPU is serial operation with low parallelism. If the CPU is used to perform SQL (Structured Query Language) operator operations, a large amount of CPU resources are used in the process of scanning data blocks, resulting in precious CPU resources being unable to perform other operations. For this reason, the prior art provides a new heterogeneous computing platform of CPU plus FPGA (Field Programmable Gate Array) for data processing.

[0003] However, when the new heterogeneous computing platform of CPU plus FPGA performs data processing, it is necessary to frequently move a large amount of data between the Host side where the CPU is located and the Device side where the FPGA is located. This can easily lead to read and write competition problems, causing the I / O (Input / Output) bandwidth to reach a bottleneck, increasing the workload of the data movement tool DMA and the read and write competition in the data storage unit DDR. Summary of the invention

[0004] In view of this, the embodiments of this specification provide two data processing methods. One or more embodiments of this specification simultaneously involve two data processing devices, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided a data processing method, which is applied to a first data processor, comprising:

[0006] receiving a data query request sent by a second data processor;

[0007] According to the data query request, the initial data of the data storage unit is read and processed to obtain the target data and the data query table corresponding to the target data;

[0008] The target data and the data query table are stored in the data storage unit, and a query end instruction is sent to the second data processor, so that the second data processor obtains the data query table according to the query end instruction, and determines whether to obtain the target data from the data storage unit according to the data query table.

[0009] According to a second aspect of an embodiment of this specification, a data processing method is provided, which is applied to a second data processor, including:

[0010] In response to a query end instruction sent by the first data processor, the data query table is acquired, wherein the query end instruction is that the first data processor reads and processes the initial data of the data storage unit according to the data query request, obtains the target data and the data query table corresponding to the target data, and stores the target data and the data query table corresponding to the target data in the data storage unit before sending;

[0011] According to the data query table, it is determined whether to obtain the target data from the data storage unit.

[0012] According to a third aspect of an embodiment of this specification, there is provided a data processing device, applied to a first data processor, comprising:

[0013] a request receiving module, configured to receive a data query request sent by the second data processor;

[0014] A data acquisition module is configured to read and process the initial data of the data storage unit according to the data query request, and obtain the target data and a data query table corresponding to the target data;

[0015] The instruction sending module is configured to store the target data and the data query table in the data storage unit, and send a query end instruction to the second data processor, so that the second data processor obtains the data query table according to the query end instruction, and determines whether to obtain the target data from the data storage unit according to the data query table.

[0016] According to a fourth aspect of an embodiment of this specification, a data processing device is provided, which is applied to a second data processor, including:

[0017] A query table acquisition module is configured to acquire a data query table in response to a query end instruction sent by the first data processor, wherein the query end instruction is that the first data processor will read and process the initial data of the data storage unit according to the data query request, obtain the target data and the data query table corresponding to the target data, and store the target data and the data query table corresponding to the target data in the data storage unit before sending;

[0018] The acquisition determination module is configured to determine whether to acquire the target data from the data storage unit according to the data query table.

[0019] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0020] Memory and processor;

[0021] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above data processing method are implemented.

[0022] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data processing method are implemented.

[0023] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0024] A data processing method provided by an embodiment of the present specification receives a data query request sent by a second data processor; reads and processes the initial data of a data storage unit according to the data query request to obtain target data and a data query table corresponding to the target data; stores the target data and the data query table in the data storage unit, and sends a query end instruction to the second data processor, so that the second data processor obtains the data query table according to the query end instruction, and determines whether to obtain the target data from the data storage unit according to the data query table.

[0025] Specifically, the data processing method saves resources of the second data processor and reduces latency by offloading the original computing tasks that consume a lot of energy on the second data processor to the first data processor; obtains target data and a data query table corresponding to the target data by reading and processing the initial data of the data storage unit in the first data processor; so that the second data processor can obtain the data query table without moving the target data after each data processing, and only needs to move the data query table, which effectively reduces the amount of data moved and reduces the problem of read-write contention. According to the data query table, it is determined whether to obtain the target data from the data storage unit. By analyzing the data query table, only valid target data is obtained, which reduces the workload of the data moving tool and the read-write contention in the data storage unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a specific processing diagram of a data processing method provided by an embodiment of this specification;

[0027] Figure 2 is a flow chart of a data processing method provided by one embodiment of this specification applied to a first data processor;

[0028] Figure 3 It is a structural schematic diagram of a data query table in a data processing method provided by an embodiment of this specification;

[0029] Figure 4 It is a schematic diagram of a data generation process in a data processing method provided by an embodiment of this specification;

[0030] Figure 5 is a schematic diagram of the structure of a first data processor provided by an embodiment of this specification;

[0031] Figure 6 is a flow chart of a data processing method provided by one embodiment of this specification applied to a second data processor;

[0032] Figure 7 is a flowchart of a specific processing process of a data processing method provided by an embodiment of this specification applied to a first data processor;

[0033] Figure 8 is a flowchart of a specific processing process of a data processing method provided by an embodiment of this specification applied to a second data processor;

[0034] Fig. 9 is a structural schematic diagram of a data processing device provided by an embodiment of this specification applied to a first data processor;

[0035] Fig.10is a structural schematic diagram of a data processing device provided by an embodiment of this specification applied to a second data processor;

[0036] Fig.11 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0037] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0038] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0040] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0041] First, the terms involved in one or more embodiments of this specification are explained.

[0042] SQL: Structured Query Language, a structured query language, is a database query and programming language used to access data and query, update, and manage relational database systems.

[0043] CPU: Central Processing Unit, the central processing unit, as the operation and control core of a computer system, is the final execution unit for information processing and program running.

[0044] FPGA: Field Programmable Gate Array, a programmable gate array. In the embodiments of this specification, the FPGA is heterogeneous acceleration hardware and is a further development product based on programmable devices such as PAL (Programmable Array Logic) and GAL (Generic Array Logic). It appears as a semi-custom circuit in the field of application-specific integrated circuits (ASICs), which not only solves the deficiencies of custom circuits but also overcomes the drawback of limited gate circuits in the original programmable devices.

[0045] DDR: Double Data Rate, DDR SDRAM: Double Data Rate Synchronous Dynamic Random Access Memory.

[0046] PCIe: PCI-Express (peripheral component interconnect express) is a high-speed serial computer expansion bus standard. PCIe belongs to high-speed serial point-to-point dual-channel high-bandwidth transmission.

[0047] DMA: Direct Memory Access. In the embodiments of this specification, the DMA is a component for data transfer and can transfer data in the CPU memory to the DDR of the FPGA. The DMA is a data transfer tool based on the PCIe interface.

[0048] Driver: A driver program is a program that interacts with a specific physical device (or specific software).

[0049] I / O: Input / Output, input / output.

[0050] Range operator: Represents a range and can calculate various integers and floating-point numbers, such as income data, dates, etc. The result is a range like 100 < a < 1000.

[0051] Like operator: Fuzzy query, for example, retrieving a certain field (such as a person's name) from a text.

[0052] Shift-Or algorithm: It is a bit operation algorithm based on converting the pattern string into a bit vector (called a mask) to detect whether there is a match in the text string.

[0053] The host side refers to the part executed on the CPU, and the device side refers to the part executed on the FPGA. Usually, the host side program will prepare the data and copy it to the memory of the graphics card, and then the FPGA will execute the device side program to process the data and obtain the data processing results. After completion, the host side program will retrieve the data processing results from the memory of the graphics card. This data processing process frequently moves data between the host side where the CPU is located and the device side where the FPGA is located, which can easily lead to IO bandwidth bottlenecks.

[0054] Based on this, an embodiment of the present specification provides a data processing method, which can obtain target data and a data query table corresponding to the target data after processing the data, so that a second data processor can obtain the data query table. There is no need to move the target data after each data processing, and only the data query table needs to be moved, which effectively reduces the amount of data moved and reduces the problem of read and write contention. According to the data query table, it is determined whether to obtain the target data from the data storage unit. By analyzing the data query table, only valid target data is obtained, which reduces the workload of the data moving tool and the read and write contention in the data storage unit, and improves the performance of the entire data processing process.

[0055] Specifically, this data processing method can be applied to cache nodes in the acceleration of big data in-path computing to achieve near-storage acceleration. The cache node will offload SQL operators that require the CPU to perform a large number of operations to the FPGA for heterogeneous acceleration. This FPGA-based hardware and software system can be implemented through the reasonable hardwareization of SQL basic operators, reasonable planning of host and device data, statistical information aggregation of data processing results, and multi-level lookup table design, which can effectively reduce the CPU load in the computing node, while greatly improving the overall system bandwidth, reducing system latency, and breaking through the CPU performance bottleneck.

[0056] In this specification, two data processing methods are provided. This specification also involves two data processing devices, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0057] See also Figure 1 , Figure 1 A specific processing schematic diagram of a data processing method provided according to an embodiment of the present specification is shown.

[0058] The data processing method provided in the embodiment of this specification is applied to Figure 1 The data processing architecture shown in is used as an example.

[0059] The data processing architecture includes a host side and a device side. The core of the host side is a central processing unit (CPU), and the core of the device side is a field programmable gate array (FPGA).

[0060] Among them, the central processing unit CPU is the host end, which is mainly responsible for data parsing and control operations. The central processing unit CPU will pull files that need to be operated by SQL from various external storage devices, such as SSD (Solid-State Drive), OSS (Object Storage Service), etc. and store them in the memory, and preliminarily parse the files into data blocks. After the parsing is completed, the control command is sent to the data migration tool to orderly send the data blocks through the PCIe interface and the data migration tool DMA to the data storage unit DDR on the programmable gate array FPGA board. At the same time, the central processing unit CPU will also be responsible for parsing and sending the configuration information sent by the upper-layer application to the programmable gate array FPGA for the first query operator (such as range operator) / second query operator (such as Like operator) and other operations. After the programmable gate array FPGA completes the processing, it will send an end signal to the central processing unit CPU, and the central processing unit CPU will read the processing results back from the data storage unit DDR in the programmable gate array FPGA board, and further process the results.

[0061] The programmable gate array FPGA is the device end, which is mainly responsible for complex calculation operations on data blocks. The data moving tool DMA receives the control command sent by the central processing unit CPU and moves the data in the memory to the data storage DDR. After receiving the start command sent by the central processing unit CPU, the programmable gate array FPGA will read and calculate the data from the data storage unit DDR according to the read data starting address, read data length and other information pre-configured by the central processing unit CPU. After the programmable gate array FPGA completes the calculation, it will send a completion command to the central processing unit CPU to notify it to read the processing results back for subsequent processing.

[0062] During specific implementation, the central processing unit CPU sends a control command through the driver Driver, the data moving tool DMA receives the control command from the central processing unit CPU and starts moving the data blocks, and the data moving tool DMA moves the data blocks in order through the PCIe interface and the data moving tool DMA to the data storage unit DDR of the programmable gate array FPGA; the programmable gate array FPGA reads and processes the data blocks, obtains the processing results, writes the processed data blocks back to the data storage unit DDR, and the data moving tool DMA returns the processing results to the driver Driver of the central processing unit CPU.

[0063] In actual applications, the CPU sends a control command, and the data moving tool DMA moves the data block from the memory to the DDR according to the CPU's control command. The CPU sends a data query request, and the FPGA reads the data by reading the data start address, the data length and other information according to the data query request, and queries the data through the first query operator / the second query operator. The FPGA writes the queried data block back to the DDR. After processing the data block, the programmable gate array FPGA will perform an additional data write operation on the DDR. The data is the aggregate information of the processed data block, which is regarded as a padding data block; the FPGA processes a data block to be processed, generates a result data block and an additional padding data block, and the previous The padding data block of a data block and the padding data block of the next data block are connected head to tail on the DDR; after the FPGA operation is completed, it will send a completion command to the CPU to notify it to read back the result, and the CPU will send a control end command. After receiving the control end command from the CPU, the DMA will move the large data block composed of multiple padding data blocks from the DDR to the memory at one time; the CPU will parse this group of padding data blocks as a whole, and during the parsing process, the CPU will confirm whether the result data block on the DDR is a valid data block, that is, its internal information contains a valid processing result, such as whether the condition is hit or not; if not, the CPU will directly skip the reading of the data block, parse the padding data block information of the next data block, and determine whether to retrieve the result data block from the DDR.

[0064] Based on this, the data processing method provided in the embodiment of this specification adds an FPGA to offload the original CPU-intensive computing tasks to the FPGA, saving CPU resources and reducing latency. The CPU obtains a set of padding data blocks without having to move the data after each processing, which can effectively reduce the amount of data movement in big data scenarios, reduce the workload on the DMA side and the read and write competition on the DDR side.

[0065] The following combination Figure 2 , the data processing method is further described. Figure 2A flow chart showing a data processing method provided by one embodiment of the present specification applied to a first data processor is shown, which specifically includes the following steps.

[0066] Step 202: Receive a data query request sent by a second data processor.

[0067] Among them, the first data processor can be understood as a processor that can process data, such as FPGA, ASIC (Application Specific Integrated Circuit), CPU, DPU (multi-core CPU based on industry standards, high performance and software programmable), etc.; the second data processor can be understood as a processor that can process data, such as FPGA, ASIC, CPU, DPU, etc.

[0068] Specifically, the first data processor is applied to the Device side, and the second data processor is applied to the Host side. For ease of understanding, the embodiments of this specification take the first data processor as FPGA and the second data processor as CPU as an example to describe the data processing method in detail.

[0069] When the data processing method is applied to a database query acceleration scenario, the data query request can be understood as a request for an SQL query on data, such as a request to search for a person named abb in a novel, a request to search for income data in the range of 100 to 200, etc.

[0070] For ease of understanding, the following embodiments all apply the data processing method to a database query acceleration scenario, where the data query request is a request to search for a person named abb in a novel, and are described in detail.

[0071] Then, receiving the data query request sent by the second data processor can be understood as FPGA receiving a request sent by the CPU to search for a person named abb in a certain novel.

[0072] In one or more embodiments of this specification, in order to facilitate offloading the data processing tasks of the CPU to the FPGA to reduce the burden on the CPU, the specific implementation method is as follows:

[0073] Before receiving the data query request sent by the second data processor, the method includes:

[0074] The data migration controller determines initial data that is moved from the memory to the data storage unit based on the data migration instruction sent by the second data processor.

[0075] Among them, the data movement controller can be understood as a controller that can move data, such as DMA, DMA2D (with DMA function, in addition to color filling, image copying and other functions); the data movement instruction can be understood as an instruction to move data; the initial data can be understood as the CPU pulling the files that need to perform SQL operations from various external storage devices such as SSD, OSS, etc. and storing them in the memory, and performing preliminary parsing of the files. Data blocks.

[0076] Taking the data transfer controller as DMA as an example, a detailed introduction is given.

[0077] Then, determining that the data movement controller moves initial data from the memory to the data storage unit based on the data movement instruction sent by the second data processor can be understood as determining that the DMA moves data blocks from the memory to the DDR based on the data movement instruction sent by the CPU.

[0078] In actual applications, the DMA receives a data movement instruction from the CPU, starts data block movement, and moves the data block in the memory to the DDR, so that the first data processor can subsequently read and process the data block on the DDR.

[0079] The data processing method provided in the embodiment of this specification moves the data block from the memory to the data storage unit of the first processor through the data mover, thereby offloading the data processing task of the CPU to the first data processor and reducing the burden on the CPU.

[0080] Step 204: According to the data query request, the initial data of the data storage unit is read and processed to obtain the target data and a data query table corresponding to the target data.

[0081] Among them, the data storage unit can be understood as a memory module that can store data, which can be a data storage unit within the FPGA or an external storage unit, which is connected to the FPGA through an interface for data storage, such as DRAM (Dynamic RAM), DDR, etc.; the initial data can be understood as a data block moved from the memory to the data storage unit; the target data can be understood as the data after reading and processing the data in the data storage unit; the data query table corresponding to the target data can be understood as a query table after processing the data in the target data, such as a query table after aggregating the data in the target data, and the data query table contains aggregation information of the target data, including but not limited to the maximum value Max, minimum value Min, count Count, sum Sum and other aggregation information of the target data.

[0082] Taking a data query request of searching for a person named abb in a novel and a data storage unit of DDR as an example, a detailed description is given.

[0083] Then, according to the data query request, the initial data of the data storage unit is read and processed to obtain the target data and the data query table corresponding to the target data. It can be understood that according to the request to find a person named abb in a certain novel, the data in the certain novel stored in the DDR is read and processed to obtain the data in the certain novel and the data query table corresponding to the data.

[0084] In one or more embodiments of this specification, data exists in the form of data blocks, so as to facilitate subsequent rapid processing of each data block, and the specific implementation method is as follows:

[0085] The method of reading and processing the initial data of the data storage unit according to the data query request to obtain the target data and the data query table corresponding to the target data includes:

[0086] According to the data query request, data is read and processed on at least two initial data blocks of the data storage unit to obtain at least two target data blocks and a data query table corresponding to each of the at least two target data blocks.

[0087] Still taking the data query request as a request to search for the name abb in a novel and the data storage unit as DDR as an example, a detailed description is given.

[0088] Then, according to the data query request, at least two initial data blocks of the data storage unit are read and processed to obtain at least two target data blocks and a data query table corresponding to each of the at least two target data blocks. It can be understood that according to a request to find a person named abb in a certain novel, the field data in the certain novel stored in the DDR is read and processed to obtain the field data in the certain novel and a data query table corresponding to the field data.

[0089] In the data processing method provided in the embodiment of this specification, data exists in the form of data blocks, so as to facilitate subsequent rapid processing of each data block and obtain a data query table corresponding to each data block.

[0090] In one or more embodiments of the present specification, each target data block and the data query table corresponding to each target data block are stored in a reasonable order, so that the subsequent CPU can determine whether to obtain the target data in order by analyzing the data query table, thereby improving the processing efficiency. The specific implementation method is as follows:

[0091] The storing the target data and the data query table in the data storage unit comprises:

[0092] Determining a sequence relationship of each target data block in the at least two target data blocks according to a data reading sequence;

[0093] According to the sequence relationship of the target data blocks, the target data blocks and the data query tables corresponding to the target data blocks are stored in the data storage unit.

[0094] The data reading order can be understood as the reading order of sequentially reading each data block; the sequential relationship can be understood as the order in which the first-read data block is processed first.

[0095] Then, according to the data reading order, determining the order relationship of each target data block in the at least two target data blocks;

[0096] According to the sequential relationship of the target data blocks, the target data blocks and the data query tables corresponding to the target data blocks are stored in the data storage unit. It can be understood that according to the reading order of sequentially reading the data blocks, the data blocks read first are first processed and stored in the DDR, and the sequential relationship between the target data blocks is determined. According to the sequential relationship, the target data blocks and the data query tables corresponding to the target data blocks are stored in the DDR.

[0097] The data processing method provided in the embodiment of this specification stores each target data block and the data query table corresponding to each target data block in a data storage unit in a reasonable order, so that the subsequent CPU can determine whether to obtain the target data in order by analyzing the data query table, thereby improving processing efficiency.

[0098] In one or more embodiments of the present specification, data aggregation is performed on the target data by a preset aggregation method to accurately generate a data query table corresponding to the target data, so that the second data processor can quickly determine whether to obtain the target data from the data storage unit according to the data query table. The specific implementation method is as follows:

[0099] The method of reading and processing at least two initial data blocks of the data storage unit according to the data query request to obtain at least two target data blocks and a data query table corresponding to each of the at least two target data blocks includes:

[0100] According to the data query request, at least two initial data blocks of the data storage unit are read to obtain at least two read data blocks, and the at least two read data blocks are processed to obtain at least two target data blocks;

[0101] Aggregating data of each target data block in the at least two target data blocks according to a preset aggregation method to obtain aggregated data of each target data block;

[0102] The aggregated data of each target data block is processed according to a preset query structure to generate a data query table corresponding to each target data block.

[0103] Among them, the read data block can be understood as the data block to be queried in the data storage unit; the target data block can be understood as the data block after the queried data is queried; the preset aggregation method can be understood as a pre-set aggregation method, such as an aggregation method for the maximum value Max, minimum value Min, count Count, etc. of the target data block.

[0104] Aggregate data can be understood as, taking a data query request for a character named abb in a novel as an example, recording the data block containing the field data of abb as a valid data block, setting this data block to "1", and using the Count aggregation method to count the number of data containing the field data of abb in the data block, which is the aggregate data.

[0105] The preset query structure can be understood as a query structure having a primary index, a secondary index, and aggregated statistical information, wherein the statistical granularity of the aggregated statistical information is the aggregated data within the primary index.

[0106] Then, according to the data query request, data is read from at least two initial data blocks of the data storage unit to obtain at least two read data blocks, and the at least two read data blocks are processed to obtain at least two target data blocks. It can be understood that according to the data query request, data is read from multiple initial data of the data storage unit to obtain multiple data blocks to be queried, and the multiple data blocks to be queried are processed to obtain multiple data blocks after the data blocks to be queried are queried.

[0107] In practical applications, taking a data query request for a character named abb in a novel as an example, data aggregation is performed on each of the at least two target data blocks according to a preset aggregation method to obtain aggregated data of each target data block. This can be understood as performing data aggregation on each of the multiple queried data blocks, recording the data block containing the field data of abb as a valid data block, setting this data block to "1", and counting the number of data blocks containing the field data of abb through the Count aggregation method.

[0108] Taking a data query request for querying income data in the range of 100 to 200 as an example, data aggregation is performed on each of the at least two target data blocks according to a preset aggregation method to obtain aggregated data of each target data block. It can be understood that data aggregation is performed on each of the multiple queried data blocks, and the data blocks containing income data in the range of 100 to 200 are recorded as valid data blocks. The MAX aggregation method can be used. For example, if the maximum value in a data block is 80, the valid data blocks in the multiple data blocks after the query can be determined by directly skipping the data block.

[0109] The aggregated data of each target data block is processed according to a preset query structure to generate a data query table corresponding to each target data block. This can be understood as processing the aggregated data of each queried data block according to a query structure with a primary index, a secondary index, and aggregated statistical information to generate a data query table corresponding to each queried data block.

[0110] The data processing method provided in the embodiments of this specification aggregates data for each target data block through a preset aggregation method, and accurately generates a data query table corresponding to each target data block, so that the second data processor can quickly determine whether to obtain the target data from the data storage unit based on the data query table.

[0111] In one or more embodiments of this specification, a data query table corresponding to the target data is reasonably generated by presetting the query structure, so that the subsequent second data processor can quickly process the data according to the structure in the data query table. The specific implementation method is as follows:

[0112] The processing of the aggregated data of each target data block according to the preset query structure to generate a data query table corresponding to each target data block includes:

[0113] According to the secondary index query structure, the statistical data of each data row in the target data block corresponding to the aggregated data is determined as the first index layer, and the statistical data corresponding to each data byte in each data row is determined as the second index layer;

[0114] Determine the query data corresponding to the valid statistical data in the second index layer and the location information of the corresponding target data block;

[0115] Generate a data query table corresponding to each target data block according to the aggregated data, the first index layer, the second index layer, and the location information;

[0116] The query data is data queried from the corresponding target data block according to the data query request.

[0117] Among them, the secondary index structure can be understood as an index structure including a first index layer and a second index layer; a data row can be understood as a place where data is stored in a data block, and a data block can contain multiple data rows; statistical data can be understood as data set to "0" or "1" in the above embodiment; data bytes can be understood as data bytes in a data row, and a row of data can contain multiple data bytes; location information can be understood as location information of query data in its corresponding data block; query data can be understood as real result data.

[0118] See also Figure 3 , Figure 3 A schematic diagram of the structure of a data query table in a data processing method provided by an embodiment of the present specification is shown.

[0119] First, after the Data is processed, the Result data block and the Padding data block are obtained.

[0120] Specifically, based on the characteristics of FPGA hardware parallel processing, bypass statistics and other operations can be performed on the data stream being processed without increasing data delay to generate Result data and Padding data.

[0121] See also Figure 4 , Figure 4 A schematic diagram of a data generation process in a data processing method provided in one embodiment of the present specification is shown.

[0122] The input data stream is sent to the processing unit in the first data processor, and the Result data block is obtained through the processing unit, and the Padding data block is obtained through the statistical unit of the first data processor in the bypass.

[0123] The input data stream may be understood as the read initial data in the above embodiment.

[0124] Among them, Data can be understood as the initial data in the above embodiment; Result data block can be understood as the target data block in the above embodiment; Padding data block can be understood as the data query table in the above embodiment.

[0125] Figure 3 The Padding data group in the dotted box is the additional information after the FPGA processes a data block each time. The additional information of multiple data blocks will be combined and read by the CPU through DMA at one time.

[0126] As can be seen from the figure, the Padding data block is stored in the form of a lookup index table. In the schematic diagram, the LookupTable 0 to Lookup Table N part is the first-level Lookup table of the data block, which can be considered as a primary index, that is, the first index layer in the above embodiment, and the Lookup Table 0_0 to Lookup Table 1_m part is a secondary index, that is, the second index layer in the above embodiment. At the same time, aggregate statistical information such as LT_0_Max-LT_N_Sum will be added at the end of the file, and its statistical granularity is the data statistical information within the primary index, that is, the aggregate data of the above embodiment. As shown in the figure, the total length of each Lookup Table is i Byte, and the total length of each Lookup Table x_index (subscript) is k Byte.

[0127] It can be seen from the structural diagram of the data query table that the data query table is stored in the form of a search index table, so that the subsequent CPU can quickly determine whether to obtain target data from the data storage unit based on this structure.

[0128] Still taking the data query request for a character named abb in a novel as an example, a data block a contains data rows 1, 2, and 3, data row 1 contains data bytes 1 and 2, data row 2 contains data bytes 3 and 4, and data row 3 contains data bytes 5 and 6; after querying the data block a, it is found that data byte 2 in data row 1 of the data block a contains abb, and data byte 5 in data row 3 contains abb, then data rows 1 and 3 are set to "1", and data row 2 is set to "0"; and data byte 2 in data row 1 and data byte 5 in data row 3 that specifically contain abb are set to "1", and other data bytes are set to "0"; and the Count aggregation is used to calculate the Count_a data of the data block a to be 2.

[0129] Then, based on the above embodiment, according to the secondary index query structure, the statistical data of each data row in the target data block corresponding to the aggregated data is determined as the first index layer, and the statistical data corresponding to each data byte in each data row is determined as the second index layer. It can be understood that in the data block a corresponding to the Count_a data, the "1", "0", "1" corresponding to the data rows 1, 2, and 3 are determined as the first index layer, and the "0", "1", "0", "0", "1", "0" corresponding to the data bytes 1, 2, 3, 4, 5, and 6 are determined as the second index layer;

[0130] Determining the query data corresponding to the valid statistical data in the second index layer and the position information of the corresponding target data block can be understood as determining the real result data corresponding to the data set to "1" in the second index layer and the position information of the corresponding data block, such as determining the real result data corresponding to the data set to "1" and the position information of the corresponding data block a is address 2 and address 5;

[0131] According to the aggregate data, the first index layer, the second index layer and the location information, a data query table corresponding to each target data block is generated. It can be understood that according to the aggregate data Count_a, the first index layer, the second index layer and the location information, a data query table corresponding to data block a is generated.

[0132] The data processing method provided in this specification reasonably generates a data query table corresponding to the target data through a preset query structure, so that a subsequent second data processor can quickly perform data processing according to the structure in the data query table.

[0133] In one or more embodiments of this specification, data is read accurately according to the data reading start address and the data reading length, and the read data is processed reasonably according to the data query operator. The specific implementation method is as follows:

[0134] The data query request carries a data reading start address and a data reading length;

[0135] The method of reading data from at least two initial data blocks of a data storage unit according to the data query request to obtain at least two read data blocks, and processing the at least two read data blocks to obtain at least two target data blocks includes:

[0136] According to the data reading start address and the data reading length carried in the data query request, read data from at least two initial data blocks of the data storage unit to obtain at least two read data blocks;

[0137] The at least two read data blocks are processed according to a data query operator corresponding to the data query request to obtain at least two target data blocks.

[0138] Among them, the data query operator can be understood as an operator for querying data, such as the Range operator, the Like operator, etc.; the data query operator corresponding to the data query request can be understood as, if the data query request is a query request for a certain name of a person in a novel, then the data query operator corresponding to the data query request is the Like operator; if the data query request is a request for querying income data within the range of 100 to 200, then the data query operator corresponding to the data query request is the Range operator.

[0139] Then, according to the data reading start address and the data reading length carried in the data query request, data is read from at least two initial data blocks of the data storage unit to obtain at least two read data blocks. It can be understood that according to the data reading start address and the data reading length carried in the data query request, data is read from multiple data blocks of the data storage unit to obtain multiple data blocks to be queried in the data storage unit.

[0140] According to the data query operator corresponding to the data query request, the at least two read data blocks are processed to obtain at least two target data blocks. It can be understood that according to the data query operator corresponding to the data query request, multiple data blocks to be queried are queried to obtain multiple data blocks after the data to be queried is queried.

[0141] Still taking the data query request for a query request for a character named abb in a novel as an example, a detailed description is given.

[0142] For example, the query start address is to start reading from address 2, the reading length is 5 bytes, and data is read from multiple data blocks of the data storage unit to obtain the data blocks to be queried; when the data query request is a query request for a character named abb in a novel, the data query operator corresponding to the query request is the Like operator, and multiple data blocks to be queried are queried through the Like operator to obtain multiple data blocks after the query of the data to be queried.

[0143] The data processing method provided in the embodiments of this specification accurately reads data according to the data reading start address and the data reading length, and reasonably processes the read data according to the data query operator.

[0144] Step 206: Store the target data and the data query table in the data storage unit, and send a query end instruction to the second data processor, so that the second data processor obtains the data query table according to the query end instruction, and determines whether to obtain the target data from the data storage unit according to the data query table.

[0145] The query end instruction may be understood as an instruction for ending the query of the query data.

[0146] Then, the target data and the data query table are stored in the data storage unit, and a query end instruction is sent to the second data processor, so that the second data processor obtains the data query table according to the query end instruction, and determines whether to obtain the target data from the data storage unit according to the data query table. It can be understood that the data after the query of the data to be queried and its corresponding data query table are stored in the DDR, and an instruction for ending the query of the data to be queried is sent to the CPU, so that the CPU obtains the data query table according to the instruction, and determines whether to obtain the data after the query of the data to be queried from the DDR according to the data query table.

[0147] In one or more embodiments of this specification, in order to enable the second data processor to obtain the data query table, and then quickly determine whether to obtain the target data from the data storage unit according to the data query table, the specific implementation method is as follows:

[0148] The step of sending a query end instruction to the second data processor so that the second data processor acquires the data query table according to the query end instruction, and determines whether to acquire the target data from the data storage unit according to the data query table, comprises:

[0149] A query end instruction is sent to the second data processor, so that the second data processor uses a data migration controller to move the data query table to the memory according to the query end instruction, and determines whether to obtain the target data from the data storage unit based on processing the data query table in the memory.

[0150] In a specific implementation, a query end instruction is sent to the second data processor, so that the second data processor uses a data move controller to move the data query table to the memory according to the query end instruction, and determines whether to obtain the target data from the data storage unit based on the processing of the data query table in the memory. This can be understood as sending a query end instruction to the CPU, so that the CPU uses DMA to move the data query table to the memory according to the query end instruction, and determines whether to obtain the data after querying the data to be queried from the DDR based on the processing of the data query table in the memory.

[0151] The data processing method provided in this specification sends a query end instruction to the second data processor, so that the second data processor obtains a data query table, and then quickly determines whether to obtain target data from the data storage unit according to the data query table.

[0152] The first data processor includes a control unit and a reading unit;

[0153] Accordingly, the receiving of the data query request sent by the second data processor includes:

[0154] The control unit receives a data query request sent by the second data processor, and sends the data query request to the reading unit.

[0155] The first data processor may be understood as the FPGA in the above embodiment.

[0156] Specifically, the control module receives the data query request sent by the CPU, and sends the data query request to the reading module.

[0157] The control unit in the first data processor can receive the data query request sent by the second data processor and reasonably send the data query request to the reading unit.

[0158] The first data processor further includes a selection unit and a processing unit;

[0159] Correspondingly, the step of reading data from at least two initial data blocks of a data storage unit according to the data query request to obtain at least two read data blocks, and processing the at least two read data blocks to obtain at least two target data blocks includes:

[0160] The reading unit receives the data query request, reads data from at least two initial data blocks of the data storage unit, obtains at least two read data blocks, and sends a data reading completion instruction to the control unit;

[0161] The processing unit, in response to the data processing instruction sent by the control unit, processes the at least two read data blocks to obtain at least two target data blocks, and sends a data processing completion instruction to the control unit, wherein the data processing instruction is sent by the control unit according to the data reading completion instruction;

[0162] Correspondingly, performing data aggregation on each of the at least two target data blocks according to a preset aggregation mode to obtain aggregated data of each target data block includes:

[0163] The selection unit, in response to the data aggregation instruction sent by the control unit, performs data aggregation on each of the at least two target data blocks according to a preset aggregation method to obtain aggregated data of each target data block, wherein the data aggregation instruction is sent by the control unit according to the data processing completion instruction;

[0164] Correspondingly, the aggregated data of each target data block is processed according to the preset query structure to generate a data query table corresponding to each target data block, including:

[0165] The selection unit processes the aggregated data of each target data block according to a preset query structure, generates a data query table corresponding to each target data block, and sends a data aggregation completion instruction to the control unit.

[0166] For the specific implementation, please refer to the above embodiments, which will not be described in detail here.

[0167] Through the reading, processing, and selection units in the first data processor, data can be quickly read and processed, and data aggregation operations can be performed on the data to generate a data query table, so that the subsequent second data processor can obtain target data according to the data query table.

[0168] The first data processor further includes a write-back unit;

[0169] Accordingly, storing the target data and the data query table in the data storage unit and sending a query end instruction to the second data processor includes:

[0170] The write-back unit, in response to the data storage instruction sent by the control unit, stores the target data and the data query table in the data storage unit, and sends a data storage completion instruction to the control unit, wherein the data storage instruction is sent by the control unit according to the data aggregation completion instruction;

[0171] The control unit sends a query end instruction to the second data processor according to the data storage completion instruction.

[0172] Through the write-back module in the first data processor, data can be efficiently written back to the data storage device.

[0173] The data processing method provided by the embodiment of the present description obtains target data and a data query table corresponding to the target data by reading and processing the initial data of the data storage unit in the first data processor; so that the second data processor can obtain the data query table without moving the target data after each data processing, and only the data query table needs to be moved, which effectively reduces the amount of data moved and reduces the problem of read-write contention. According to the data query table, it is determined whether to obtain the target data from the data storage unit. By analyzing the data query table, only valid target data is obtained, which reduces the workload of the data moving tool and the read-write contention in the data storage unit.

[0174] See also Figure 5 , Figure 5A schematic diagram of the structure of a first data processor provided according to an embodiment of the present specification is shown.

[0175] The first data processor includes a control unit, a reading unit, a selection unit, a processing unit, and a write-back unit.

[0176] Specifically, the control unit is responsible for performing register configuration on each sub-unit in the first data processor, such as a reading unit, a selecting unit, a processing unit, and a writing back unit.

[0177] The reading unit receives the control signal from the control unit, completes the loading of the data reading address and the reading length information, starts to read the data from the DDR, and sends the read data to the selection unit.

[0178] After receiving the control signal from the control unit, the selection unit will select the data distribution mode and data path according to the control signal. The design of this unit is mainly responsible for the data pre-integration of the Range operator and Like operator in the processing unit and the data aggregation operation after the operation is completed;

[0179] The processing unit is the core unit of the first data processor, which completes the acceleration service for the basic operators Range and Like in big data. The processing unit starts a specific operator which is selected by the control unit.

[0180] In practical applications, in the Range processing unit, operations such as size comparison and result aggregation of multiple data types are mainly implemented. The supported data types include commonly used integer 32, integer 64, floating point 32, floating point 64, etc., and the supported operations include left value and right value comparison. At the same time, multiple comparison types such as greater than, less than, not equal to, greater than or equal to, etc. can be supported. The operation code can be sent to the control unit through software to parse and configure the processing unit to realize any form of size comparison. At the same time, the processing unit can perform aggregation work such as minimum value Min, maximum value Max, count Count, sum Sum, etc. according to the results of the previous stage size comparison, which can effectively reduce the CPU calculation amount on the Host side. Therefore, the entire data processing process can be processed by FPGA in the whole process. At the same time, in order to further reduce the processing time of the CPU on the Host side, data filling is performed at the end of the data processing result of the data block, that is, the data query table in the above embodiment adds a multi-level search function.

[0181] In the Like processing unit, the Shift-Or algorithm is mainly used to implement the string matching function in the data stream. To implement a complete Like operation, the CPU needs to intervene after the FPGA operation on the Device side is completed, and perform a partial secondary match on the result. If you need to match "example", the pattern string configuration register in the Like operator will be configured to "example", and then the data stream will be read for string matching. When the match is hit, the detailed hit position in the result data stream will be set to "1". The entire data stream is processed uninterruptedly and repeatedly set to "1" after the match is hit until the data ends. After the CPU result data is read back, a secondary match will be performed to complete the precise hit of "%example%".

[0182] The write-back unit receives the control signal from the control unit and completes operations such as data write-back address and data receiving interface selection. When all data blocks of the overall processing unit have been processed, the write-back unit will send a calculation completion signal to the control unit.

[0183] Based on this, using the first data processor FPGA to push as much calculation as possible to the cache node can optimize the query speed of large data sets of analytical databases and improve the overall performance indicators of the database. Using field programmable gate array FPGA in the cache node for acceleration, data scanning can be performed without using the CPU, accelerating the query speed of scanning large data sets. At the same time, these FPGA components allow the aggregation of intermediate results in high-speed memory.

[0184] See also Figure 6 , Figure 6 A flow chart showing a data processing method provided according to an embodiment of the present specification applied to a second data processor is shown.

[0185] Step 602: In response to a query end instruction sent by the first data processor, a data query table is obtained, wherein the query end instruction is that the first data processor will read and process the initial data of the data storage unit according to the data query request, obtain the target data and the data query table corresponding to the target data, and store the target data and the data query table in the data storage unit before sending.

[0186] The first data processor may be understood as a processor that can process data, such as FPGA, ASIC (Application Specific Integrated Circuit), and the like.

[0187] When the data processing method is applied to a database query acceleration scenario, the query end instruction can be understood as an instruction to end the SQL query on the data.

[0188] Then, in response to the query end instruction sent by the first data processor, the data query table is obtained, wherein the query end instruction is that the first data processor will read and process the initial data of the data storage unit according to the data query request, obtain the target data and the data query table corresponding to the target data, and store the target data and the data query table in the data storage unit before sending. It can be understood that in response to the instruction sent by the FPGA to end the SQL query on the data, the data query table is obtained, wherein the query end instruction is that the FPGA will read and process the initial data of the DDR according to the data query request, obtain the target data and the data query table corresponding to the target data, and store the target data and the data query table in the DDR before sending.

[0189] In one or more embodiments of the present specification, a first data migration instruction is sent to a data migration controller to reasonably and quickly obtain a data query table from a memory, and the specific implementation method is as follows:

[0190] The step of acquiring the data query table in response to the query end instruction sent by the first data processor comprises:

[0191] In response to the query end instruction sent by the first data processor, sending a first data migration instruction to the data migration controller;

[0192] When it is determined that the data migration controller migrates the data query table to the memory according to the first data migration instruction, the data query table is acquired from the memory.

[0193] The first data moving instruction may be understood as an instruction for moving the data query table from the data storage unit to the memory.

[0194] Then, in response to the query end instruction sent by the first data processor, a first data move instruction is sent to the data move controller, and when it is determined that the data move controller moves the data query table to the memory according to the first data move instruction, the data query table is obtained from the memory. It can be understood that in response to the query end instruction sent by the FPGA, an instruction to move data from the data storage unit to the memory is sent to the DMA, and when it is determined that the DMA moves the data query table to the memory according to the instruction, the data query table is obtained from the memory.

[0195] In actual applications, DMA receives the first data movement instruction from the CPU, starts data block movement, and moves the large data blocks consisting of the data query tables corresponding to each data block from DDR to the memory at one time. The CPU obtains a group of large data blocks and subsequently parses the large data blocks.

[0196] The data processing method provided in the embodiment of this description sends a first data migration instruction to the data migration controller to reasonably and quickly obtain a data query table from the memory.

[0197] In one or more embodiments of the present specification, a data migration controller is used to migrate data from a memory to a data storage unit to reduce the burden of the first data processor. The specific implementation method is as follows:

[0198] Before acquiring the data query table in response to the query end instruction sent by the first data processor, the method further includes:

[0199] Sending a second data migration instruction to the data migration controller, so that the data migration controller migrates at least two initial data blocks from the memory to the data storage unit according to the second data migration instruction;

[0200] Correspondingly, the query end instruction is that the first data processor will read and process at least two initial data blocks of the data storage unit according to the data query request, obtain at least two target data blocks and a data query table corresponding to each target data block in the at least two target data blocks, and store each target data block and the data query table corresponding to each target data block in the data storage unit and then send it.

[0201] The second data moving instruction may be understood as an instruction for moving the initial data from the memory to the data storage unit.

[0202] Then, sending a second data move instruction to the data move controller so that the data move controller moves at least two initial data blocks from the memory to the data storage unit according to the second data move instruction can be understood as sending an instruction to move the initial data from the memory to the data storage unit to the DMA so that the DMA moves multiple initial data from the memory to the DDR according to the instruction.

[0203] Correspondingly, the query end instruction is that the first data processor will read and process at least two initial data blocks of the data storage unit according to the data query request, obtain at least two target data blocks and a data query table corresponding to each target data block in the at least two target data blocks, and store each target data block and the data query table corresponding to each target data block in the data storage unit and then send it. This has been explained in the above embodiment and will not be repeated here.

[0204] The data processing method provided by the embodiment of the present description utilizes a data migration controller to migrate data from a memory to a data storage unit, so as to reduce the burden of a first data processor.

[0205] Step 604: Determine whether to obtain the target data from the data storage unit according to the data query table.

[0206] In specific implementation, it is determined whether to obtain the data after querying the data to be queried from the DDR according to the data query table.

[0207] In one or more embodiments of the present specification, whether to obtain target data from a data storage unit is quickly determined based on an index query of a data query table, and the specific implementation method is as follows:

[0208] The data query table is an index table of a secondary index query structure;

[0209] Correspondingly, determining whether to obtain the target data from the data storage unit according to the data query table includes:

[0210] Parse the data query table to determine the aggregated data, the first index layer, the second index layer and the position information of the data query table, wherein the aggregated data is data aggregation performed on each of the at least two target data blocks according to a preset aggregation method, the first index layer is determined according to the statistical data of each data row in the target data block corresponding to the aggregated data, the second index layer is determined according to the statistical data corresponding to each data byte in each data row, and the position information is determined according to the query data corresponding to the valid statistical data in the second index layer, at the position of the corresponding target data block;

[0211] When it is determined that the aggregated data meets the first preset query requirement, reading statistical data of the first index layer;

[0212] When it is determined that the statistical data of the first index layer meets the second preset query requirement, reading the statistical data of the second index layer;

[0213] In the case of determining that the statistical data of the second index layer meets the third preset query requirement, determining location information of the query data corresponding to the statistical data of the second index layer in its corresponding target data block;

[0214] According to the location information, query data in the target data block corresponding to the location information is acquired from the data storage unit.

[0215] Among them, the first preset query requirement can be understood as the requirement that the aggregated data is not 0; the second preset requirement can be understood as the requirement that the data row in the data block is set to "1", that is, the requirement that the data row contains valid data; the third preset requirement can be understood as the data byte in the data row is set to "1", that is, the requirement that the data byte is valid data.

[0216] Still using the above embodiment for detailed description, the data query table is parsed, and when the aggregate data Count_a is not 0, it is determined that the corresponding data block a contains valid data, and according to the first index layer, it is determined that data rows 1 and 3 contain valid data, and according to the second index layer, it is determined that the data corresponding to data bytes 2 and 5 are valid data, and the corresponding location address is determined, and the real result data can be retrieved from the DDR according to the location address;

[0217] The data processing method provided in the embodiments of the present specification quickly determines whether to obtain target data from a data storage unit by obtaining a data query table and performing an index query on the data query table.

[0218] See also Figure 7 , Figure 7 A flowchart showing a specific processing process of a data processing method provided according to an embodiment of the present specification applied to a first data processor is shown.

[0219] Step 1: The data moving tool DMA receives a control command from the central processing unit CPU to start moving the data block, moving it from the memory to the data storage unit.

[0220] Among them, the CPU can be understood as the second data processor in the above embodiment; the control command can be understood as the data moving instruction in the above embodiment; and the data block can be understood as the initial data in the above embodiment.

[0221] Specifically, taking the data storage unit as DDR as an example, the data moving tool DMA receives the data moving instruction sent by the central processing unit CPU to start the data block moving, and moves it from the memory to the DDR.

[0222] In practical applications, data exists in the form of data blocks.

[0223] Step 2: The programmable gate array FPGA reads the data block on the data storage unit and performs specific operations.

[0224] Among them, FPGA can be understood as the first data processor in the above embodiment.

[0225] In actual applications, the FPGA reads data according to the read data start address, read data length and other information configured in advance by the CPU, and performs operations according to the first query operator / second query operator.

[0226] Step 3: The FPGA writes the processed data block back to the data storage unit.

[0227] The processed data block (Res, result data block) can be understood as the target data in the above embodiment.

[0228] In actual applications, FPGA processes the data block according to the data query operator to obtain the result data block.

[0229] Step 4: After processing the data block, the FPGA will perform an additional data write operation on the data storage unit, in which the data is the aggregate information of the result data block, which is regarded as a Padding data block.

[0230] The Padding data block can be understood as the data query table in the above embodiment, and one data block corresponds to one padding data block.

[0231] Repeat the above steps. Each time the FPGA processes a data block to be queried, it generates a result data block and an additional Padding data block. The Padding data block of the previous data block and the Padding data block of the next data block are connected head to tail on the DDR.

[0232] Specifically, it can be seen from the figure that the padding data (Block_0_padding) of the 0th data block is connected head to tail with the padding data (Block_1_padding) of the 1st data block, the padding data (Block_1_padding) of the 1st data block is connected head to tail with the padding data (Block_2_padding) of the 2nd data block, and so on, the padding data (Block_N-1_padding) of the N-1th data block is connected head to tail with the padding data (Block_N_padding) of the Nth data block.

[0233] Step 5: After repeating the above steps N times, the data moving tool DMA receives a control command from the central processing unit CPU to start moving the data blocks, and moves the large data block consisting of N padding data blocks from the data storage unit to the memory at one time.

[0234] The control command may be understood as the first data moving instruction in the above embodiment.

[0235] In actual applications, the large data block composed of Padding data blocks is moved to the memory, and the CPU will parse this group of Padding data blocks as a whole. During the parsing process, the CPU will confirm which result data block on the DDR is a valid data block, that is, its internal information contains a valid processing result, such as whether the condition is hit or not. Otherwise, the CPU will directly skip reading the data block, parse the Padding data block information of the next data block, and then determine whether to retrieve the result data block from the DDR.

[0236] Through the data processing method provided in the embodiments of this specification, a large amount of data in a big data scenario is filtered, which effectively reduces the amount of data moved, reduces the workload on the DMA side and the read and write contention on the DDR side.

[0237] See also Figure 8 , Figure 8 A flowchart showing a specific processing process of a data processing method provided by one embodiment of the present specification applied to a second data processor is shown.

[0238] Step 802: Start.

[0239] Specifically, after responding to the query end instruction sent by the first data processor, the second data processor starts data processing.

[0240] Step 804: The CPU obtains all padding data blocks.

[0241] The CPU may be understood as the second data processor in the above embodiment, and the padding data block may be understood as the data query table in the above embodiment.

[0242] Specifically, the CPU obtains a large data block composed of padding data blocks, and obtains all padding data blocks.

[0243] Step 806: Parse the padding data block.

[0244] Specifically, the CPU parses the Padding data block corresponding to a Result data block. For the Padding information of the Range / Like mode, the CPU will first read the content of LT_0_Count. If LT_0_Count=0, it will read the content of LT_1_Count until LT_x_Count!=0. For details, please refer to the first preset query requirement of the above embodiment, which will not be repeated here.

[0245] Step 808: Count=0.

[0246] Determine whether LT_0_Count is 0. If so, it means that the corresponding result data does not contain valid data, jump to step 806, and continue to parse the next padding data block; if not, it means that the corresponding result data contains valid data, jump to step 810. For details, please refer to the above embodiment, which will not be repeated here.

[0247] Step 810: Read the Result data corresponding to the padding data.

[0248] When Count is not 0, the padding data corresponding to the Result data is read. In fact, what is read is the set of the first index layer, the second index layer, and the real result data.

[0249] Step 812: Parse the Lookup Table data.

[0250] Specifically, when LT_x_Count!=0, the CPU will jump the processing pointer to the start position of the Lookup Table x data corresponding to LT_x_Count. The total length of each Lookup Table is i Byte.

[0251] Step 814: Byte i=0.

[0252] To determine whether Byte i is equal to 0, the CPU pointer will traverse each Byte in Lookup Table x. If the read Byte = 0, the next Byte data will be read; if the Byte corresponding to index 2 is 1, the CPU pointer will jump to the Lookup Table x_index (index = 2) corresponding to the Index. If Byte i = 0, jump to step 816; if Byte i is not 0, jump to step 818. For details, please refer to the second preset query requirement of the above embodiment, which will not be repeated here.

[0253] Step 816: Byte i is last.

[0254] Determine whether Byte i is the last byte. If so, it means that the traversal of this data block is completed, jump to step 806, and parse the next padding data block; if not, it means that the traversal of this data block is not completed, jump to step 810, and continue to read the Result data corresponding to the padding data.

[0255] Step 818: Parse the Lookup Table x_index data.

[0256] In step 814, when Byte i is not 0, the CPU pointer will jump to the data start address of the second index layer Lookup Table x_index, and the total length of each Lookup Table x_index is k Byte.

[0257] Step 820: index k=0.

[0258] To determine whether index k is equal to 0, the CPU pointer will traverse each Byte in the Lookup Table x. If the read Byte = 0, the next Byte data will be read; if the Byte corresponding to the subscript index 1 is 1, the CPU will perform address calculation, find the starting address in the corresponding Result data and perform data reading and analysis; if index k = 0, jump to step 822; if index k is not 0, jump to step 824. For details, please refer to the third preset query requirement of the above embodiment, which will not be repeated here.

[0259] Step 822: index k is last.

[0260] Determine whether index k is the last byte. If so, it means that the traversal of this lookup table is completed, jump to step 812, and parse the next Lookup Table data; if not, it means that the traversal of this lookup table is not completed, jump to step 818, and continue to parse the Lookup Table x_index data.

[0261] Step 824: The CPU reads the corresponding Result data to complete the analysis.

[0262] In step 820, when index k is not 0, the CPU will perform address calculation, find out the starting address in the corresponding Result data, and perform data reading and analysis.

[0263] Step 826: Last i and k.

[0264] Determine whether i and k is the last one. If so, jump to step 828; if not, jump to step 818 and continue parsing the Lookup Table x_index data.

[0265] Step 828: Single padding parsing is completed.

[0266] When i and k are the last ones, it means that a single padding parsing is completed.

[0267] Step 830: Last padding.

[0268] Determine whether this padding data block is the last one. If so, it means that the entire data traversal is completed, and jump to step 842; if not, jump to step 806 and continue to parse the next padding data block.

[0269] Step 832: End.

[0270] End the entire data processing process.

[0271] The data processing method provided in the embodiments of this specification, when applied to SQL acceleration in big data scenarios, the results of a large number of data blocks after processing will become sparse data (with less valid content, that is, the entire data block is 0 and the data is not hit). In this case, the CPU can efficiently skip the all-zero data block through the query structure of the data query table, which can effectively reduce the indexing time of the system in actual usage scenarios.

[0272] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Fig. 9 FIG. 1 shows a schematic diagram of a data processing device provided by an embodiment of the present specification applied to a first data processor. Fig. 9 As shown, the device comprises:

[0273] The request receiving module 902 is configured to receive a data query request sent by the second data processor;

[0274] The data acquisition module 904 is configured to read and process the initial data of the data storage unit according to the data query request, and obtain the target data and the data query table corresponding to the target data;

[0275] The instruction sending module 906 is configured to store the target data and the data query table in the data storage unit, and send a query end instruction to the second data processor, so that the second data processor obtains the data query table according to the query end instruction, and determines whether to obtain the target data from the data storage unit according to the data query table.

[0276] Optionally, the data acquisition module 904 is further configured to:

[0277] According to the data query request, data is read and processed on at least two initial data blocks of the data storage unit to obtain at least two target data blocks and a data query table corresponding to each of the at least two target data blocks.

[0278] Optionally, the data acquisition module 904 is further configured to:

[0279] According to the data query request, at least two initial data blocks of the data storage unit are read to obtain at least two read data blocks, and the at least two read data blocks are processed to obtain at least two target data blocks;

[0280] Aggregating data of each target data block in the at least two target data blocks according to a preset aggregation method to obtain aggregated data of each target data block;

[0281] The aggregated data of each target data block is processed according to a preset query structure to generate a data query table corresponding to each target data block.

[0282] Optionally, the data acquisition module 904 is further configured to:

[0283] According to the secondary index query structure, the statistical data of each data row in the target data block corresponding to the aggregated data is determined as the first index layer, and the statistical data corresponding to each data byte in each data row is determined as the second index layer;

[0284] Determine the query data corresponding to the valid statistical data in the second index layer and the location information of the corresponding target data block;

[0285] Generate a data query table corresponding to each target data block according to the aggregated data, the first index layer, the second index layer, and the location information;

[0286] The query data is data queried from the corresponding target data block according to the data query request.

[0287] Optionally, the data acquisition module 904 is further configured to:

[0288] According to the data reading start address and the data reading length carried in the data query request, read data from at least two initial data blocks of the data storage unit to obtain at least two read data blocks;

[0289] The at least two read data blocks are processed according to a data query operator corresponding to the data query request to obtain at least two target data blocks.

[0290] Optionally, the data acquisition module 904 is further configured to:

[0291] Determining a sequence relationship of each target data block in the at least two target data blocks according to a data reading sequence;

[0292] According to the sequence relationship of the target data blocks, the target data blocks and the data query tables corresponding to the target data blocks are stored in the data storage unit.

[0293] Optionally, the instruction sending module 906 is further configured to:

[0294] A query end instruction is sent to the second data processor, so that the second data processor uses a data migration controller to move the data query table to the memory according to the query end instruction, and determines whether to obtain the target data from the data storage unit based on processing the data query table in the memory.

[0295] The data processing device provided in the embodiment of this specification arbitrarily designs the hardware circuit in the FPGA, adds a statistical module next to the result data path in the FPGA, and can be further parsed by the CPU after the batch data block processing is completed. It can also directly skip a large number of data blocks through multi-level indexing, without the need for a full traversal operation that consumes a lot of computing power, thereby reducing the consumption of CPU resources.

[0296] Fig.10 FIG. 1 shows a schematic diagram of a data processing device provided by an embodiment of the present specification applied to a second data processor. Fig.10 As shown, the device comprises:

[0297] The query table acquisition module 1002 is configured to acquire the data query table in response to the query end instruction sent by the first data processor, wherein the query end instruction is that the first data processor will read and process the initial data of the data storage unit according to the data query request, obtain the target data and the data query table corresponding to the target data, and store the target data and the data query table in the data storage unit before sending;

[0298] The acquisition determination module 1004 is configured to determine whether to acquire the target data from the data storage unit according to the data query table.

[0299] Optionally, the device further comprises:

[0300] The data migration module is configured as follows:

[0301] Sending a second data migration instruction to the data migration controller, so that the data migration controller migrates at least two initial data blocks from the memory to the data storage unit according to the second data migration instruction;

[0302] Correspondingly, the query end instruction is that the first data processor will read and process at least two initial data blocks of the data storage unit according to the data query request, obtain at least two target data blocks and a data query table corresponding to each target data block in the at least two target data blocks, and store each target data block and the data query table corresponding to each target data block in the data storage unit and then send it.

[0303] Optionally, the query table acquisition module 1002 is further configured to:

[0304] In response to the query end instruction sent by the first data processor, sending a first data migration instruction to the data migration controller;

[0305] When it is determined that the data migration controller migrates the data query table to the memory according to the first data migration instruction, the data query table is acquired from the memory.

[0306] Optionally, the acquisition determination module 1004 is further configured to:

[0307] Parse the data query table to determine the aggregated data, the first index layer, the second index layer and the position information of the data query table, wherein the aggregated data is data aggregation performed on each of the at least two target data blocks according to a preset aggregation method, the first index layer is determined according to the statistical data of each data row in the target data block corresponding to the aggregated data, the second index layer is determined according to the statistical data corresponding to each data byte in each data row, and the position information is determined according to the query data corresponding to the valid statistical data in the second index layer, at the position of the corresponding target data block;

[0308] When it is determined that the aggregated data meets the first preset query requirement, reading statistical data of the first index layer;

[0309] When it is determined that the statistical data of the first index layer meets the second preset query requirement, reading the statistical data of the second index layer;

[0310] When it is determined that the statistical data of the second index layer meets the third preset query requirement, determining the location information of the query data corresponding to the statistical data of the second index layer in the corresponding target data block;

[0311] According to the location information, query data in the target data block corresponding to the location information is acquired from the data storage unit.

[0312] The data processing device provided in the embodiments of this specification generates additional statistical data through the operation of FPGA, so that the CPU can read the overall statistical information and then perform necessary data movement, which can effectively improve the end-to-end bandwidth of the overall system in big data scenarios while reducing the overall system latency.

[0313] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing system embodiment, since it is basically similar to the data processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the data processing method embodiment.

[0314] Fig.11 The structure block diagram of a computing device 1100 provided according to an embodiment of the present specification is shown. The components of the computing device 1100 include but are not limited to a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1140, and a database 1150 is used to store data.

[0315] The computing device 1100 also includes an access device 1140 that enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0316] In one embodiment of the present specification, the above components of the computing device 1100 and Fig.11 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.11 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0317] The computing device 1100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1100 may also be a mobile or stationary server.

[0318] The processor 1120 is used to execute the following computer executable instructions, which implement the steps of the above data processing method when executed by the processor.

[0319] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the data processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the data processing method embodiment.

[0320] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned data processing method when executed by a processor.

[0321] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the data processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the data processing method embodiment.

[0322] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data processing method.

[0323] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the data processing method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the data processing method embodiment.

[0324] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0325] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0326] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0327] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0328] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a first data processor, comprising: receiving a data query request sent by a second data processor; According to the data query request, the initial data of the data storage unit is read and processed to obtain the target data and the data query table corresponding to the target data; The target data and the data query table are stored in the data storage unit, and a query end instruction is sent to the second data processor, so that the second data processor obtains the data query table according to the query end instruction, and determines whether to obtain the target data from the data storage unit according to the data query table.

2. The data processing method according to claim 1, wherein the initial data of the data storage unit is read and processed according to the data query request to obtain the target data and the data query table corresponding to the target data, comprising: According to the data query request, at least two initial data blocks of the data storage unit are read to obtain at least two read data blocks, and the at least two read data blocks are processed to obtain at least two target data blocks; Aggregating data of each target data block in the at least two target data blocks according to a preset aggregation method to obtain aggregated data of each target data block; The aggregated data of each target data block is processed according to a preset query structure to generate a data query table corresponding to each target data block.

3. The data processing method according to claim 2, wherein the aggregated data of each target data block is processed according to a preset query structure to generate a data query table corresponding to each target data block, comprising: According to the secondary index query structure, the statistical data of each data row in the target data block corresponding to the aggregated data is determined as the first index layer, and the statistical data corresponding to each data byte in each data row is determined as the second index layer; Determine the query data corresponding to the valid statistical data in the second index layer and the location information of the corresponding target data block; Generate a data query table corresponding to each target data block according to the aggregated data, the first index layer, the second index layer, and the location information; The query data is data queried from the corresponding target data block according to the data query request.

4. The data processing method according to claim 2, wherein the data query request carries a data reading start address and a data reading length; The method of reading data from at least two initial data blocks of a data storage unit according to the data query request to obtain at least two read data blocks, and processing the at least two read data blocks to obtain at least two target data blocks includes: According to the data reading start address and the data reading length carried in the data query request, read data from at least two initial data blocks of the data storage unit to obtain at least two read data blocks; The at least two read data blocks are processed according to a data query operator corresponding to the data query request to obtain at least two target data blocks.

5. The data processing method according to claim 2, wherein storing the target data and the data query table in the data storage unit comprises: Determining a sequence relationship of each target data block in the at least two target data blocks according to a data reading sequence; According to the sequence relationship of the target data blocks, the target data blocks and the data query tables corresponding to the target data blocks are stored in the data storage unit.

6. The data processing method according to claim 1 or 5, wherein sending a query end instruction to the second data processor so that the second data processor acquires the data query table according to the query end instruction, and determines whether to acquire the target data from the data storage unit according to the data query table, comprises: A query end instruction is sent to the second data processor so that the second data processor uses a data migration controller to move the data query table to the memory according to the query end instruction, and determines whether to obtain the target data from the data storage unit based on processing the data query table in the memory.

7. The method according to claim 2, wherein the first data processor comprises a control unit and a reading unit; Accordingly, the receiving a data query request sent by the second data processor includes: The control unit receives a data query request sent by the second data processor, and sends the data query request to the reading unit.

8. The method according to claim 7, wherein the first data processor further comprises a selection unit and a processing unit; Correspondingly, the step of reading data from at least two initial data blocks of a data storage unit according to the data query request to obtain at least two read data blocks, and processing the at least two read data blocks to obtain at least two target data blocks includes: The reading unit receives the data query request, reads data from at least two initial data blocks of the data storage unit, obtains at least two read data blocks, and sends a data reading completion instruction to the control unit; The processing unit, in response to the data processing instruction sent by the control unit, processes the at least two read data blocks to obtain at least two target data blocks, and sends a data processing completion instruction to the control unit, wherein the data processing instruction is sent by the control unit according to the data reading completion instruction; Correspondingly, performing data aggregation on each of the at least two target data blocks according to a preset aggregation method to obtain aggregated data of each target data block includes: The selection unit, in response to the data aggregation instruction sent by the control unit, performs data aggregation on each of the at least two target data blocks according to a preset aggregation method to obtain aggregated data of each target data block, wherein the data aggregation instruction is sent by the control unit according to the data processing completion instruction; Accordingly, the aggregated data of each target data block is processed according to the preset query structure to generate a data query table corresponding to each target data block, including: The selection unit processes the aggregated data of each target data block according to a preset query structure, generates a data query table corresponding to each target data block, and sends a data aggregation completion instruction to the control unit.

9. The method according to claim 8, the first data processor further comprising a write-back unit; Accordingly, storing the target data and the data query table in the data storage unit and sending a query end instruction to the second data processor includes: The write-back unit, in response to the data storage instruction sent by the control unit, stores the target data and the data query table in the data storage unit, and sends a data storage completion instruction to the control unit, wherein the data storage instruction is sent by the control unit according to the data aggregation completion instruction; The control unit sends a query end instruction to the second data processor according to the data storage completion instruction.

10. A data processing method, applied to a second data processor, comprising: In response to a query end instruction sent by the first data processor, the data query table is acquired, wherein the query end instruction is that the first data processor reads and processes the initial data of the data storage unit according to the data query request, obtains the target data and the data query table corresponding to the target data, and stores the target data and the data query table in the data storage unit before sending; According to the data query table, it is determined whether to obtain the target data from the data storage unit.

11. The method according to claim 10, wherein the acquiring of the data query table in response to the query end instruction sent by the first data processor comprises: In response to the query end instruction sent by the first data processor, sending a first data migration instruction to the data migration controller; When it is determined that the data migration controller migrates the data query table to the memory according to the first data migration instruction, the data query table is acquired from the memory.

12. The method according to claim 10, before acquiring the data query table in response to the query end instruction sent by the first data processor, further comprises: Sending a second data migration instruction to the data migration controller, so that the data migration controller migrates at least two initial data blocks from the memory to the data storage unit according to the second data migration instruction; Correspondingly, the query end instruction is that the first data processor will read and process at least two initial data blocks of the data storage unit according to the data query request, obtain at least two target data blocks and a data query table corresponding to each target data block in the at least two target data blocks, and store each target data block and the data query table corresponding to each target data block in the data storage unit and then send it.

13. The method according to claim 12, wherein determining whether to obtain the target data from the data storage unit according to the data query table comprises: Parse the data query table to determine the aggregated data, the first index layer, the second index layer and the position information of the data query table, wherein the aggregated data is data aggregation performed on each of the at least two target data blocks according to a preset aggregation method, the first index layer is determined according to the statistical data of each data row in the target data block corresponding to the aggregated data, the second index layer is determined according to the statistical data corresponding to each data byte in each data row, and the position information is determined according to the query data corresponding to the valid statistical data in the second index layer, at the position of the corresponding target data block; When it is determined that the aggregated data meets the first preset query requirement, reading statistical data of the first index layer; When it is determined that the statistical data of the first index layer meets the second preset query requirement, reading the statistical data of the second index layer; In the case of determining that the statistical data of the second index layer meets the third preset query requirement, determining location information of the query data corresponding to the statistical data of the second index layer in its corresponding target data block; According to the location information, query data in the target data block corresponding to the location information is acquired from the data storage unit.

14. A data processing device, applied to a first data processor, comprising: a request receiving module, configured to receive a data query request sent by the second data processor; A data acquisition module is configured to read and process the initial data of the data storage unit according to the data query request, and obtain the target data and a data query table corresponding to the target data; The instruction sending module is configured to store the target data and the data query table in the data storage unit, and send a query end instruction to the second data processor, so that the second data processor obtains the data query table according to the query end instruction, and determines whether to obtain the target data from the data storage unit according to the data query table.

Images