A data processing method, device, storage medium and electronic device

Abstract

Embodiments of the present application provide a data processing method and device, a storage medium and an electronic device. The method is applied to a server and includes: in response to a data processing request, copying to-be-processed data in a file cache area to a target cache area, the file cache area and the target cache area being deployed in a kernel address space of an operating system; calling a data reading function in a user address space of the operating system to copy the to-be-processed data in the target cache area to an acceleration board, a virtual address of virtual memory of the user address space and a virtual address of virtual memory of the file cache area pointing to the same physical address; and controlling the acceleration board to process the to-be-processed data to obtain a processing result. Through the present application, the problem of a large number of data copying times in the process of writing data in a disk to an acceleration board in the related art is solved, thereby achieving the effect of reducing data processing time and improving heterogeneous acceleration processing performance of a database in a server.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] In the context of the digital age, database-based data storage and retrieval have become increasingly common. However, with the accumulation and aggregation of massive amounts of data, the data volume is also growing larger. To retrieve key data from databases more quickly, many heterogeneous acceleration computing methods based on databases have been proposed. In database heterogeneous acceleration computing solutions, hardware heterogeneous acceleration controllers are typically used for acceleration, such as CPUs (Central Processing Units) or FPGAs (Field Programmable Gate Arrays). In hardware heterogeneous acceleration solutions using FPGAs, when querying data in a database installed on a server, the data copying between the user address space and kernel address space in the server's operating system consumes a significant amount of query time, impacting the performance of heterogeneous database acceleration queries.

[0003] like Figure 1 The overall framework of heterogeneous database computing shown is in Figure 1 In this heterogeneous accelerated computing solution, the FPGA accelerator card acts as a hardware accelerator. The database runs in the user address space of the operating system. When the database performs a data query, it calls a data read function in the user address space and then enters the kernel address space. When the database queries some data for the first time, this data is not in the file cache of the kernel address space. DMA (Direct Memory Access) is needed to copy the data to the file cache of the kernel address space, completing the first copy. Then, in response to the data read operation, the CPU copies the data to the database buffer in the user address space, completing the second copy. Then, a data write function is called in the user address space to write the data to the DMA buffer in the kernel address space, completing the third copy. Finally, the DMA engine is started to copy the data to the FPGA accelerator card, completing the fourth copy. Therefore, in the existing heterogeneous accelerated computing solution, a single data write operation from the database to the FPGA accelerator card requires four copies. This significantly reduces the performance of heterogeneous accelerated database queries, especially for queries on large data tables in the database, where this approach incurs considerable query time costs. Summary of the Invention

[0004] This application provides a data processing method, apparatus, storage medium, and electronic device to at least solve the problem of numerous data copying operations during the process of writing data from a disk to an accelerator board in the related art.

[0005] According to one embodiment of this application, a data processing method is provided, applied in a server, comprising: responding to a data processing request, copying data to be processed in a file cache area to a target cache area, wherein the file cache area and the target cache area are deployed in the kernel address space of the operating system; calling a data read function in the user address space of the operating system to copy the data to be processed in the target cache area to an accelerator board, wherein the virtual address of the virtual memory in the user address space and the virtual address of the virtual memory in the file cache area point to the same physical address; and controlling the accelerator board to process the data to be processed to obtain a processing result.

[0006] In one exemplary embodiment, the data processing method further includes: detecting whether there is data to be processed in the file cache area before copying the data to be processed in the file cache area to the target cache area; and copying the data to be processed from the storage space of the peripheral device to the file cache area when there is no data to be processed in the file cache area.

[0007] In one exemplary embodiment, the data processing method further includes: after copying the data to be processed from the storage space of the peripheral device to the file cache area, calling a database system call function to obtain the data file address of the data to be processed in the storage space of the peripheral device; and mapping the data file address to obtain the virtual address of the file cache area.

[0008] In one exemplary embodiment, the data processing method further includes: constructing a target cache region in the driver layer of the kernel address space; determining the target data volume corresponding to the data to be processed; creating a target file identifier corresponding to the target data volume; and copying the data to be processed in the file cache region to the target cache region based on the target file identifier and the virtual address of the file cache region.

[0009] In one exemplary embodiment, the data processing method further includes: controlling a first controller in an accelerator board to acquire data to be processed and sending the data to be processed to a second controller in the accelerator board; controlling the second controller to store the data to be processed in a target memory in the accelerator board; and controlling a computing unit in the accelerator board to perform calculations on the data to be processed to obtain a processing result.

[0010] In one exemplary embodiment, the data processing method further includes: detecting whether the target object has access permission to access the data to be processed; when the target object has access permission, calling a data read function in the user address space, and starting the direct memory access driver through the data read function so that the direct memory access driver copies the data to be processed to the accelerator board.

[0011] In an exemplary embodiment, the data processing method further includes: before calling a data read function in the user address space of the operating system to copy the data to be processed in the target cache area to the acceleration board, constructing a heterogeneous accelerated computing system call interface through a hook function; obtaining the program script corresponding to the target functional unit through the heterogeneous accelerated computing system call interface, wherein the target functional unit includes at least one of the following: a data query startup unit, a physical path acquisition unit, a data read unit, a mapping unit for implementing physical address and virtual address mapping, and a direct memory access driver startup unit; compiling the program script corresponding to the target functional unit to generate a dynamic library; and deploying the dynamic library in the operating system on which the target database is installed.

[0012] According to another embodiment of this application, a data processing apparatus is provided, applied in a server, comprising: a first copy module, configured to respond to a data processing request and copy data to be processed in a file cache area to a target cache area, wherein the file cache area and the target cache area are deployed in the kernel address space of the operating system; a second copy module, configured to call a data read function in the user address space of the operating system to copy the data to be processed in the target cache area to an accelerator board, wherein the virtual address of the virtual memory in the user address space and the virtual address of the virtual memory in the file cache area point to the same physical address; and a data processing module, configured to control the accelerator board to process the data to be processed and obtain a processing result.

[0013] According to yet another embodiment of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0014] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] This application enables data sharing between the user address space and the kernel address space because the virtual address of the virtual memory in the user address space and the virtual address of the file cache region point to the same physical address. Since data sharing exists between the user address space and the kernel address space, data copying does not require copying between them, thus reducing the number of data copies during heterogeneous database computation and improving operating system performance. Furthermore, this application uses an external acceleration board to process the data, avoiding direct CPU processing and thus conserving CPU resources, further improving operating system performance.

[0016] Therefore, the solution provided in this application solves the problem of excessive data copying during the process of writing data from the disk to the acceleration board in related technologies, thereby reducing data processing time and improving the performance of heterogeneous database acceleration processing in the server. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating the overall framework of heterogeneous database computing within related technologies.

[0018] Figure 2 This is a hardware structure block diagram of a server according to an embodiment of this application;

[0019] Figure 3 This is a flowchart of a data processing method according to an embodiment of this application;

[0020] Figure 4 This is a data mapping framework diagram for heterogeneous database computing according to an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of a virtual memory model according to an embodiment of this application;

[0022] Figure 6 This is a flowchart illustrating the data flow transmission according to an embodiment of this application;

[0023] Figure 7 This is a flowchart of a data processing method according to an embodiment of this application;

[0024] Figure 8 This is a data mapping framework diagram for heterogeneous database computing according to an embodiment of this application;

[0025] Figure 9 This is a structural block diagram of a data processing apparatus according to an embodiment of this application;

[0026] Figure 10 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0029] The methods and embodiments provided in this application can be executed on a server, computer terminal, or similar computing device. Taking running on a server as an example, Figure 2 This is a hardware structure block diagram of a server for a data processing method according to an embodiment of this application. For example... Figure 2 As shown, a server may include one or more ( Figure 2 Only one is shown in the image. A processor 202 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 204 for storing data are also shown. The server may further include a transmission device 206 for communication functions and an input / output device 208. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the server described above. For example, the server may also include components that are more complex than... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0030] The memory 204 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing method in this embodiment. The processor 202 executes various functional applications and data processing by running the computer program stored in the memory 204, thus implementing the aforementioned method. The memory 204 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 204 may further include memory remotely located relative to the processor 202, and these remote memories can be connected to a server via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0031] The transmission device 206 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the server's communication provider. In one example, the transmission device 206 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 206 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0032] This embodiment provides a data processing method running on a server. Figure 3 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps:

[0033] Step S302: In response to the data processing request, copy the data to be processed in the file cache area to the target cache area.

[0034] In step S302, the file cache region and the target cache region are deployed within the kernel address space of the server's operating system. The server's operating system can be, but is not limited to, Linux. The aforementioned target cache region can be, but is not limited to, a DMA buffer.

[0035] Optional, Figure 4 This diagram illustrates the data mapping framework for heterogeneous computing in a server database, consisting of... Figure 4 It is known that the server's kernel address space contains a file cache area and a DMA buffer (i.e., the target cache area). When performing heterogeneous computations on the database, the data to be processed needs to be copied from the file cache area to the DMA buffer in the kernel address space via CPU copying.

[0036] It should be noted that, as Figure 4 As shown, if the data to be processed is the first data processed by the database on the server, the server needs to copy the data to be processed from the disk file to the file cache area through DMA copy, and then copy the data to be processed from the file cache area to the DMA buffer through CPU copy. If the data to be processed is not the first data processed by the database on the server, the server does not need to copy the data to be processed from the disk file to the file cache area through DMA copy, and can directly copy the data to be processed from the file cache area to the DMA buffer through CPU copy.

[0037] Step S304: In the user address space of the operating system, the data reading function is called to copy the data to be processed in the target cache area to the accelerator board.

[0038] In step S304, the virtual address of the virtual memory in the user address space and the virtual address of the virtual memory in the file cache region point to the same physical address. The aforementioned acceleration board can be, but is not limited to, an FPGA acceleration card, used to accelerate the computation of the data to be processed.

[0039] It's important to note that virtual memory is used for memory management in computer systems. Virtual memory provides independent memory space for each running process, creating the illusion that each process has its own independent memory. Virtual memory is merely a logical structure within the operating system; it's not a physical memory module like RAM. When a process accesses data, it is actually accessing data in physical memory or on the disk. Furthermore, virtual memory is essentially an intermediary layer between the operating system's physical memory and the processes. It hides physical memory from the processes, providing them with a simpler and easier-to-use interface, as well as more complex functionality. All modern operating systems use virtual memory, replacing physical addresses with virtual addresses.

[0040] Optional, Figure 5 A schematic diagram of a virtual memory model is shown, consisting of... Figure 5 It is understood that the user virtual memory in the user buffer area corresponding to the user address space and the kernel virtual memory in the kernel buffer area corresponding to the kernel address space are mapped to the same physical memory, while the disk is mapped to a different physical memory. Therefore, when the server performs data read operations, there is no need to copy data between the user address space and the kernel address space, reducing the number of data copies. Furthermore, in this embodiment, the virtual memory capacity can be much larger than the physical memory capacity.

[0041] In addition, the method proposed in this application can be used for heterogeneous database computing, that is, heterogeneous database computing can be achieved by switching between mmap and write.

[0042] like Figure 4As shown, the mmap method is used in the database user space to enable data sharing between the user buffer area in the user address space and the file buffer area in the kernel address space, both using the same physical address. After the disk file is copied to the file buffer area in the kernel address space, it is not necessary to copy the data to the user address space of the database installed on the server. At this point, the data to be processed only needs to be copied to the DMA buffer in the kernel address space via CPU copy. Finally, the write function is called in the user address space to initiate DMA, copying the data in the DMA buffer in the kernel address space to the FPGA board. The entire process requires only three data copies, reducing one data copy process compared to existing solutions, thereby reducing database query time and improving the performance of heterogeneous database acceleration.

[0043] Step S306: Control the acceleration board to process the data to be processed and obtain the processing result.

[0044] In step S306, after the data to be processed is copied to the accelerator board, the database installed on the server can perform calculations on the data to be processed through the accelerator board.

[0045] It should be noted that, in this embodiment, an external acceleration board is used to process the data to be processed, which avoids the problem of CPU resource consumption caused by direct data processing on the CPU, and further improves the performance of the operating system.

[0046] Based on the scheme defined in steps S302 to S306 above, it can be understood that by sharing data between the user address space and the kernel address space, in response to a data processing request, the data to be processed in the file cache area is copied to the target cache area, and the data reading function in the user address space of the operating system is called to copy the data to be processed in the target cache area to the accelerator board. Then, the accelerator board is controlled to process the data to be processed to obtain the processing result. The file cache area and the target cache area are deployed in the kernel address space of the operating system, and the virtual address of the virtual memory in the user address space and the virtual address of the virtual memory in the file cache area point to the same physical address.

[0047] It is noteworthy that, since the virtual addresses of the user address space's virtual memory and the virtual addresses of the file cache region's virtual memory point to the same physical address, this application enables data sharing between the user address space and the kernel address space. Because of this data sharing, no data copying is required between the user address space and the kernel address space during data copying, thereby reducing the number of data copies during heterogeneous database computation and improving operating system performance. Furthermore, this application uses an external acceleration board to process the data, avoiding the CPU resource consumption problem caused by direct data processing on the CPU, further improving operating system performance.

[0048] Therefore, the solution provided in this application solves the problem of excessive data copying during the process of writing data from the disk to the acceleration board in related technologies, thereby reducing data processing time and improving the performance of heterogeneous database acceleration processing in the server.

[0049] In one alternative embodiment, Figure 6 This illustrates the data stream transmission flow corresponding to the data processing method, by... Figure 6 As can be seen, when a user sends a data processing request to the server, after receiving the request, the server copies the data to be processed from the disk to the file cache area in the kernel address space. Then, it copies the data to be processed from the file cache area to the DMA buffer. Finally, DMA is initiated, and the data in the DMA buffer is copied to the FPGA board to realize the processing of the data.

[0050] It's important to note that the server only needs to copy the data to be processed from the disk to the file cache area during the first query. Subsequent queries do not require copying data from the disk again, as the data already exists in the file cache. Therefore, during database-driven data acceleration calculations, subsequent queries only require two data copies: one from the file cache area to the DMA buffer, and then from the DMA buffer to the FPGA board. In other words, before copying the data to be processed from the file cache area to the target cache area, the server checks if the data exists in the file cache area. If not, it copies the data from the storage space of an external device to the file cache area. This external device can be, but is not limited to, the disk mentioned above.

[0051] In one alternative embodiment, Figure 7 A flowchart of the data processing method provided in this embodiment is shown. The following will be combined with... Figure 7The schemes defined in steps S302 to S306 will be explained.

[0052] Optional, such as Figure 7 As shown, before copying data, the server receives a data processing request and copies the data to be processed from the disk to the file cache area. After copying the data to be processed from the disk to the file cache area, the server implements the mapping between the virtual address of the file cache area and the address of physical memory.

[0053] Specifically, after copying the data to be processed from the storage space of the peripheral device to the file cache area, the server calls the database system call function to obtain the data file address of the data to be processed in the storage space of the peripheral device, and maps the data file address to obtain the virtual address of the file cache area.

[0054] Optional, such as Figure 7 As shown, the server uses database system call functions to obtain the data file address and then uses mmap technology to map the data file address, thereby obtaining the virtual address of the file cache area. mmap technology is a method for implementing memory-mapped files.

[0055] Furthermore, such as Figure 7 As shown, after obtaining the virtual address corresponding to the file cache region, the server copies the data to be processed in the file cache region to the target cache region. Specifically, the server constructs the target cache region in the driver layer of the kernel address space and determines the target data volume corresponding to the data to be processed; then, it creates a target file identifier corresponding to the target data volume, and copies the data to be processed in the file cache region to the target cache region based on the target file identifier and the virtual address of the file cache region.

[0056] Optional, such as Figure 7 As shown, the server requests a DMA buffer in the driver layer of the kernel address space and creates target file identifiers with the same amount of target data as the data to be processed; that is, the number of target file identifiers is the same as the target data amount. Then, the server copies the data to be processed into the DMA buffer according to the target file identifiers and the virtual address of the file cache area.

[0057] Furthermore, such as Figure 7 As shown, after copying the data to be processed to the DMA buffer, the server calls the write function in the user address space to start DMA and copy the data to be processed from the DMA buffer to the FPGA board.

[0058] After the data to be processed is copied from the DMA buffer to the FPGA board, the server can control the accelerator board to process the data and obtain the processing result. Specifically, the server controls the first controller in the accelerator board to acquire the data to be processed and send it to the second controller in the accelerator board; then, the server controls the second controller to store the data to be processed in the target memory in the accelerator board and controls the computing unit in the accelerator board to perform calculations on the data to be processed to obtain the processing result.

[0059] Optional, Figure 8 A data mapping framework diagram for heterogeneous database computing in an optional server is shown. Figure 8 In this system, the server operates on Linux and uses an x86 architecture CPU as the host. The database installed on the server is PostgreSQL version 11.2. Peripheral devices include disks, specifically mechanical hard drives, and FPGA cards for acceleration. The PCIe (Peripheral Component Interconnect Express) controller (the first controller) controls data transmission on the PCIe bus, while the DDR (Double Data Rate) controller (the second controller) manages the FPGA card's memory.

[0060] Specifically, the server uses mmap technology to achieve cache sharing between the user address space and the kernel address space, eliminating the need to copy data between the two spaces. Within the kernel address space, the CPU can directly copy the data to be processed to the DMA buffer, and then the write system call function is used in the user address space to start the DMA engine (i.e., ...). Figure 8 The DMA driver in the kernel copies data from the kernel address space to the FPGA board. When data from the host is transferred to the FPGA board via the PCIe bus, the data to be processed is stored in the FPGA board's memory. After acceleration computing is started, the acceleration computing unit in the FPGA board retrieves the data to be processed from a specified address in the board's memory, performs acceleration computing, saves the calculation result at another address in the board's memory, and finally returns it to the host via the PCIe bus.

[0061] In an alternative embodiment, before calling the data read function in the operating system's user address space to copy the data to be processed in the target cache area to the accelerator card, the server can also compile the dynamic library by calling a hook function.

[0062] Specifically, the server constructs a heterogeneous accelerated computing system call interface through hook functions, and obtains the program script corresponding to the target functional unit through the heterogeneous accelerated computing system call interface; then, it compiles the program script corresponding to the target functional unit to generate a dynamic library; finally, it deploys the dynamic library on the operating system where the target database is installed.

[0063] Optionally, the aforementioned target functional unit includes at least one of the following: a data query initiation unit, a physical path acquisition unit, a data reading unit, a mapping unit for implementing physical address and virtual address mapping, and a direct memory access driver initiation unit. That is, in this embodiment, the initiation of data query, acquisition of physical path, reading of data, mapping of physical address and virtual address, and initiation of DMA are implemented through the hook function mechanism supported by the PostgreSQL database.

[0064] It should be noted that by implementing the above functionality through hook functions, corresponding API interfaces can be added to call the corresponding functional units without modifying the database source code. Then, the code corresponding to the hooked functional unit is compiled into a dynamic library (.so file), and finally, the dynamic library is added to the operating system with the database installed. Furthermore, the functions of copying data from disk to the file cache area, allocating DMA buffers, creating target file identifiers, copying data from the file cache area to the DMA buffer, and copying data from the DMA buffer to the FPGA board are all implemented through the Linux system kernel's PCIe-XDMA driver.

[0065] In one optional embodiment, during the process of calling the data read function in the operating system's user address space to copy the data to be processed from the target cache area to the accelerator card, the server can also detect whether the target object has access permission to access the data to be processed. If the target object has access permission, the server calls the data read function in the user address space, which starts the direct memory access driver so that the direct memory access driver can copy the data to be processed to the accelerator card. That is, only users with access permission can read and / or process the data.

[0066] As described above, this application provides a data processing method that enables data mapping for heterogeneous database computing. By leveraging the database hook mechanism and combining it with a virtual memory model, and employing the mmap technology principle, data sharing between the database's user address space and kernel address space is achieved. Data is directly copied to the DMA buffer via a driver in the kernel address space, and then the DMA is initiated through the write system call in the user address space, thereby copying the data to the FPGA acceleration board. This entire process reduces the number of data copies during server-accelerated queries, lowers database query time, and improves the performance indicators of heterogeneous database accelerated queries.

[0067] In addition, the solution provided in this application is applicable to big data computing-intensive industries such as finance and the Internet, and can improve the efficiency of related business operations.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0069] This embodiment also provides a data processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0070] Figure 9 This is a structural block diagram of a data processing apparatus according to an embodiment of this application, applied in a server, such as... Figure 9 As shown, the device includes: a first copy module 901, a second copy module 903, and a data processing module 905.

[0071] The first copy module 901 is used to respond to data processing requests and copy the data to be processed in the file cache area to the target cache area. The file cache area and the target cache area are deployed in the kernel address space of the operating system. The second copy module 903 is used to call the data reading function in the user address space of the operating system to copy the data to be processed in the target cache area to the accelerator board. The virtual address of the virtual memory in the user address space and the virtual address of the virtual memory in the file cache area point to the same physical address. The data processing module 905 is used to control the accelerator board to process the data to be processed and obtain the processing result.

[0072] It should be noted that the first copy module 901, the second copy module 903, and the data processing module 905 mentioned above correspond to steps S302 to S306 of the above embodiments. The three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0073] Optionally, the data processing device further includes a data detection module and a third copy module. The data detection module is used to detect whether the data to be processed exists in the file cache area before copying the data to be processed from the file cache area to the target cache area. The third copy module is used to copy the data to be processed from the storage space of the peripheral device to the file cache area when the data to be processed does not exist in the file cache area.

[0074] Optionally, the data processing device further includes a first acquisition module and a mapping module. The first acquisition module is used to, after copying the data to be processed from the storage space of the peripheral device to the file cache area, call a database system call function to obtain the data file address of the data to be processed within the storage space of the peripheral device; the mapping module is used to map the data file address to obtain the virtual address of the file cache area.

[0075] Optionally, the first copy module includes: a first construction module, a determination module, a second construction module, and a fourth copy module. The first construction module is used to construct the target cache region in the driver layer of the kernel address space; the determination module is used to determine the target data volume corresponding to the data to be processed; the second construction module is used to create a target file identifier corresponding to the target data volume; and the fourth copy module is used to copy the data to be processed from the file cache region to the target cache region based on the target file identifier and the virtual address of the file cache region.

[0076] Optionally, the data processing module includes: a first control module, a second control module, and a third control module. The first control module controls the first controller in the accelerator board to acquire the data to be processed and send it to the second controller in the accelerator board. The second control module controls the second controller to store the data to be processed in the target memory of the accelerator board. The third control module controls the computing unit in the accelerator board to perform calculations on the data to be processed and obtain the processing result.

[0077] Optionally, the data processing module includes a permission detection module and a startup module. The permission detection module detects whether the target object has permission to access the data to be processed. The startup module, when the target object has permission, calls a data read function in the user address space, which then starts the direct memory access driver, allowing the direct memory access driver to copy the data to be processed to the accelerator board.

[0078] Optionally, the data processing device further includes: a third construction module, a second acquisition module, a script compilation module, and a deployment module. The third construction module is used to construct a heterogeneous accelerated computing system call interface via hook functions before copying the data to be processed in the target cache area to the accelerator board by calling the data reading function in the user address space of the operating system. The second acquisition module is used to acquire the program script corresponding to the target functional unit through the heterogeneous accelerated computing system call interface. The target functional unit includes at least one of the following: a data query startup unit, a physical path acquisition unit, a data reading unit, a mapping unit for implementing physical address and virtual address mapping, and a direct memory access driver startup unit. The script compilation module is used to compile the program script corresponding to the target functional unit to generate a dynamic library. The deployment module is used to deploy the dynamic library in the operating system on which the target database is installed.

[0079] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0080] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0081] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0082] Embodiments of this application also provide an electronic device. Figure 10 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 10 As shown, it includes a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0083] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0084] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0085] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular hardware and software combination.

[0086] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, Applied in servers, including: In response to a data processing request, the data to be processed in the file cache area is copied to the target cache area, wherein the file cache area and the target cache area are deployed in the kernel address space of the operating system; The data reading function is called in the user address space of the operating system to copy the data to be processed in the target cache area to the acceleration board. The virtual address of the virtual memory in the user address space and the virtual address of the virtual memory in the file cache area point to the same physical address to realize data sharing between the user address space and the kernel address space. The acceleration board is controlled to process the data to be processed, and the processing result is obtained; The data processing method is applied to heterogeneous database computing in a server. The method of calling a data reading function in the user address space of the operating system to copy the data to be processed in the target cache area to the accelerator card includes: calling a data reading function in the user address space of the operating system to start the direct memory access driver and copy the data to be processed in the target cache area to the accelerator card. Copying data to be processed from a file cache region to a target cache region includes: constructing the target cache region at the driver layer of the kernel address space; determining the target data volume corresponding to the data to be processed; creating a target file identifier corresponding to the target data volume; and copying the data to be processed from the file cache region to the target cache region based on the target file identifier and the virtual address of the file cache region. Specifically, if the data to be processed is data being processed for the first time, then the data to be processed is copied from the disk file to the file cache area via DMA copy, and then copied from the file cache area to the target cache area via CPU copy; if the data to be processed is not data being processed for the first time, then the data to be processed is copied from the file cache area to the target cache area via CPU copy. Before the data read function is invoked in the user address space of the operating system to copy the data to be processed in the target cache area to the accelerator board, the method further includes: constructing a heterogeneous accelerated computing system call interface through a hook function; obtaining the program script corresponding to the target functional unit through the heterogeneous accelerated computing system call interface, wherein the target functional unit includes at least one of the following: a data query startup unit, a physical path acquisition unit, a data read unit, a mapping unit for implementing physical address and virtual address mapping, and a direct memory access driver startup unit; compiling the program script corresponding to the target functional unit to generate a dynamic library; and deploying the dynamic library in the operating system on which the target database is installed. Before copying the data to be processed from the file cache area to the target cache area, the method further includes: detecting whether the data to be processed exists in the file cache area; if the data to be processed does not exist in the file cache area, copying the data to be processed from the storage space of the peripheral device to the file cache area; after copying the data to be processed from the storage space of the peripheral device to the file cache area, calling a database system call function to obtain the data file address of the data to be processed in the storage space of the peripheral device; and mapping the data file address to obtain the virtual address of the file cache area.

2. The method according to claim 1, characterized in that, Controlling the acceleration board to process the data to be processed and obtain the processing result includes: The first controller in the acceleration board acquires the data to be processed and sends the data to the second controller in the acceleration board. The second controller is controlled to store the data to be processed into the target memory in the acceleration board; The computing unit in the acceleration board is controlled to perform calculations on the data to be processed, and the processing result is obtained.

3. The method according to claim 1, characterized in that, The operating system invokes a data read function in its user address space to copy the data to be processed within the target cache area to the accelerator board, including: Detect whether the target object has access permissions to the data to be processed; When the target object has the access permission, the data read function is called in the user address space, and the direct memory access driver is started through the data read function so that the direct memory access driver copies the data to be processed to the accelerator board.

4. A data processing apparatus, characterized in that, Applied in servers, including: The first copy module is used to respond to data processing requests and copy the data to be processed in the file cache area to the target cache area, wherein the file cache area and the target cache area are deployed in the kernel address space of the operating system; The second copy module is used to call the data reading function in the user address space of the operating system to copy the data to be processed in the target cache area to the acceleration board. The virtual address of the virtual memory in the user address space and the virtual address of the virtual memory in the file cache area point to the same physical address to realize data sharing between the user address space and the kernel address space. The data processing module is used to control the acceleration board to process the data to be processed and obtain the processing result; The data processing method is applied to heterogeneous database computing in a server. The method of calling a data reading function in the user address space of the operating system to copy the data to be processed in the target cache area to the accelerator card includes: calling a data reading function in the user address space of the operating system to start the direct memory access driver and copy the data to be processed in the target cache area to the accelerator card. Specifically, if the data to be processed is data being processed for the first time, then the data to be processed is copied from the disk file to the file cache area via DMA copy, and then copied from the file cache area to the target cache area via CPU copy; if the data to be processed is not data being processed for the first time, then the data to be processed is copied from the file cache area to the target cache area via CPU copy. The data processing device further includes: a data detection module and a third copy module, wherein the data detection module is used to detect whether there is data to be processed in the file cache area before copying the data to be processed in the file cache area to the target cache area; the third copy module is used to copy the data to be processed from the storage space of the peripheral device to the file cache area when there is no data to be processed in the file cache area. The data processing device further includes: a first acquisition module and a mapping module, wherein the first acquisition module is used to call a database system call function to obtain the data file address of the data to be processed in the storage space of the peripheral device after copying the data to be processed from the storage space of the peripheral device to the file cache area; the mapping module is used to map the data file address to obtain the virtual address of the file cache area; The data processing device further includes: a third construction module, a second acquisition module, a script compilation module, and a deployment module. The third construction module is used to construct a heterogeneous accelerated computing system call interface via hook functions before copying the data to be processed in the target cache area to the accelerator board by calling a data reading function in the user address space of the operating system. The second acquisition module is used to acquire the program script corresponding to the target functional unit through the heterogeneous accelerated computing system call interface. The target functional unit includes at least one of the following: a data query startup unit, a physical path acquisition unit, a data reading unit, a mapping unit for implementing physical address and virtual address mapping, and a direct memory access driver startup unit. The script compilation module is used to compile the program script corresponding to the target functional unit to generate a dynamic library. The deployment module is used to deploy the dynamic library in the operating system on which the target database is installed.

5. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 3.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 3.

Images