GPU-based binary large file parsing method, system, medium and terminal

Abstract

The present invention provides a binary large file parsing method, system, medium and terminal based on GPU, including: mapping the binary large file through CPU, reading several partial files of preset size each time and copying them to GPU video memory; extracting the DDS data packet frame header and frame footer data and the subject data to be parsed that conform to the target format; storing the data to be parsed as cache. Based on the cache, parse the RTPS data in the DDS protocol through the parsing function, find the RTPS data packet according to the corresponding byte offset; read the field of the corresponding subject data, match the data packet, parse the successfully matched data packet, save the parsing result as str type summary, copy it from GPU to CPU, and use CPU to write it to the hard disk and save it as csv file. The present invention shortens the copying time, directly operates the data in the CPU memory, improves the parsing efficiency, and shortens the time.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a large binary file parsing method, system, medium and terminal based on GPU. Background Art

[0002] At present, when a large binary file exceeds 100G, the common practice is to use the CPU to parse the file. Considering that the CPU is a serial parse, parsing one file after another, it leads to long parsing time and low efficiency.

[0003] Patent document CN112099963A discloses a method and device for binary log file parsing, which relates to the field of computer technology. A specific implementation of the method includes: obtaining a binary log file; determining a position point of the binary log file to be parsed corresponding to the specified time according to a specified time; and parsing the binary log file according to the position point.

[0004] However, although patent document CN112099963A reduces CPU consumption and binary log file parsing delay to a certain extent, there is still room for shortening the parsing time compared with the present invention, and the present invention uses GPU for parallel parsing.

[0005] Patent document CN105122806A discloses an information processing device, a control method, a program, and a storage medium, wherein the GPU obtains a plurality of blocks having a predetermined number of pixels obtained by segmenting an image of an encoding object, and generates an intermediate block for each block after eliminating spatial redundancy. Furthermore, the GPU parses the data string of each intermediate block, generates determination information for determining the position of data whose value in the data string is included in a predetermined range, and outputs at least a portion of the data of the intermediate block and the corresponding determination information to a storage device for each block. In addition, the CPU refers to the corresponding determination information for each of the plurality of blocks, reads at least a portion of the data corresponding to the determination information from the storage device, and performs compression processing.

[0006] However, patent document CN105122806A adopts CPU to parse serially one by one, which is not as effective as GPU and is time-consuming. Summary of the invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a GPU-based binary large file parsing method, system, medium and terminal.

[0008] A binary large file parsing method based on GPU provided by the present invention includes:

[0009] Step S1: Mapping the large binary file through the CPU, reading several partial files of preset size each time, and copying the read data to the GPU memory;

[0010] Step S2: extracting the DDS data packet frame header and frame footer data conforming to the target format and the subject data to be parsed from the read binary data;

[0011] Step S3: storing the data to be parsed according to the subject data as a cache;

[0012] Step S4: Based on the cache, parse the RTPS data in the DDS protocol through a parsing function, and find the RTPS data packet according to the corresponding byte offset;

[0013] Step S5: Read the fields of the corresponding subject data, match the data packets, parse the successfully matched data packets, save the parsing results as str type, and finally copy the parsing results from GPU to CPU, write them to the hard disk by CPU and save them as csv file.

[0014] Preferably, the preset size includes 256M and 512M;

[0015] In step S3, vector or encapsulated list dynamic container method is used to store data;

[0016] The RTPS is a standardized transport layer protocol of DDS, which is used for the actual transmission of underlying data;

[0017] When the parsing result is saved as str type, it is saved additionally every 25ms.

[0018] Preferably, step S2 comprises:

[0019] Step S2.1: Calculate the number of DDS headers in the partial file;

[0020] Step S2.2: Allocate memory on the device for the DDS data packet frame header, frame trailer, and subject data;

[0021] Step S2.3: Write a parsing function, call CUDA kernel to parse the DDS data packet frame header and frame tail, and extract the subject data according to the position of the DDS data packet frame header and frame tail;

[0022] Step S2.4: Wait for CUDA execution to complete and clean up the memory allocated by CUDA.

[0023] Preferably, the step S4 comprises:

[0024] Step S4.1: Find the offset of the RTPS marker and parse from the beginning of the data;

[0025] Step S4.2: If the RTPS tag is found, parse the RTPS data packet and execute step S4.3; if the RTPS tag is not found, return None;

[0026] Step S4.3: parsing the RTPS packet header corresponding to the RTPS tag, and storing the RTPS packet header information and corresponding data information;

[0027] Step S4.4: Calculate the offset of the next RTPS data packet;

[0028] Step S4.5: Recursively call the function to parse the next RTPS data packet.

[0029] A binary large file parsing system based on GPU provided by the present invention includes:

[0030] Module M1: Maps the binary large file through the CPU, reads several parts of the file with preset size each time, and copies the read data to the GPU memory;

[0031] Module M2: extract the DDS data packet frame header and frame tail data that conform to the target format and the subject data to be parsed from the read binary data;

[0032] Module M3: storing the data to be parsed according to the subject data as a cache;

[0033] Module M4: Based on the cache, parse the RTPS data in the DDS protocol through a parsing function, and find the RTPS data packet according to the corresponding byte offset;

[0034] Module M5: Read the fields of the corresponding topic data, match the data packets, parse the successfully matched data packets, save the parsing results as str type, and finally copy the parsing results from GPU to CPU, write them to the hard disk by CPU and save them as csv files.

[0035] Preferably, the preset size includes 256M and 512M;

[0036] The module M3 uses vector or encapsulated list dynamic container system for data storage;

[0037] The RTPS is a standardized transport layer protocol of DDS, which is used for the actual transmission of underlying data;

[0038] When the parsing result is saved as str type, it is saved additionally every 25ms.

[0039] Preferably, the module M2 comprises:

[0040] Module M2.1: Calculate the number of DDS headers in the partial file;

[0041] Module M2.2: Allocate memory on the device for the DDS data packet header, footer, and subject data;

[0042] Module M2.3: Write a parsing function, call CUDAkernel to parse the DDS data packet frame header and frame tail, and extract the subject data according to the position of the DDS data packet frame header and frame tail;

[0043] Module M2.4: Wait for CUDA trigger to complete and clean up the memory allocated by CUDA.

[0044] Preferably, the module M4 comprises:

[0045] Module M4.1: Find the offset of the RTPS marker and parse from the beginning of the data;

[0046] Module M4.2: If the RTPS tag is found, parse the RTPS data packet to trigger module M4.3; if the RTPS tag is not found, return None;

[0047] Module M4.3: Parsing the RTPS header corresponding to the RTPS tag, and storing the RTPS header information and corresponding data information;

[0048] Module M4.4: Calculate the offset of the next RTPS data packet;

[0049] Module M4.5: Recursively call the function to parse the next RTPS packet.

[0050] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the GPU-based binary large file parsing method are implemented.

[0051] According to the present invention, a smart mobile terminal includes the computer-readable storage medium storing a computer program, or includes the GPU-based binary large file parsing system.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The present invention solves the problem of two copies of data from the CPU to the GPU memory and from the GPU to the CPU by adopting a zero-copy method, and shortens the copy time. The GPU uses a large page memory pointer to read the CPU memory, reducing the number and time of copies and directly operating the data in the CPU memory.

[0054] 2. The present invention uses GPU multi-threading and a huge number of threads, and multiple threads simultaneously process parsing data to improve parsing efficiency and shorten time.

[0055] 3. The present invention adopts a shared memory approach to optimize parallel computing performance, allowing multiple threads to efficiently access and share data without frequently accessing slower global memory, thereby solving the problem of frequent access to global memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0057] Figure 1 It is a schematic diagram of the working method of the present invention. DETAILED DESCRIPTION

[0058] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0059] In the present invention, when a large binary file exceeds 100G, the current common practice is to use the CPU to parse the file. Considering that the CPU is a serial parse, parsing one file after another, the parsing time is relatively long. Considering that the GPU is a parallel processing technology, parallel processing can shorten the time. Therefore, it is considered to use the GPU to perform parallel parsing of large files, which requires mixed programming with the CPU. The CPU is used to read the file and the result is copied to the GPU for parallel operation to shorten the parsing time. Therefore, it is a new, efficient, and unprecedented big data parsing method, which has many application scenarios in big data parsing.

[0060] According to a GPU-based binary large file parsing method provided by the present invention, Figure 1 As shown, including:

[0061] Step S1: Map the binary file through the CPU, read several parts of the file of preset size each time, and copy the read data to the GPU memory. The preset size includes 256M and 512M.

[0062] Step S2: Write a cuda development code to extract the DDS data packet frame header and frame footer data that conform to the target format and the subject data to be parsed from the read binary data.

[0063] Step S2.1: Calculate the number of DDS headers in the partial file.

[0064] Step S2.2: Allocate memory on the device for the DDS data packet frame header, frame trailer, and subject data.

[0065] Step S2.3: Write a parsing function, call CUDAkernel to parse the DDS data packet frame header and frame tail, and extract the subject data according to the DDS data packet frame header and frame tail position. For example, Dowrd findmessagehead(uinthead) is used to find the DDS frame header inline function, and Dowrd findmessageeof(uinteof) is used to find the DDS frame tail inline function.

[0066] Step S2.4: Wait for CUDA execution to complete and clean up the memory allocated by CUDA.

[0067] Step S3: The subject data is stored in a vector or encapsulated list dynamic container technology to be parsed as a cache.

[0068] Step S4: Based on the cache, a parsing function is written to parse the RTPS data in the DDS protocol, and the RTPS data packet is found according to the corresponding byte offset. RTPS is a standardized transport layer protocol of DDS, which is used for the actual transmission of underlying data.

[0069] Step S4.1: Find the offset of the RTPS marker and parse from the beginning of the data.

[0070] Step S4.2: If the RTPS marker is found, parse the RTPS data packet and execute step S4.3; if the RTPS marker is not found, return None.

[0071] Step S4.3: Parse the RTPS packet header corresponding to the RTPS tag, and store the RTPS packet header information and corresponding data information.

[0072] Step S4.4: Calculate the offset of the next RTPS data packet.

[0073] Step S4.5: Recursively call the function to parse the next RTPS data packet.

[0074] Step S5: Read the fields of the corresponding subject data, match the data packets, parse the successfully matched data packets, save the parsing results as str type, append and save them every 25ms, and finally copy the parsing results from GPU to CPU, write them to the hard disk by CPU and save them as csv files.

[0075] The present invention also provides a GPU-based binary large file parsing system, which can be implemented by executing the process steps of the GPU-based binary large file parsing method, that is, those skilled in the art can understand the GPU-based binary large file parsing method as a preferred implementation of the GPU-based binary large file parsing system.

[0076] A binary large file parsing system based on GPU provided by the present invention includes:

[0077] Module M1: Maps the binary file through the CPU, reads several parts of the file with preset sizes each time, and copies the read data to the GPU memory. The preset sizes include 256M and 512M.

[0078] Module M2: Extract the DDS data packet frame header and frame trailer data that conform to the target format and the subject data to be parsed from the read binary data. Module M2 includes: Module M2.1: Calculate the number of DDS headers in the said partial file. Module M2.2: Allocate memory on the device for the DDS data packet frame header and frame trailer, and subject data. Module M2.3: Write a parsing function, call the CUDA kernel to parse the DDS data packet frame header and frame trailer, and extract the subject data based on the position of the DDS data packet frame header and frame trailer. Module M2.4: Wait for the CUDA trigger to complete and clean up the memory allocated by CUDA.

[0079] Module M3: Store the data to be parsed according to the subject data as a cache. In the module M3, a vector or a packaged list dynamic container system is used to store data.

[0080] Module M4: Based on the cache, parse the RTPS data in the DDS protocol through the parsing function, and find the RTPS data packet according to the corresponding byte offset. The RTPS is a standardized transport layer protocol of DDS, which is used for the actual transmission of underlying data. The module M4 includes: Module M4.1: Find the offset of the RTPS marker and parse from the beginning of the data. Module M4.2: If the RTPS marker is found, parse the RTPS data packet to trigger module M4.3. If the RTPS marker is not found, return None. Module M4.3: Parse the RTPS packet header corresponding to the RTPS marker, and store the RTPS packet header information and corresponding data information. Module M4.4: Calculate the offset of the next RTPS data packet. Module M4.5: Recursively call the function to parse the next RTPS data packet.

[0081] Module M5: read the fields of the corresponding subject data, match the data packets, parse the successfully matched data packets, save the parsing results as str type, and finally copy the parsing results from GPU to CPU, write them to the hard disk by CPU and save them as csv files. When the parsing results are saved as str type, they are saved additionally every 25ms.

[0082] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the GPU-based binary large file parsing method are implemented.

[0083] According to the present invention, a smart mobile terminal includes the computer-readable storage medium storing a computer program, or includes the GPU-based binary large file parsing system.

[0084] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0085] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A binary large file parsing method based on GPU, characterized in that: include: Step S1: Mapping the large binary file through the CPU, reading several partial files of preset size each time, and copying the read data to the GPU memory; Step S2: extracting the DDS data packet frame header and frame footer data conforming to the target format and the subject data to be parsed from the read binary data; Step S3: storing the data to be parsed according to the subject data as a cache; Step S4: Based on the cache, parse the RTPS data in the DDS protocol through a parsing function, and find the RTPS data packet according to the corresponding byte offset; the step S4 includes: Step S4.1: Find the offset of the RTPS marker and parse from the beginning of the data; Step S4.2: If the RTPS marker is found, parse the RTPS data packet and execute step S4.3; if the RTPS marker is not found, return None; Step S4.3: parsing the RTPS packet header corresponding to the RTPS tag, and storing the RTPS packet header information and corresponding data information; Step S4.4: Calculate the offset of the next RTPS data packet; Step S4.5: recursively call the function to parse the next RTPS data packet; Step S5: Read the fields of the corresponding subject data, match the data packets, parse the successfully matched data packets, save the parsing results as str type, and finally copy the parsing results from GPU to CPU, write them to the hard disk by CPU and save them as csv file.

2. The GPU-based binary large file parsing method according to claim 1, characterized in that: The preset sizes include 256M and 512M; In step S3, vector or encapsulated list dynamic container method is used to store data; The RTPS is a standardized transport layer protocol of DDS, which is used for the actual transmission of underlying data; When the parsing result is saved as str type, it is saved additionally every 25ms.

3. The GPU-based binary large file parsing method according to claim 1, characterized in that: The step S2 comprises: Step S2.1: Calculate the number of DDS headers in the partial file; Step S2.2: Allocate memory on the device for the DDS data packet frame header, frame trailer, and subject data; Step S2.3: Write a parsing function, call CUDA kernel to parse the DDS data packet frame header and frame tail, and extract the subject data according to the position of the DDS data packet frame header and frame tail; Step S2.4: Wait for CUDA execution to complete and clean up the memory allocated by CUDA.

4. A binary large file parsing system based on GPU, characterized in that: include: Module M1: Maps the binary large file through the CPU, reads several parts of the file with preset size each time, and copies the read data to the GPU memory; Module M2: extract the DDS data packet frame header and frame tail data that conform to the target format and the subject data to be parsed from the read binary data; Module M3: storing the data to be parsed according to the subject data as a cache; Module M4: Based on the cache, parse the RTPS data in the DDS protocol through a parsing function, and find the RTPS data packet according to the corresponding byte offset; the module M4 includes: Module M4.1: Find the offset of the RTPS marker and parse from the beginning of the data; Module M4.2: If the RTPS tag is found, parse the RTPS data packet to trigger module M4.3; if the RTPS tag is not found, return None; Module M4.3: Parsing the RTPS header corresponding to the RTPS tag, and storing the RTPS header information and corresponding data information; Module M4.4: Calculate the offset of the next RTPS data packet; Module M4.5: Recursively call the function to parse the next RTPS packet; Module M5: Read the fields of the corresponding topic data, match the data packets, parse the successfully matched data packets, save the parsing results as str type, and finally copy the parsing results from GPU to CPU, write them to the hard disk by CPU and save them as csv files.

5. The GPU-based binary large file parsing system according to claim 4, characterized in that: The preset sizes include 256M and 512M; The module M3 uses vector or encapsulated list dynamic container system for data storage; The RTPS is a standardized transport layer protocol of DDS, which is used for the actual transmission of underlying data; When the parsing result is saved as str type, it is saved additionally every 25ms.

6. The GPU-based binary large file parsing system according to claim 4, characterized in that: The module M2 comprises: Module M2.1: Calculate the number of DDS headers in the partial file; Module M2.2: Allocate memory on the device for the DDS data packet header, footer, and subject data; Module M2.3: Write a parsing function, call CUDAkernel to parse the DDS data packet frame header and frame tail, and extract the subject data according to the position of the DDS data packet frame header and frame tail; Module M2.4: Wait for CUDA trigger to complete and clean up the memory allocated by CUDA.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the GPU-based large binary file parsing method according to any one of claims 1 to 3 are implemented.

8. An intelligent mobile terminal, characterized in that: A computer-readable storage medium storing a computer program as claimed in claim 7, or a binary large file parsing system based on a GPU as claimed in any one of claims 4 to 6.

Images