GPU (Graphics Processing Unit) image striping processing method and device and storage medium

Abstract

The invention relates to a GPU (Graphics Processing Unit) image striping processing method and device and a storage medium, and the method comprises the steps: initializing striping collection video image frames, and pre-dividing each video image frame into at least two pieces of strip data which are connected in sequence; starting striping to collect the video image frame, storing one stripe of the video image frame into a DMA (Direct Memory Access) buffer area by a graphics processor after collecting data of one stripe of the video image frame, and updating a stripe index; a memory mapping address is obtained through a DMA buffer area, and an application program directly accesses and processes video image frame data through the memory mapping address and accesses each video image frame through memory mapping; and if the stripe data of the video image frame is readable, copying the stripe data through OpenCL and sending the stripe data to the encoder with the corresponding serial number for encoding processing. According to the GPU image striping processing method, continuous acquisition and efficient processing of image data can be ensured, and meanwhile, the stability of image quality and the real-time performance of processing are ensured.

Description

Technical Field

[0001] The present invention relates to the field of video image processing, and in particular to a GPU image striping processing method, device and storage medium. Background Art

[0002] In the traditional KVM and matrix switching fields, lossless video signal quality and no perceived delay are required. The implementation scheme is to transmit the video signal after shallow compression through the FPGA chip. The data bandwidth is very large, so optical fiber or independent gigabit network is generally used on the channel. Its disadvantages are very obvious: high construction cost and poor scalability. When sharing the video signal with a third-party system, special hardware equipment is required for acquisition and encoding.

[0003] The H264 and H265 deep video signal compression algorithms have low bandwidth, low construction cost, and high scalability, but their image quality is lossy and the latency is high. The image is converted into YUV420 and then encoded. When converted to YUV420, 4 pixels share 1 UV component, and a lot of color information has been lost, which cannot meet the requirements of visual lossless image quality. This is particularly prominent in computer desktop image encoding scenarios. Summary of the invention

[0004] The present invention provides a GPU image striping processing method and a cloud PC management platform, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of the present invention is a GPU image striping processing method, a GPU image striping processing method, the GPU image striping processing method is applied to a GPU image striping processing device, the GPU image striping processing device comprises: an image acquisition device; a host computer, the host computer is connected to the image acquisition device, the host computer at least comprises a central processing unit (CPU), a graphics processing unit (GPU) and a memory, the GPU image striping processing method comprises the following steps: S100, initializing striped acquisition of video image frames, each of the video image frames being pre-divided into at least two sequentially connected strip data; S200, starting striped acquisition of video image frames, after each acquisition of a strip of video image frame data, a graphics processing unit (GPU) stores a strip of the video image frame into a DMA buffer (dmabuf), and updates a strip index; S300, obtaining a memory mapping address through a DMA buffer (dmabuf), the application program directly accesses and processes the video image frame data through the memory mapping address, and accesses each video image frame through the memory mapping; S400: If the strip data of the video image frame is readable, copy the strip data through OpenCL and send it to the encoder of the corresponding sequence number for encoding processing.

[0006] Further, the step S100 includes the following steps: S110, configuring parameters of a video image frame based on a Linux video device kernel driver interface (Video for Linux 2, V4L2), wherein the parameters of the video image frame include at least a video frame rate, a video resolution, and a video pixel format; S120, create a DMA buffer (dmabuf) through the VIDIOC_REQBUFS command of the Linux video device kernel driver interface (Video for Linux 2, V4L2); S130, mapping the DMA buffer (dmabuf) to the file system through the VIDIOC_QUERYBUF command of the Linux video device kernel driver interface (Video for Linux 2, V4L2).

[0007] Further, the step S200 includes the following steps: S210, start the video stream through the VIDIOC_STREAMON command of the Linux video device kernel driver interface (Video for Linux 2, V4L2); S220, performing initial stability calibration, and discarding preset video image frame data; S230, after completing the initial stability calibration, enter the main processing loop, continuously obtain video image frames from the DMA buffer (dmabuf), and continue to collect and transmit video image frames with preset video image frame parameters; S240, after each stripe is completed, updating the stripe index, wherein the number of the stripe indexes is consistent with the number of stripes of the video image frame; S250, after each video image frame is completed, the DMA buffer (dmabuf) is dequeued and requeued for the next round of acquisition and processing.

[0008] Further, the step S300 includes the following steps: S310, querying detailed information of each DMA buffer (dmabuf) through a VIDIOC_QUERYBUF command, and obtaining file descriptor (FD) information of each DMA buffer (dmabuf) used for memory mapping; S320, calling the mmap function through the file descriptor (FD), mapping the DMA buffer (dmabuf) to the address space of the application, the mmap function returns a pointer to the DMA buffer (dmabuf), and the application directly accesses the data of the DMA buffer (dmabuf) through the pointer.

[0009] Further, in step S320, Each video image frame contains pixel data in YUV format. Each pixel of the video image frame is directly read in the order of Y channel, U channel and V channel through the pointer, and then the pixel data is subjected to one or more combinations of brightness adjustment processing, color correction processing and image analysis processing.

[0010] Further, in step S100, The video image frame is pre-divided into a first stripe, a second stripe, a third stripe and a fourth stripe which are connected in sequence.

[0011] Further, in step S200, when the video pixel format is YUV420, the video resolution is 1920x1080 and the video frame rate is 60HZ, in order to prevent data errors caused by system processing speed or stripe writing delay during the acquisition process, a fixed time interval is used for reading operation, specifically, The time interval between the first stripe and the fourth stripe of the previous frame is 6.166 milliseconds, the time interval between the second stripe and the first stripe is 4.166 milliseconds, the time interval between the third stripe and the second stripe is 4.166 milliseconds, and the time interval between the fourth stripe and the third stripe is 4.166 milliseconds.

[0012] Further, in step S400, S410, the system initializes the OpenCL environment, creates an OpenCL context to manage computing resources in a graphics processing unit (GPU) device, creates a command queue for submitting data transmission and computing tasks to the graphics processing unit (GPU) device, and compiles an OpenCL kernel program for copying stripe data to a coding cache; S420, allocating a memory object in the OpenCL context as a data source and a target buffer, wherein the target buffer is a coding buffer, and the target buffer is ultimately used as input data of the encoder; S430, writing an OpenCL data copy kernel for copying stripe data, and setting data copy kernel parameters, wherein the data copy kernel parameters include a source buffer, a target buffer, and a width and height of the stripe; S440, after starting the kernel, copy the strip data from the DMA buffer (dmabuf) to the target buffer through the OpenCL API function, and use the two-dimensional range to execute the kernel to ensure that the data of each pixel is copied correctly.

[0013] S450, data identification and forwarding: After the data copy kernel is executed, the stripe data will be written into the encoding cache, the encoding cache will be transferred to the encoder, and the encoding process will be started.

[0014] Furthermore, the present invention also provides a GPU image striping processing device, which is used to execute the GPU image striping processing method. The GPU image striping processing device includes: Image acquisition equipment; A host computer is connected to the image acquisition device, and the host computer at least includes a central processing unit (CPU), a graphics processing unit (GPU) and a memory.

[0015] Furthermore, the present invention also provides a computer-readable storage medium storing program instructions, wherein the program instructions implement the GPU image striping processing method when executed by a processor.

[0016] The beneficial effects of the present invention are: The GPU image striping processing method divides the video image frame into stripes. Through the striping division, the system can ensure the continuous acquisition and efficient processing of image data, while ensuring the stability of image quality and the real-time processing. Through DMA cache post-processing, the processing speed and efficiency of video images are accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The figure is an overall flow chart of the GPU image striping processing method.

[0018] Figure 2 It is a schematic diagram of the structures of the first stripe, the second stripe, the third stripe and the fourth stripe in the GPU image striping processing method. DETAILED DESCRIPTION

[0019] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0020] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, top, bottom, etc. used in the present invention are only relative to the relative positional relationship of the components of the present invention in the drawings.

[0021] In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art. The terms used in this specification are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used herein includes any combination of one or more related listed items.

[0022] It should be understood that, although the terms first, second, third, etc. may be used to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0023] Reference Figure 1 to Figure 2 In some embodiments, the technical solution of the present invention is a GPU image striping processing method, a GPU image striping processing method, the GPU image striping processing method is applied to a GPU image striping processing device, the GPU image striping processing device comprises: an image acquisition device; a host computer, the host computer is connected to the image acquisition device, the host computer at least comprises a central processing unit (CPU), a graphics processing unit (GPU) and a memory, refer to Figure 1 , the GPU image striping processing method comprises the following steps: S100, initializing striped acquisition of video image frames, each of the video image frames being pre-divided into at least two sequentially connected strip data; S200, starting striped acquisition of video image frames, after each acquisition of a strip of video image frame data, a graphics processing unit (GPU) stores a strip of the video image frame into a DMA buffer (dmabuf), and updates a strip index; S300, obtaining a memory mapping address through a DMA buffer (dmabuf), the application program directly accesses and processes the video image frame data through the memory mapping address, and accesses each video image frame through the memory mapping; S400: If the strip data of the video image frame is readable, copy the strip data through OpenCL and send it to the encoder of the corresponding sequence number for encoding processing.

[0024] The beneficial effects of the present invention are: The GPU image striping processing method divides the video image frame into stripes. Through the striping division, the system can ensure the continuous acquisition and efficient processing of image data, while ensuring the stability of image quality and the real-time processing. Through DMA cache post-processing, the processing speed and efficiency of video images are accelerated.

[0025] Specifically, taking 1080P video frame as an example, refer to Figure 2 , divide a frame of 1080P data into 4 strips for collection, and the resolution of each strip is 1920x270. After the system obtains one quarter of the video data, that is, one strip, it is copied into the encoding cache through the GPU and sent to the corresponding encoder.

[0026] Further, refer to Figure 1 , the step S100 includes the following steps; S110, configuring parameters of a video image frame based on a Linux video device kernel driver interface (Video for Linux 2, V4L2), wherein the parameters of the video image frame include at least a video frame rate, a video resolution, and a video pixel format; S120, create a DMA buffer (dmabuf) through the VIDIOC_REQBUFS command of the Linux video device kernel driver interface (Video for Linux 2, V4L2); S130, mapping the DMA buffer (dmabuf) to the file system through the VIDIOC_QUERYBUF command of the Linux video device kernel driver interface (Video for Linux 2, V4L2).

[0027] Specifically, the GPU image striping processing method can perform efficient data transmission and efficiently collect image data from the camera using direct memory access (DMA) technology through the Video for Linux 2 interface (Video for Linux 2, V4L2). The following are the specific steps: 1) Video data acquisition initialization: The system configures the video source through the Video for Linux 2 interface (V4L2), sets the frame rate, resolution (1920x1080), pixel format (YUV420), and starts the video stream. During this process, the Video for Linux 2 interface (V4L2) will create a buffer queue for subsequent use.

[0028] 2) Buffer allocation and mapping: The system uses the VIDIOC_REQBUFS command of the Video for Linux 2 interface (V4L2) to allocate a DMA buffer for transmission to achieve efficient transmission from video acquisition to encoding memory. The allocated buffer is then mapped through the VIDIOC_QUERYBUF command so that the system can access the data in the buffer.

[0029] 3) Data acquisition trigger: The system calls the VIDIOC_STREAMON command to start the video stream and start the transmission of image data from the video source. Each frame of video acquisition is automatically transferred to the mapped DMA buffer, avoiding CPU involvement and reducing system burden.

[0030] Further, refer to Figure 1 , the step S200 includes the following steps; S210, start the video stream through the VIDIOC_STREAMON command of the Linux video device kernel driver interface (Video for Linux 2, V4L2); S220, performing initial stability calibration, and discarding preset video image frame data; S230, after completing the initial stability calibration, enter the main processing loop, continuously obtain video image frames from the DMA buffer (dmabuf), and continue to collect and transmit video image frames with preset video image frame parameters; S240, after each stripe is completed, updating the stripe index, wherein the number of the stripe indexes is consistent with the number of stripes of the video image frame; S250, after each video image frame is completed, the DMA buffer (dmabuf) is dequeued and requeued for the next round of acquisition and processing.

[0031] Specifically, in steps S220 and S230, stability calibration is first performed. Before entering the main processing loop, the system discards the first 8 frames of images, which are usually unstable due to initialization and other reasons. Loop processing mechanism: After completing the initial calibration, the system enters a loop to continuously obtain image frames from the acquisition queue.

[0032] In step S250, after completing the processing of an entire video image frame (four strips), the system will dequeue the DMA buffer and requeue it for the next round of acquisition and processing. Under normal circumstances, after the system acquires a complete frame of image, it should perform a dequeue operation to obtain video frame data, process the data and requeue it. According to the above steps, part of the strip data can be collected and sent for encoding before dequeuing, so as to achieve parallel processing and improve efficiency.

[0033] Further, refer to Figure 1 , the step S300 comprises the following steps: S310, querying detailed information of each DMA buffer (dmabuf) through a VIDIOC_QUERYBUF command, and obtaining file descriptor (FD) information of each DMA buffer (dmabuf) used for memory mapping; S320, calling the mmap function through the file descriptor (FD), mapping the DMA buffer (dmabuf) to the address space of the application, the mmap function returns a pointer to the DMA buffer (dmabuf), and the application directly accesses the data of the DMA buffer (dmabuf) through the pointer.

[0034] Specifically, for each acquired frame, the system obtains its memory mapping address through the DMA buffer (dmabuf) to directly access and process the image data. The following are the specific steps: 1) DMA buffer allocation: The system allocates a DMA buffer for the video stream through the VIDIOC_REQBUFS command of the V4L2 interface. The buffer is used to store the image frame data collected by the camera and supports direct memory access (DMA), enabling data to be efficiently transferred between hardware.

[0035] 2) Get the dmabuf file descriptor of the buffer: When the DMA buffer is allocated, the system queries the detailed information of each buffer through the VIDIOC_QUERYBUF command, especially getting the dmabuf file descriptor (FD) of the buffer. This dmabuf file descriptor will be used for memory mapping so that the system can directly access the image data in the buffer.

[0036] 3) Memory mapping: The system calls the mmap() function to map the DMA buffer to the user space. The specific implementation is as follows: The system calls mmap() through the dmabuf file descriptor (FD) to map the buffer to the application's address space. mmap() returns a pointer to the buffer, allowing the application to directly access the data in the buffer. In this way, the system can operate the image data collected by the camera directly in the memory without data copying, thereby improving the data processing speed.

[0037] 4) Data access and processing: After the memory mapping is completed, the system can directly access the image data of each video frame through the pointer. For example, assuming that each frame of the image contains pixel data in YUV format, the system can directly read each pixel in the order of Y, U, and V channels through the pointer. In real-time video processing, the system can operate on this data, such as adjusting brightness, color correction, or performing image analysis.

[0038] Further, refer to Figure 1 In step S320, Each video image frame contains pixel data in YUV format. Each pixel of the video image frame is directly read in the order of Y channel, U channel and V channel through the pointer, and then the pixel data is subjected to one or more combinations of brightness adjustment processing, color correction processing and image analysis processing.

[0039] Further, refer to Figure 1 In the step S100, The video image frame is pre-divided into a first stripe, a second stripe, a third stripe and a fourth stripe which are connected in sequence.

[0040] Specifically, each frame is divided into four strips for striped acquisition to process the data of each strip in parallel. After processing a strip, the system switches the strip index to the next strip and continues processing. The strip index cycles between one and four until all four strips in a frame are processed.

[0041] Further, refer to Figure 1 In step S200, when the video pixel format is YUV420, the video resolution is 1920x1080 and the video frame rate is 60HZ, in order to prevent data errors caused by system processing speed or stripe writing delay during the acquisition process, a fixed time interval is used for reading operations, specifically, The time interval between the first stripe and the fourth stripe of the previous frame is 6.166 milliseconds, the time interval between the second stripe and the first stripe is 4.166 milliseconds, the time interval between the third stripe and the second stripe is 4.166 milliseconds, and the time interval between the fourth stripe and the third stripe is 4.166 milliseconds.

[0042] Specifically, when the video frame rate is 60, the time interval is calculated by dividing 1 second by 60 frames and then by four stripes. The video resolution can be 720P, 1920x1200, and 1440x900. When the video frame rate is 30 frames, the time intervals are 10.333ms, 8.333ms, 8.333ms, and 8.333ms, respectively.

[0043] Further, refer to Figure 1 In the step S400, S410, the system initializes the OpenCL environment, creates an OpenCL context to manage computing resources in a graphics processing unit (GPU) device, creates a command queue for submitting data transmission and computing tasks to the graphics processing unit (GPU) device, and compiles an OpenCL kernel program for copying stripe data to a coding cache; S420, allocating a memory object in the OpenCL context as a data source and a target buffer, wherein the target buffer is a coding buffer, and the target buffer is ultimately used as input data of the encoder; S430, writing an OpenCL data copy kernel for copying stripe data, and setting data copy kernel parameters, wherein the data copy kernel parameters include a source buffer, a target buffer, and a width and height of the stripe; S440, after starting the kernel, copy the strip data from the DMA buffer (dmabuf) to the target buffer through the OpenCL API function, and use the two-dimensional range to execute the kernel to ensure that the data of each pixel is copied correctly.

[0044] S450, data identification and forwarding: After the data copy kernel is executed, the stripe data will be written into the encoding cache, the encoding cache will be transferred to the encoder, and the encoding process will be started.

[0045] Further, refer to Figure 1 The present invention further proposes a GPU image striping processing device, which is used to execute the GPU image striping processing method. The GPU image striping processing device comprises: Image acquisition equipment; A host computer is connected to the image acquisition device, and the host computer at least includes a central processing unit (CPU), a graphics processing unit (GPU) and a memory.

[0046] Further, refer to Figure 1 The present invention also provides a computer-readable storage medium storing program instructions, which implement the GPU image striping processing method when executed by a processor.

[0047] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of protection of the present disclosure. All should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. A GPU image striping processing method, wherein the GPU image striping processing method is applied to a GPU image striping processing device, wherein the GPU image striping processing device comprises: An image acquisition device; a host computer connected to the image acquisition device, the host computer at least comprising a central processing unit, a graphics processing unit and a memory, wherein the GPU image striping processing method comprises the following steps: S100, initializing striping to collect video image frames, each of the video image frames being pre-divided into at least two sequentially connected strip data; S200, start striping to capture video image frames, after each capture of a strip of video image frame data, the graphics processor stores a strip of the video image frame into a DMA buffer and updates a strip index; S300, obtaining a memory mapping address through a DMA buffer, the application program directly accesses and processes the video image frame data through the memory mapping address, and accesses each video image frame through memory mapping; S400: If the strip data of the video image frame is readable, copy the strip data through OpenCL and send it to the encoder of the corresponding sequence number for encoding processing.

2. The GPU image striping processing method according to claim 1, characterized in that: The step S100 includes the following steps: S110, configuring parameters of a video image frame based on a Linux video device kernel driver interface, wherein the parameters of the video image frame include at least a video frame rate, a video resolution, and a video pixel format; S120, creating a DMA buffer through the VIDIOC_REQBUFS command of the Linux video device kernel driver interface; S130 , mapping the DMA buffer to the file system through the VIDIOC_QUERYBUF command of the Linux video device kernel driver interface.

3. The GPU image striping processing method according to claim 1, characterized in that: The step S200 includes the following steps: S210, start the video stream through the VIDIOC_STREAMON command of the Linux video device kernel driver interface; S220, performing initial stability calibration, and discarding preset video image frame data; S230, after completing the initial stability calibration, entering the main processing loop, continuously acquiring video image frames from the DMA buffer, and continuously acquiring and transmitting video image frames with preset video image frame parameters; S240, after each stripe is completed, updating the stripe index, wherein the number of the stripe indexes is consistent with the number of stripes of the video image frame; S250, after each video image frame is completed, the DMA buffer (dmabuf) is dequeued and requeued for the next round of acquisition and processing.

4. The GPU image striping processing method according to claim 1, characterized in that: The step S300 includes the following steps: S310, querying detailed information of each DMA buffer through a VIDIOC_QUERYBUF command, and obtaining file descriptor information of each DMA buffer used for memory mapping; S320, calling the mmap function through the file descriptor to map the DMA buffer to the address space of the application program, the mmap function returns a pointer pointing to the DMA buffer, and the application program directly accesses the data in the DMA buffer through the pointer.

5. The GPU image striping processing method according to claim 4, characterized in that: In the step S320, Each video image frame contains pixel data in YUV format. Each pixel of the video image frame is directly read in the order of Y channel, U channel and V channel through the pointer, and then the pixel data is subjected to one or more combinations of brightness adjustment processing, color correction processing and image analysis processing.

6. The GPU image striping processing method according to claim 1, characterized in that: In the step S100, The video image frame is pre-divided into a first stripe, a second stripe, a third stripe and a fourth stripe which are connected in sequence.

7. The GPU image striping processing method according to claim 6, characterized in that: In step S200, when the video pixel format is YUV420, the video resolution is 1920x1080 and the video frame rate is 60HZ, in order to prevent data errors caused by system processing speed or stripe writing delay during the acquisition process, a fixed time interval is used for reading operations, specifically, The time interval between the first stripe and the fourth stripe of the previous frame is 6.166 milliseconds, the time interval between the second stripe and the first stripe is 4.166 milliseconds, the time interval between the third stripe and the second stripe is 4.166 milliseconds, and the time interval between the fourth stripe and the third stripe is 4.166 milliseconds.

8. The GPU image striping processing method according to claim 1, characterized in that: In the step S400, S410, the system initializes the OpenCL environment, creates an OpenCL context to manage computing resources in the graphics processor device, creates a command queue for submitting data transmission and computing tasks to the graphics processor device, and compiles an OpenCL kernel program for copying stripe data to the encoding cache; S420, allocating a memory object in the OpenCL context as a data source and a target buffer, wherein the target buffer is a coding buffer, and the target buffer is ultimately used as input data of the encoder; S430, writing an OpenCL data copy kernel for copying stripe data, and setting data copy kernel parameters, wherein the data copy kernel parameters include a source buffer, a target buffer, and a width and height of the stripe; S440, after starting the kernel, copy the stripe data from the DMA buffer to the target buffer through the OpenCL API function, and execute the kernel using a two-dimensional range to ensure that the data of each pixel is correctly copied; S450, data identification and forwarding: After the data copy kernel is executed, the stripe data will be written into the encoding cache, the encoding cache will be transferred to the encoder, and the encoding process will be started.

9. A GPU image striping processing device, used to execute the GPU image striping processing method according to any one of claims 1 to 8, characterized in that: The GPU image striping processing device comprises: Image acquisition equipment; A host computer is connected to the image acquisition device, and the host computer at least includes a central processing unit, a graphics processing unit and a memory.

10. A computer-readable storage medium storing program instructions, wherein the program instructions, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images