A real-time video image defogging method and system based on multi-GPU acceleration

Abstract

The present invention is applicable to the field of GPU technology, and provides a real-time defogging method and system for video images based on multi-GPU acceleration, the method comprising: S1, the camera collects foggy images, and sends the foggy images to each GPU in a frame order loop; S2, for each GPU, the received foggy image data is processed by an image defogging algorithm in a parallel acceleration manner to achieve image defogging; S3, the defogged image data is directly stored in the video memory, and the stable display of the defogged video image is achieved through a double buffering method. The present invention not only solves the problem of real-time defogging of full HD or even ultra-HD foggy video images, but also further improves the defogging performance by reducing the time consumption of copying the image data from the video memory to the main memory after multi-GPU acceleration; in addition, the present invention adopts a method of sending image data to the display and displaying image data separately in two buffers to ensure the stable display of the defogged video image.

Description

Technical Field

[0001] The present invention belongs to the field of GPU technology, and in particular relates to a real-time defogging method and system for video images based on multi-GPU acceleration. Background Art

[0002] In haze weather, due to the influence of suspended particles in the air, the video images captured by outdoor monitoring and satellite photography systems will have low contrast and loss of image detail information; at the same time, the blurriness of the input image will directly affect the subsequent extraction and processing of information.

[0003] For this problem, the current conventional method is to use an image defogging algorithm to process the image. Since the video images shot in haze weather are blurred and the image details are lost, and the image defogging algorithm is highly complex and takes a long time to run, it is difficult to meet the real-time requirements of engineering applications. In practice, there are many factors that affect the real-time defogging of foggy video images, such as the amount of data per frame of video images. The higher the image resolution, the longer the defogging time. In addition, when the image data volume is large, the data transmission between the CPU and GPU is very time-consuming. Therefore, in haze weather, the foggy video images shot by full HD or even ultra-high-definition cameras will suffer from severe frame loss and screen display freezes after being defogged by the image defogging algorithm.

[0004] In recent years, GPU chip technology has developed rapidly, and the computing power of GPU has gradually been applied in the fields of artificial intelligence and image processing. It is of great significance to realize real-time defogging of full HD or even ultra-HD foggy video images based on GPU computing power. However, due to the limitations of the usage environment, a single GPU is sometimes difficult to meet the actual usage needs. For a large amount of video image data, a reasonable design of the CPU and GPU collaborative operation method can maximize the real-time defogging of foggy video images. For CPU and GPU collaborative acceleration, the general method is to copy data from main memory to video memory, and the data in video memory is copied back to main memory for display after GPU acceleration processing. The copying of data from main memory to video memory and from video memory to main memory is very time-consuming, which seriously affects the GPU acceleration performance.

[0005] Therefore, the image dehazing algorithm is highly complex, and the data transmission between the CPU and GPU is very time-consuming when the image data volume is large. Real-time dehazing and stable display are key requirements for engineering applications. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a real-time defogging method and system for video images based on multi-GPU acceleration, aiming to solve the technical problems of high complexity, high time consumption, unstable display and inability to achieve real-time processing in existing technical solutions.

[0007] The present invention adopts the following technical solution:

[0008] On the one hand, the real-time defogging method for video images based on multi-GPU acceleration comprises the following steps:

[0009] Step S1, the camera collects foggy images, and sends the foggy images to each GPU in a loop in frame order;

[0010] Step S2: for each GPU, performing image defogging algorithm processing on the received foggy image data in a parallel acceleration manner to achieve image defogging;

[0011] Step S3: directly storing the defogging image data in the video memory, and realizing stable display of the defogging video image through a double buffering method.

[0012] On the other hand, the video image real-time defogging system based on multi-GPU acceleration includes:

[0013] The acquisition and sending module is used for the camera to acquire foggy images and send the foggy images to each GPU in a loop in frame order;

[0014] The parallel image defogging module is used to perform image defogging algorithm processing on the received foggy image data in a parallel acceleration manner for each GPU to achieve image defogging;

[0015] The double buffer display module is used to store the defogging image data directly in the video memory, and realize the stable display of the defogging video image through the double buffer method.

[0016] The beneficial effects of the present invention are as follows: the present invention aims at the defogging problem of a large amount of video image data, rationally designs the CPU and GPU collaborative operation method, the high parallel computing part is executed on the GPU side, the low parallel computing part is executed on the CPU side, and the image data after multi-GPU defogging is directly stored in the video memory, which reduces the time consumption of moving from the video memory to the main memory, and can improve the defogging performance of foggy video images as much as possible; in addition, the image data sent to the display and the image data currently being displayed are placed separately in two buffers, which ensures the stable display of the video image after defogging, and can effectively improve customer satisfaction. The real-time defogging method and system for video images based on multi-GPU acceleration provided by the present invention can meet the real-time defogging of full HD or even ultra-HD foggy video images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an overall flow chart of a real-time video image defogging method based on multi-GPU acceleration provided by an embodiment of the present invention;

[0018] Figure 2 It is a specific flow chart of the real-time defogging method for video images;

[0019] Figure 3 is a specific flow chart of step S1;

[0020] Figure 4 This is a schematic diagram of the execution process of the defogging algorithm, taking the dark channel prior defogging algorithm as an example;

[0021] Figure 5 This is a schematic diagram of the execution flow process of the double buffering method. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0023] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0024] Embodiment 1:

[0025] like Figure 1 , 2 As shown, the real-time defogging method for video images based on multi-GPU acceleration provided in this embodiment includes the following steps:

[0026] Step S1: The camera collects foggy images and sends the foggy images to each GPU in a loop in frame order.

[0027] First, build the high-resolution camera acquisition hardware and software environment, and set the camera exposure time according to actual needs. The camera starts to collect foggy images and transmits them to different GPUs in turn. For example, if there are n GPUs in total, the first frame of image data is sent to the first GPU, the second frame of image data is sent to the second GPU, the nth frame of image data is sent to the nth GPU, and the n+1th frame of image data is sent to the first GPU, and so on. Figure 3 As shown, the specific process of this step is as follows:

[0028] S11, after the camera is initialized, the foggy image is collected and the foggy image data is stored in the main memory;

[0029] S12, creating a thread for each GPU, calling the parallel acceleration API interface through the thread, and setting the GPU that executes the parallel acceleration image defogging algorithm;

[0030] S13, determining the target GPU corresponding to the foggy image data of the current frame in frame order, allocating space in the target GPU video memory, and copying the image data stored in the main memory to the video memory.

[0031] In this embodiment, after the camera captures a frame of image, the image data is saved in the main memory. Each GPU has a corresponding thread. Since the image data needs to be sent to the corresponding GPU in the order of frames, the threads can be called in sequence. The GPU that currently executes the parallel accelerated image defogging algorithm is called through the API interface, such as through OpenCL or CUDA. Find the GPU corresponding to the foggy image data of the current frame, which is defined as the target GPU here, and then allocate space in the video memory according to the size, and copy the image data of the current frame in the main memory from the main memory to the video memory.

[0032] Step S2: For each GPU, the received foggy image data is processed by an image defogging algorithm in a parallel acceleration manner to achieve image defogging.

[0033] There are many image defogging algorithms, and this embodiment does not limit the specific usage. Regardless of the specific image defogging algorithm used, this embodiment first divides the execution process of the algorithm into multiple calculation parts, and then executes the processing through the GPU or CPU according to the degree of parallelism of each calculation part. The specific process of GPU performing image defogging algorithm processing is as follows:

[0034] S21, dividing the various calculation parts of the image defogging algorithm according to the degree of parallelism, and executing the various calculation parts in sequence.

[0035] For example, the image defogging algorithm uses a dark channel prior defogging algorithm. The calculation steps of the dark channel prior defogging algorithm include obtaining the dark channel of the foggy image, calculating the atmospheric light intensity, the initial transmittance, using guided filtering to refine the initial transmittance, and obtaining the defogging image data. Among them, obtaining the dark channel of the foggy image, calculating the initial transmittance, using guided filtering to refine the initial transmittance, and obtaining the defogging image data, these calculation parts have high parallelism and are placed on the GPU for parallel acceleration; obtaining the atmospheric light intensity, the calculation part has a small amount of data and needs to compare the size of the data, the calculation part has low parallelism and is executed on the CPU. The subsequent steps are executed in the order of the above calculation parts.

[0036] S22, if the currently executed calculation part is a part with high parallelism, the current calculation part is calculated by executing the kernel function on the GPU side, and after obtaining the calculation result, if the next calculation part is a part with low parallelism, the current calculation result is also copied from the GPU video memory to the main memory;

[0037] S23, if the currently executed calculation part is a part with low parallelism, the CPU side calculates the current calculation part in serial mode, and after obtaining the calculation result, if the next calculation part is a part with high parallelism, the current calculation result is also copied from the main memory to the GPU memory;

[0038] S24. For the last calculation part to be executed, if it is a part with low parallelism, the calculation result is copied from the main memory to the GPU video memory after being calculated in serial mode on the CPU side; if it is a part with high parallelism, the current calculation part is directly calculated on the GPU side by executing the kernel function; finally, the dehazed image data is obtained and the image data is stored in the video memory.

[0039] In this embodiment, the image defogging algorithm is not necessarily all based on GPU parallel acceleration, and may involve data interaction between GPU and CPU, optimize the collaborative workflow of GPU and CPU, and reduce the defogging time based on dark channel prior defogging algorithm. The high parallelism part of the image defogging algorithm is faster based on GPU, and the low parallelism part of the defogging algorithm is faster based on CPU. Still taking advantage of the dark channel prior defogging algorithm, the specific process of S22-S24 is as follows Figure 4 As shown:

[0040] After the foggy image data of the current frame is copied from the main memory to the video memory, the kernel function is first executed to obtain the dark channel of the foggy image on the GPU side, and then the obtained dark channel data is copied from the GPU video memory to the main memory; then the atmospheric light intensity is obtained in a serial manner on the CPU side, and the atmospheric light intensity data is copied from the main memory to the video memory; then, the kernel function is executed to obtain the initial transmittance on the GPU side; the kernel function is continued to be executed, and the guided filtering algorithm is used to refine the initial transmittance on the GPU side to obtain the refined transmittance; finally, the kernel function is continued to be executed to obtain the defogged image data on the GPU side, that is, the defogged image data of the current frame is obtained.

[0041] Step S3: directly storing the defogging image data in the video memory, and realizing stable display of the defogging video image through a double buffering method.

[0042] The dehazed image data is directly stored in the video memory and managed in a queue. Figure 5 As shown, in step S3, the specific process of achieving stable display of the defogging video image by the double buffering method is as follows:

[0043] S31, setting a queue head pointer and creating two buffers, namely a first buffer and a second buffer, wherein the head pointer points to the first frame of image data after defogging;

[0044] S32, according to the direction of the head pointer, the first frame image data after defogging is moved to the first buffer, and after the moving is completed, the base address of the first buffer is set, and the head pointer points to the second frame image data after defogging;

[0045] S33, when the image data of the first buffer starts to be displayed, the second frame image data after defogging is moved to the second buffer, and after the move is completed, the base address of the second buffer is set, and after each move is completed, the head pointer points to the next frame image data after defogging;

[0046] S34, when the image data of the first buffer is displayed, an interrupt is generated, and when the interrupt is received, the third frame of image data after defogging is moved to the first buffer. At this time, the display is switched to the image data in the second buffer according to the base address of the second buffer.

[0047] S35, cyclically moving the image data and switching the buffer display in this manner to achieve real-time, continuous and stable display of the image data.

[0048] This step creates two buffers, and reads image data by pointing to the current frame image through a pointer. The first frame of defogging image data is first moved to the first buffer through blt, and the buffer base address is set. When the defogging image data of the first buffer starts to be displayed, the second frame of defogging image data is moved to the second buffer through blt, and the base address is set after the move is completed; when the image display in the first buffer is completed, a vblank interrupt will be generated, and after receiving the vblank interrupt, the third frame of defogging image data is moved to the first buffer through blt; at this time, the second frame of defogging image data in the second buffer will be switched to display through the buffer base address. Through the above-mentioned buffer display, data movement, interruption after display and continued data movement, buffer switching display and other cyclic processes, the sending and display of image data are separated to avoid the problem of video screen flickering caused by sending and displaying image data at the same time.

[0049] The method of the embodiment of the present invention sequentially transmits the foggy image data collected by the camera to multiple GPUs; then, based on each GPU, the foggy image data is defogged by a parallel accelerated defogging algorithm; finally, the defogged image data is directly stored in the video memory, and the defogged video image is stably displayed by a double buffering method. The present invention allows high-resolution foggy video images to be defogged faster, and the defogged video image can be stably displayed based on the double buffering method, which ensures the display quality of the defogged video image while achieving excellent defogging performance.

[0050] Embodiment 2:

[0051] This embodiment provides a real-time video image defogging system based on multi-GPU acceleration, and the real-time video image defogging system includes:

[0052] The acquisition and sending module is used for the camera to acquire foggy images and send the foggy images to each GPU in a loop in frame order;

[0053] The parallel image defogging module is used to perform image defogging algorithm processing on the received foggy image data in a parallel acceleration manner for each GPU to achieve image defogging;

[0054] The double buffer display module is used to store the defogging image data directly in the video memory, and realize the stable display of the defogging video image through the double buffer method.

[0055] Wherein, the acquisition and sending module includes:

[0056] The acquisition storage unit is used to acquire foggy images after the camera is initialized and store the foggy image data in the main memory;

[0057] The innovative thread unit is used to create a thread for each GPU, through which the parallel acceleration API interface is called to set the GPU that executes the parallel acceleration image dehazing algorithm;

[0058] The allocation copy unit is used to determine the target GPU corresponding to the foggy image data of the current frame in frame order, allocate space in the target GPU video memory, and copy the image data stored in the main memory to the video memory.

[0059] Wherein, the parallel image defogging module includes:

[0060] A division unit, used to divide the various calculation parts of the image defogging algorithm according to the degree of parallelism, and execute the various calculation parts in sequence;

[0061] The execution unit is used for, if the currently executed calculation part is a part with high parallelism, then the current calculation part is calculated by executing the kernel function on the GPU side, and after obtaining the calculation result, if the next calculation part is a part with low parallelism, the current calculation result is also copied from the GPU video memory to the main memory; and if the currently executed calculation part is a part with low parallelism, the current calculation part is calculated in a serial manner on the CPU side, and after obtaining the calculation result, if the next calculation part is a part with high parallelism, the current calculation result is also copied from the main memory to the GPU video memory; and for the last calculation part to be executed, if it is a part with low parallelism, the calculation result is copied from the main memory to the GPU video memory after being calculated in a serial manner on the CPU side; if it is a part with high parallelism, the current calculation part is directly calculated by executing the kernel function on the GPU side; finally, the dehazed image data is obtained and the image data is stored in the video memory.

[0062] Wherein, the double buffer display module includes.

[0063] A creation unit is used to set a queue head pointer and create two buffers, namely a first buffer and a second buffer. First, the head pointer points to the first frame of image data after defogging;

[0064] The partition buffer display unit is used to move the first frame image data after defogging to the first buffer according to the pointing of the head pointer, set the base address of the first buffer after the moving is completed, and the head pointer points to the second frame image data after defogging; when the image data of the first buffer starts to be displayed, the second frame image data after defogging is moved to the second buffer, and set the base address of the second buffer after the moving is completed. After each move, the head pointer points to the next frame image data after defogging; an interrupt is generated when the image data of the first buffer is displayed, and when the interrupt is received, the third frame image data after defogging is moved to the first buffer, and at this time, the display is switched to the image data in the second buffer according to the base address of the second buffer; the image data moving and buffer display switching are cyclically performed in this way to realize real-time, continuous and stable display of image data.

[0065] The functional modules and units of this embodiment correspond to the various steps in the first embodiment. Through this system, multi-GPU parallel acceleration of the image defogging algorithm is used to achieve real-time defogging of foggy video images; and after the GPU executes the parallel acceleration, the data will be saved in the video memory, reducing the time spent on copying the defogged image data from the video memory to the main memory; by optimizing the collaborative workflow of the GPU and the CPU, the defogging time based on the dark channel prior defogging algorithm is reduced, and the multi-GPU acceleration performance is improved by reducing the copying of the main video memory data; finally, a double buffer method is used to ensure the display quality of the defogged video image.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A real-time defogging method for video images based on multi-GPU acceleration, characterized in that: The real-time defogging method for video images comprises the following steps: Step S1, the camera collects foggy images, and sends the foggy images to each GPU in a loop in frame order; Step S2: for each GPU, performing image defogging algorithm processing on the received foggy image data in a parallel acceleration manner to achieve image defogging; Step S3: directly storing the defogging image data in the video memory, and realizing stable display of the defogging video image through a double buffering method.

2. The real-time defogging method for video images based on multi-GPU acceleration as claimed in claim 1, characterized in that: The specific process of step S1 is as follows: S11, after the camera is initialized, the foggy image is collected and the foggy image data is stored in the main memory; S12, creating a thread for each GPU, calling the parallel acceleration API interface through the thread, and setting the GPU that executes the parallel acceleration image defogging algorithm; S13, determining the target GPU corresponding to the foggy image data of the current frame in frame order, allocating space in the target GPU video memory, and copying the image data stored in the main memory to the video memory.

3. The real-time defogging method for video images based on multi-GPU acceleration as claimed in claim 2, characterized in that: In step S2, the specific process of GPU performing image defogging algorithm processing is as follows: S21, dividing the various calculation parts of the image defogging algorithm according to the degree of parallelism, and executing the various calculation parts in sequence; S22, if the currently executed calculation part is a part with high parallelism, the current calculation part is calculated by executing the kernel function on the GPU side, and after obtaining the calculation result, if the next calculation part is a part with low parallelism, the current calculation result is also copied from the GPU video memory to the main memory; S23, if the currently executed calculation part is a part with low parallelism, the CPU side calculates the current calculation part in serial mode, and after obtaining the calculation result, if the next calculation part is a part with high parallelism, the current calculation result is also copied from the main memory to the GPU memory; S24, for the last calculation part to be executed, if it is a part with low parallelism, after the CPU side uses serial calculation, the calculation result is copied from the main memory to the GPU memory; If it is a part with high parallelism, the current calculation part is directly calculated by executing the kernel function on the GPU side; finally, the defogged image data is obtained and the image data is saved in the video memory.

4. The real-time defogging method for video images based on multi-GPU acceleration as claimed in claim 3, characterized in that: The image defogging algorithm is a dark channel priori defogging algorithm. The calculation steps of the dark channel priori defogging algorithm include obtaining the dark channel of the foggy image, calculating the atmospheric light intensity, the initial transmittance, using guided filtering to perform fine processing on the initial transmittance, and obtaining the defogged image data. Among them, obtaining the dark channel of the foggy image, calculating the initial transmittance, using guided filtering to perform fine processing on the initial transmittance, and obtaining the defogged image data, these calculation parts have high parallelism and are placed on the GPU side for parallel acceleration; obtaining the atmospheric light intensity, the calculation part has a small amount of data and needs to compare the size of the data, the calculation part has low parallelism and is executed on the CPU side.

5. The real-time defogging method for video images based on multi-GPU acceleration as claimed in claim 4, characterized in that: In step S3, the defogging image data is directly stored in the video memory, and the image data is managed in a queue manner. The specific process of step S3 is as follows: S31, setting a queue head pointer and creating two buffers, namely a first buffer and a second buffer, wherein the head pointer points to the first frame of image data after defogging; S32, according to the direction of the head pointer, the first frame image data after defogging is moved to the first buffer, and after the moving is completed, the base address of the first buffer is set, and the head pointer points to the second frame image data after defogging; S33, when the image data of the first buffer starts to be displayed, the second frame image data after defogging is moved to the second buffer, and after the move is completed, the base address of the second buffer is set, and after each move is completed, the head pointer points to the next frame image data after defogging; S34, when the image data of the first buffer is displayed, an interrupt is generated, and when the interrupt is received, the third frame of image data after defogging is moved to the first buffer. At this time, the display is switched to the image data in the second buffer according to the base address of the second buffer. S35, cyclically moving the image data and switching the buffer display in this manner to achieve real-time, continuous and stable display of the image data.

6. A real-time video image defogging system based on multi-GPU acceleration, characterized in that: The video image real-time defogging system comprises: The acquisition and sending module is used for the camera to acquire foggy images and send the foggy images to each GPU in a loop in frame order; The parallel image defogging module is used to perform image defogging algorithm processing on the received foggy image data in a parallel acceleration manner for each GPU to achieve image defogging; The double buffer display module is used to store the defogging image data directly in the video memory, and realize the stable display of the defogging video image through the double buffer method.

7. The real-time video image defogging system based on multi-GPU acceleration as claimed in claim 6, characterized in that: The acquisition and transmission module comprises: The acquisition storage unit is used to acquire foggy images after the camera is initialized and store the foggy image data in the main memory; The innovative thread unit is used to create a thread for each GPU, through which the parallel acceleration API interface is called to set the GPU that executes the parallel acceleration image dehazing algorithm; The allocation copy unit is used to determine the target GPU corresponding to the foggy image data of the current frame in frame order, allocate space in the target GPU video memory, and copy the image data stored in the main memory to the video memory.

8. The real-time video image defogging system based on multi-GPU acceleration as claimed in claim 7, characterized in that: The parallel image defogging module comprises: A division unit, used to divide the various calculation parts of the image defogging algorithm according to the degree of parallelism, and execute the various calculation parts in sequence; The execution unit is used for, if the currently executed calculation part is a part with high parallelism, then the current calculation part is calculated by executing the kernel function on the GPU side, and after obtaining the calculation result, if the next calculation part is a part with low parallelism, the current calculation result is also copied from the GPU video memory to the main memory; and if the currently executed calculation part is a part with low parallelism, the current calculation part is calculated in a serial manner on the CPU side, and after obtaining the calculation result, if the next calculation part is a part with high parallelism, the current calculation result is also copied from the main memory to the GPU video memory; and for the last calculation part to be executed, if it is a part with low parallelism, the calculation result is copied from the main memory to the GPU video memory after being calculated in a serial manner on the CPU side; if it is a part with high parallelism, the current calculation part is directly calculated by executing the kernel function on the GPU side; finally, the dehazed image data is obtained and the image data is stored in the video memory.

9. The real-time video image defogging system based on multi-GPU acceleration as claimed in claim 8, characterized in that: The double buffer display module includes. A creation unit is used to set a queue head pointer and create two buffers, namely a first buffer and a second buffer. First, the head pointer points to the first frame of image data after defogging; The partition buffer display unit is used to move the first frame image data after defogging to the first buffer according to the pointing of the head pointer, set the base address of the first buffer after the moving is completed, and the head pointer points to the second frame image data after defogging; when the image data of the first buffer starts to be displayed, the second frame image data after defogging is moved to the second buffer, and set the base address of the second buffer after the moving is completed. After each move, the head pointer points to the next frame image data after defogging; an interrupt is generated when the image data of the first buffer is displayed, and when the interrupt is received, the third frame image data after defogging is moved to the first buffer, and at this time, the display is switched to the image data in the second buffer according to the base address of the second buffer; the image data moving and buffer display switching are cyclically performed in this way to realize real-time, continuous and stable display of image data.

Images