Image processing method and device based on graphic application program interface

Abstract

The invention relates to an image processing method and device based on a graphic application program interface, and the method comprises the steps: carrying out the format conversion of received original image data based on the graphic application program interface, and obtaining a foggy texture image in an RGB format; calculating an ambient light texture and global atmospheric light based on a mean filtering result of the foggy texture image; on the basis of the ambient light texture and the global atmospheric light, a fogless texture image is recovered from the foggy texture image, the problem that the speed of a single image defogging algorithm is low is solved, on the basis of ensuring a certain defogging effect, the complexity of the defogging algorithm is reduced, and the defogging efficiency is improved through one-time simple mean filtering. And a high-quality fog-free texture image can be quickly recovered.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device based on a graphical application program interface. Background Art

[0002] Image processing is one of the key technologies in the fields of video surveillance, navigation control, target tracking, remote sensing and vehicle autonomous driving. However, the performance of these outdoor image processing systems is easily affected by bad weather and their performance will be significantly reduced under bad weather conditions. Especially in bad weather such as fog and haze, the reflected light of objects will be absorbed and scattered by suspended particles in the atmosphere, which will have a significant negative impact on the quality of outdoor imaging, which is reflected in reduced visibility, color distortion, reduced contrast and loss of details.

[0003] In the prior art, there are some theories and methods for foggy image sharpening algorithms. For example, there are methods that propose to obtain the structural information of the scene from multiple images of the same scene under different weather conditions, or use multiple polarization images to estimate the ambient light. The above methods require the input of information other than a single image, which limits the practical application. In the defogging algorithm based on a single image, there is a technique that proposes to use the dark color prior to obtain the transmittance, and then use the soft cutout algorithm to correct it, and finally restore the fog-free image. However, due to the complexity of the dark color prior calculation and the time-consuming nature of the soft cutout algorithm, the defogging speed of the algorithm is still not ideal.

[0004] There is currently no effective solution to the problem that the single image defogging algorithm in the relevant technology is slow. Summary of the invention

[0005] In this embodiment, a method and device for image processing based on a graphical application program interface are provided to solve the problem of slow speed of a single image defogging algorithm in the related art.

[0006] In a first aspect, in this embodiment, a method for image processing based on a graphical application program interface is provided, the method comprising: Based on the graphics application program interface, the received original image data is converted into a format to obtain a foggy texture image in RGB format; Based on the mean filtering result of the foggy texture image, the ambient light texture and the global atmospheric light are calculated; based on the ambient light texture and the global atmospheric light, a fog-free texture image is restored from the foggy texture image.

[0007] In some of the embodiments, based on a graphics application program interface, the received raw image data is converted into a format to obtain a foggy texture image in RGB format, including: Mapping the received raw image data in pixel format into texture data using a graphics application program interface; The texture data is input into the rendering pipeline of the graphics application to perform pixel conversion to obtain a foggy texture image in RGB format.

[0008] In some embodiments, the texture data is input into the rendering pipeline of the graphics application for pixel conversion to obtain a foggy texture image in RGB format, including: Mapping the texture data into three texture images: the three texture images include a luminance texture, a first chrominance texture, and a second chrominance texture; The three texture images are converted based on a color conversion formula, and the converted pixel points are filled into an image in RGB format to obtain a foggy texture image.

[0009] In some of the embodiments, calculating the ambient light texture and the global atmospheric light based on the mean filtering result of the foggy texture image includes: Using a compute shader to allocate a thread group, the minimum value of the three RGB color channels in each pixel of the foggy texture image is calculated in parallel, and a dark channel texture is formed based on the minimum value; Using the parallel computing capability of the computing shader, a mean filtering calculation is performed on the dark channel texture to obtain a mean filtered dark channel texture; The parallel computing capability of the computing shader is utilized to respectively calculate the computing environment light texture and the global atmospheric light based on the mean filtered dark channel texture.

[0010] In some embodiments, the parallel computing capability of the compute shader is utilized to respectively compute the computed ambient light texture and the global atmospheric light based on the mean filtered dark channel texture, including: Obtaining an ambient light texture based on the dark channel texture and the mean filtered dark channel texture; Based on the mean filtered dark channel texture and the foggy texture image, a global atmospheric light is obtained.

[0011] In some of the embodiments, obtaining an ambient light texture based on the dark channel texture and the mean filtered dark channel texture includes: Based on the parallel computing capability of the computing shader, the mean of all pixels in the mean-filtered dark channel texture is calculated to obtain a pixel mean; An ambient light texture is calculated based on the pixel mean, the dark channel texture and the mean-filtered dark channel texture.

[0012] In some of the embodiments, the original image data is obtained from a memory or a video memory.

[0013] In some embodiments, the method further comprises: The fog-free texture image is copied to screen memory for display.

[0014] In a second aspect, in this embodiment, a graphical application program interface-based image processing device is provided, the device comprising: A data conversion module is used to convert the format of the received original image data based on a graphical application program interface to obtain a foggy texture image in RGB format; The defogging calculation module is used to calculate the ambient light texture and the global atmospheric light based on the mean filtering result of the foggy texture image; and restore the fog-free texture image from the foggy texture image based on the ambient light texture and the global atmospheric light.

[0015] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the image processing method based on the graphical application program interface described in the first aspect is implemented.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image processing method based on a graphical application program interface described in the first aspect is implemented.

[0017] Compared with the related art, the image processing method and device based on the graphical application program interface provided in this embodiment, based on the graphical application program interface, performs format conversion on the received original image data to obtain a foggy texture image in RGB format; based on the mean filtering result of the foggy texture image, calculates the ambient light texture and the global atmospheric light; based on the ambient light texture and the global atmospheric light, restores the fog-free texture image from the foggy texture image, solves the problem of slow speed of the single image defogging algorithm, reduces the complexity of the defogging algorithm while ensuring a certain defogging effect, and quickly restores a high-quality fog-free texture image through a simple mean filtering.

[0018] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1A hardware structure block diagram of a terminal for an image processing method based on a graphical application program interface in an embodiment of the present application; Figure 2 A flowchart of an image processing method based on a graphical application program interface in an embodiment of the present application; Figure 3 A schematic diagram of brightness texture in a preferred embodiment of the present application; Figure 4 This is a schematic diagram of a first chromaticity texture in a preferred embodiment of the present application; Figure 5 A schematic diagram of a second chromaticity texture in a preferred embodiment of the present application; Figure 6 A schematic diagram of a foggy texture image in a preferred embodiment of the present application; Figure 7 A schematic diagram of a dark channel texture in a preferred embodiment of the present application; Figure 8 This is a schematic diagram of a mean-filtered dark channel texture in a preferred embodiment of the present application; Fig. 9 This is a schematic diagram of the ambient light texture in the preferred embodiment of the present application; Fig.10 A schematic diagram of a fog-free texture image in a preferred embodiment of the present application; Fig.11 This is a structural block diagram of an image processing method and device based on a graphical application programming interface in an embodiment of the present application.

[0020] Figure numerals: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, data conversion module; 20, defogging calculation module. DETAILED DESCRIPTION

[0021] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of the image processing method based on a graphical application program interface of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0024] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image processing method based on the graphical application program interface in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0025] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0026] In this embodiment, a method for image processing based on a graphical application program interface is provided. Figure 2 is a flowchart of the image processing method based on the graphical application program interface of this embodiment. Figure 2 As shown, the process includes the following steps: Step S210: Based on the graphical application program interface, the received original image data is converted into a format to obtain a foggy texture image in RGB format.

[0027] Specifically, a graphics application programming interface (API) is a set of functions and tools that allow developers to directly access and control computer graphics hardware (such as GPU) to achieve graphics rendering, image processing and visual effects. Common graphics APIs include DirectX, OpenGL and Vulkan. D3D11 (DirectX 3D11) is a set of graphics APIs (application programming interfaces) developed by Microsoft, mainly used for graphics and audio processing in game development and multimedia applications. DirectX 11 pays special attention to 3D graphics rendering, provides rich graphics processing functions and high-performance computing capabilities, and supports complex graphics effects and real-time rendering technology. Modern video surveillance systems often use personal computers as display terminals. These devices usually run Windows operating systems, and the DirectX3d11 graphics API of the Windows operating system has become a mainstream graphics API, which is widely supported by modern graphics cards and operating systems, providing a solid foundation for efficient image processing and hardware acceleration.

[0028] Step S220, based on the mean filtering result of the foggy texture image, calculate the ambient light texture and the global atmospheric light; based on the ambient light texture and the global atmospheric light, restore the fog-free texture image from the foggy texture image.

[0029] Specifically, the following physical model can be used to describe the impact of severe weather conditions such as haze on images. The model consists of two parts: the attenuation model and the ambient light model. The model expression is: H(x)=F(x)e -rd(x) +A(1-e -rd(x) ); Where x is the spatial coordinate of the image pixel; H(x) is the observed haze image; F(x) is the haze-free image to be restored; r represents the atmospheric scattering coefficient; d represents the depth of the scene; A is the global atmospheric light, which is usually assumed to be a global constant and has nothing to do with the spatial coordinate x. -rd(x) It is called direct attenuation, which means that the reflected light from the surface of the scene is attenuated due to scattering and other effects when it propagates in the medium; A(1-e -rd(x) ) is called ambient light or atmospheric veil, which causes the color and brightness of the image to shift.

[0030] For the above formula, replace A(1-e -rd(x) ), that is, L(x)=A(1-e-rd(x) ), we can get: ; It can be seen that it is necessary to estimate the ambient light L(x) and the global atmospheric light A from the original image data H(x) in order to restore the haze-free image F(x).

[0031] By utilizing the parallel computing feature of the graphics API - DirectX 3d11, namely the compute shader thread group feature, a simple mean filter is performed on a single image in the GPU to estimate the ambient light and global atmospheric light to implement a fast dehazing algorithm with a faster execution speed. The time required to process a 2560x1440 image is reduced from about 63ms to less than 1ms. This algorithm can meet the requirements of real-time application systems. At the same time, the dehazing effect is also better than the existing technology.

[0032] In this embodiment, based on a graphical application program interface, the format of the received original image data is converted to obtain a foggy texture image in RGB format; based on the mean filtering result of the foggy texture image, the ambient light texture and the global atmospheric light are calculated; based on the ambient light texture and the global atmospheric light, a fog-free texture image is restored from the foggy texture image, thereby solving the problem of slow speed of a single image defogging algorithm, reducing the complexity of the defogging algorithm while ensuring a certain defogging effect, and quickly restoring a high-quality fog-free texture image through a simple mean filtering.

[0033] In some of the embodiments, based on a graphics application program interface, the received raw image data is converted into a format to obtain a foggy texture image in RGB format, including: Step S211 : Mapping the received original image data in pixel format into texture data using a graphics application program interface.

[0034] Specifically, the raw image data is input from the system memory or video memory. The input format of the raw image data is not limited to the YUV format, but also supports the RGBA format, NV12 format, etc. The DirectX 3d11 graphics API is used to allocate the corresponding texture to the image according to the pixel format of the image, which is recorded as texture data. In DirectX 3d11, texture is an important graphics resource used to store image data and provide color and details for geometric graphics during the rendering process. Texture is a collection of image data. Through the memory mapping / copying of the operating system, the raw image data is transferred to the texture, that is, transferred to the DirectX3d11 runtime environment. At this time, these data can be sampled and used in the rendering pipeline through the shader, and finally rendered to the screen.

[0035] Step S212: input the texture data into the rendering pipeline of the graphics application to perform pixel conversion to obtain a foggy texture image in RGB format.

[0036] Specifically, the texture data is input into the rendering pipeline of DirectX 3d11. First, the vertex shader performs vertex processing to convert the two-dimensional image vertices from the model space to the screen space, and then goes through primitive assembly and rasterization. Finally, in the pixel shader stage, the color conversion formula is used to convert each pixel of the input texture into RGB format, and the RTT (Render To Texture) technology is used to fill the output converted pixels one by one into an RGB texture, waiting for fast dehazing processing. At this time, the RGB texture is the performance of the original image data on the DirectX 3d11 graphics API.

[0037] In this embodiment, the rendering pipeline of the graphics application program interface is used to convert the input geometric data (such as vertices, lines or triangles) into a final RGB image to facilitate subsequent calculation shader processing.

[0038] In some embodiments, the texture data is input into a rendering pipeline of a graphics application for pixel conversion to obtain a foggy texture image in RGB format, including: Step S212 - a, mapping the texture data into three texture images: the three texture images include a luminance texture, a first chrominance texture, and a second chrominance texture.

[0039] Step S212-b, converting the three texture images based on a color conversion formula, and filling the converted pixel points into an image in RGB format to obtain a foggy texture image.

[0040] Specifically, taking the texture data in YUV420P format as an example, it will be mapped into three textures in the pixel shader, namely a brightness texture (Y) and two chroma textures (the first chroma texture U and the second chroma texture V), see Figures 3 to 5 , Figure 3 is the brightness texture (Y), Figure 4 is the first chroma texture (U), Figure 5 is the second chroma texture (V). Use the YUV / RGB color space conversion formula to convert the three textures of the foggy image into RGB format, such as the BT709 color space conversion formula of the international standard for high-definition television: R=1.164×Y+1.596×Cr-222.912; G=1.164×Y-0.392×Cb-0.812×Cr+135.488; B=1.164×Y+2.016×Cb-276.672; Among them, R represents the red component, G represents the green component, B represents the blue component, Y represents the brightness component, Cb represents the blue color difference component, and Cr represents the red color difference component.

[0041] Output the converted pixels and fill them into an RGB texture one by one to complete the pixel conversion. Figure 6 The output fog texture image.

[0042] In some of the embodiments, the ambient light texture and the global atmospheric light are calculated based on the mean filtering result of the foggy texture image, including: Step S221, using the compute shader to allocate thread groups, parallelly calculate the minimum value of the three RGB color channels in each pixel of the foggy texture image, and form a dark channel texture based on the minimum value, such as Figure 7 shown.

[0043] Step S222, using the parallel computing capability of the compute shader, perform mean filtering calculation on the dark channel texture to obtain a mean filtered dark channel texture, such as Figure 8 shown.

[0044] Specifically, the traditional mean filter relies on a 2D convolution kernel. This embodiment splits the 2D convolution kernel into two 1D convolution kernels for filtering, which can significantly reduce the amount of calculation. Assuming that the image size is W×H, where W is the width and H is the height, and the convolution kernel size is K×K, each pixel needs to apply a K×K convolution kernel, so the amount of calculation for each pixel is K 2 , the computational complexity of the entire image is W×H×K 2 If the 2D convolution kernel is split into two 1D convolution kernels (one for horizontal filtering and the other for vertical filtering), the calculation process is divided into two stages: horizontal filtering and vertical filtering. The amount of calculation for each row is W×H×K, the amount of calculation for each column is W×H×K, and the amount of calculation for the entire image is 2×W×H×K, which significantly reduces the amount of calculation.

[0045] Step S223 , using the parallel computing capability of the compute shader, the computed ambient light texture and the global atmospheric light are respectively calculated based on the mean filtered dark channel texture.

[0046] In the field of video surveillance, the calculation of ambient light and global atmospheric light often relies on CPU calculation, and the processing time of each frame is about tens of milliseconds, which limits the application on multiple high-definition surveillance channels, that is, it is impossible to support multiple surveillance videos for fast dehazing processing at the same time. For each additional channel, the CPU computing power consumption increases linearly. In this embodiment, the parallel computing characteristics of the graphics API are used to transfer the fast dehazing algorithm to the GPU for processing, which greatly reduces the CPU burden. The number of high-definition (for example, resolution 2560x1440) surveillance channels with load increases from 2 to 9, which improves the computing efficiency.

[0047] In some embodiments, the parallel computing capability of the computing shader is utilized to respectively calculate the computing environment light texture and the global atmospheric light based on the mean filtered dark channel texture, including: Step S223-a, based on the dark channel texture and the mean filtered dark channel texture, obtain the ambient light texture, such as Fig. 9 shown.

[0048] Specifically, the ambient light texture L o The calculation formula is as follows: ; Among them, δ is the brightness adjustment parameter, M ave (x) is the mean filtered dark channel texture, and M(x) is the dark channel texture.

[0049] Step S223 - b, obtaining global atmospheric light based on mean filtering of the dark channel texture and the foggy texture image.

[0050] Specifically, the calculation formula of global atmospheric light A is as follows: ; Among them, M ave (x) is the mean filtered dark channel texture, H c (x) is a foggy texture image.

[0051] The global atmospheric light A estimation method of this embodiment avoids the irrationality of directly selecting the gray value of the brightest point in the input image as the atmospheric light value, while taking into account the rapidity of the algorithm.

[0052] In some embodiments, obtaining an ambient light texture based on a dark channel texture and a mean-filtered dark channel texture includes: Step S310, based on the parallel computing capability of the computing shader, the mean of all pixels in the mean-filtered dark channel texture is calculated to obtain a pixel mean.

[0053] Step S320, calculating and obtaining the ambient light texture based on the pixel mean, the dark channel texture and the mean filtered dark channel texture.

[0054] Specifically, the brightness adjustment parameter δ is calculated based on the pixel mean, and the ambient light texture is calculated based on the formula in the above step S223-a. Further, in order to improve the reliability of the calculation, an upper limit is set for the brightness adjustment parameter δ, for example, 0.9, and the calculation formula is as follows: δ=min(ρm av , 0.9); Among them, ρ is an adjustable parameter and , m av is the mean value of all elements in the mean-filtered dark channel texture. av is a value related to the overall grayscale distribution of the original image, so that ρm av It has the function of automatically adjusting the overall brightness of the defogging image. The darker the original image of the foggy image, the brighter the m av The lower the value, the greater the transmittance, which prevents the brightness of the dehazed image from being too dim, and vice versa.

[0055] In this embodiment, the brightness of the defogging image can be automatically adjusted to prevent the defogging image from being too dark or too bright.

[0056] In some of the embodiments, the method further includes: copying the fog-free texture image to a screen memory for display.

[0057] In this embodiment, the video data after hardware decoding does not need to be copied back to the system memory and then the CPU is used to perform fast dehazing processing on the data. Instead, the fast dehazing processing is performed directly in the video memory, and hardware rendering is performed directly after the processing is completed. This avoids the need to copy the hardware decoded data back to the memory for processing due to reliance on CPU processing, further optimizing processing efficiency and time consumption.

[0058] The present embodiment is described and illustrated below through preferred embodiments.

[0059] This preferred embodiment analyzes the physical model of foggy image degradation and gradually analyzes the transmittance estimation method. It uses only a simple mean filter to provide an estimation method for ambient light and global atmospheric light, and can automatically adjust the brightness of the defogged image to prevent the defogged image from being too dark or too bright. The specific derivation steps are as follows: 1. Physical model of foggy image degradation.

[0060] The model consists of two parts: attenuation model and ambient light model. The model expression is: H(x)=F(x)e -rd(x) +A(1-e -rd(x) ) formula (1); Where x is the spatial coordinate of the image pixel; H(x) is the observed haze image; F(x) is the haze-free image to be restored; r represents the atmospheric scattering coefficient; d represents the depth of the scene; A is the global atmospheric light, which is usually assumed to be a global constant and has nothing to do with the spatial coordinate x. -rd(x) It is called direct attenuation, which means that the reflected light from the surface of the scene is attenuated due to scattering and other effects when it propagates in the medium; A(1-e -rd(x) ) is called ambient light or atmospheric veil, which causes the color and brightness of the image to shift.

[0061] For formula (1), replace A(1-e -rd(x) ),Right now: L(x)=A(1-e -rd(x) ) Formula (2); Then formula (1) is rewritten as: Formula (3); It can be seen that it is necessary to estimate the ambient light L(x) and the global atmospheric light A from the foggy image H(x) in order to restore the fog-free image F(x).

[0062] 2. Estimate the ambient light.

[0063] Replace the exponential decay term e with the medium transmittance t(x) -rd(x) ,Right now: t(x)=e -rd(x) Formula (4); Then formula (1) is simplified to: H(x)=F(x)t(x)+A(1-t(x)) Formula (5); From formula (5), we can get: A(1-t(x))≤H(x) Formula (6); Find the minimum value of the three color channels on both sides of equation (6) and record: Formula (7); Formula (8); H in formula (7) c (x) represents the image of a color channel corresponding to the foggy image H(x), where the variable c is taken from the set of three colors {r, g, b}; A in formula (8) o is a constant representing the global atmospheric light for one color channel.

[0064] Then equation (6) can be rewritten as A(1-t(x))≤M(x), that is: Formula (9); From equation (4), we know that t(x) decreases as the scene depth d increases. The smaller the t(x) value, the greater the degree of attenuation of the scene reflected light at pixel x. When t(x) is 0, it means that the scene reflected light attenuates to 0, and H(x)=A. When H(x) is 1, it means that the scene reflected light has no attenuation at all, and H(x)=F(x). The transmittance t(x) has a trend similar to the scene depth, that is, it is smooth at the depth slow change and jumps at the depth discontinuity.

[0065] The preferred embodiment approximates the transmittance t(x) by the following method: Perform mean filtering on the right side of equation (9): Formula (10); Among them, S a represents the size of the mean filter window, Ω(x) is the S of pixel (x). a ×S a The result after mean filtering can reflect the general trend of t(x), but it differs from the real t(x) by a certain absolute value. Therefore, an offset value is added to the result of mean filtering and recorded as: Formula (11); Thus, a rough estimate of the transmittance is obtained: Formula (12); The last term on the right side of formula (12) is the offset value to be compensated, and .remember ,but: Formula (13); The value of δ is analyzed as follows: when δ is small, the transmittance is large, and the restored image has more residual fog and the image is whiter overall; when δ is large, the transmittance is small, and the restored image is darker overall. In order to prevent the image from being too dark or too bright after defogging, the following settings are used: Formula (14); Among them, ρ is an adjustable parameter and , m av is the mean of all elements in M(x). (Note that it is necessary to constrain m av , the value is between [0 1]. If the input image range is [0 1], then m avDirectly take the mean of all elements in M(x); if the input image range is [0 255], then it is necessary to calculate the mean of all elements in M(x) and then reduce it by 255 times accordingly. av is a value related to the overall grayscale distribution of the original image, so that It has the function of automatically adjusting the overall brightness of the dehazed image. The darker the original image, the brighter the m av The lower the value, the greater the transmittance, which prevents the brightness of the dehazed image from being too dim, and vice versa.

[0066] To ensure It is a positive value and not too small, so we need to set the upper limit of δ. Here, the upper limit is set to 0.9. Formula (15); Combining equations (9), (13) and (15), we can obtain: Formula (16); remember , from formula (2) we can get L(x)=A o (1-t(x)), then the ambient light (Air light) of a single color channel is estimated as follows: Formula (17).

[0067] 3. Estimate global atmospheric light.

[0068] In the prior art, the gray value of the point with the highest brightness in the input image is selected as the global atmospheric light A. Although the steps are simple, it is not very reasonable. Another prior art first selects the brightest 0.1% of pixels from the dark primary color channel, and then selects the maximum gray value of these pixels in the original input image as the global atmospheric light A. This processing method is more reasonable and effective, but the acquisition process of the dark primary color channel is relatively time-consuming.

[0069] This preferred embodiment will use the following method to estimate the global atmospheric light A. It is easy to see that the value range of the expression on the left side of the first equal sign in equation (10) should be [0 1]. Therefore, it can be concluded that A o ≥max(M ave (x)), in general, it is believed that: , then .

[0070] Therefore A o It can be expressed as: Formula (18); Among them, H c(x) represents a foggy image under a color channel; 0≤ε≤1. Since the exact value of ε is difficult to obtain, considering the algorithm's speed, ε is taken as 0.5, then: Formula (19).

[0071] This estimation method avoids the irrationality of directly selecting the gray value of the brightest point in the input image as the atmospheric light value, while also taking into account the speed of the algorithm.

[0072] 4. Restore fog-free images.

[0073] After estimating the ambient light L(x) and the global atmospheric light A, the physical model is used to restore the image F(x): Formula (20).

[0074] Based on the above principle of fast dehazing, this preferred embodiment uses GPU (Graphics Processing Unit) to implement, and uses its parallel processing capability to speed up the execution speed of the dehazing algorithm. The specific process is as follows: 1. Input the original foggy image from the system memory or video memory, and use the DirectX 3d11 graphics API to allocate a corresponding texture to the image according to the pixel format of the image, recorded as the input texture, that is, the foggy image data is mapped into the texture data of DirectX3d11.

[0075] 2. The input texture of the foggy image mapping is transmitted to the rendering pipeline of DirectX 3d11. First, the vertex processing of the vertex shader converts the 2D image vertices from the model space to the screen space, and then the primitives are assembled and rasterized. Finally, the color conversion formula is used in the pixel shader stage to convert each pixel of the input texture into RGB format, and the RTT (Render To Texture) technology is used to fill the converted pixels one by one into an RGB texture to obtain the foggy texture image H(x), see Figure 6 At this point, the RGB texture is the representation of the foggy image on the DirectX 3d11 graphics API.

[0076] 3. According to formula (7), the pixels of the fog texture image H(x) are traversed by computing the shader and assigning thread groups, the minimum value of the three color channels of RGB for each pixel is obtained, and the minimum value pixel is output to a blank texture to form a dark channel texture M(x), see Figure 7 .

[0077] 4. According to formula (11), the parallel computing characteristics of the compute shader are used to perform mean filtering on M(x) and output it to a blank texture to form a mean filtered dark channel texture M ave (x), see Figure 8.

[0078] 5. Using the parallel computing characteristics of the compute shader, calculate the mean value m of all pixels in M(x) av .

[0079] 6. According to formula (17), using M ave (x) and M(x) to calculate the ambient light L(x), and output the L(x) pixel to a blank texture to form the ambient light texture L(x), see Fig. 9 .

[0080] 7. According to formula (19), using M ave (x) and H(x) to obtain the global atmospheric light A.

[0081] 8. Using the parallel computing characteristics of the compute shader, according to formula (20), input the fog texture image H(x), the ambient light texture L(x) and the estimated global atmospheric light A, and output the fog-free texture F(x), see Fig.10 , and copies it to the screen memory area for display.

[0082] Considering that many modern video surveillance systems currently support hardware acceleration of video decoding, that is, using specialized hardware (such as GPU, dedicated decoding chips, etc.) to accelerate the video decoding process. Compared with software decoding, it can significantly improve decoding efficiency, reduce CPU burden, and achieve higher decoding quality and lower power consumption. In addition, modern video surveillance systems use personal computers as display terminals. These devices usually run Windows operating systems, among which the DirectX 3d11 graphics API has become the mainstream graphics API, which is widely supported by modern graphics cards and operating systems, providing a solid foundation for efficient image processing and hardware acceleration. This preferred embodiment uses a GPU (graphics processing unit) to implement the calculation of a fast dehazing algorithm based on the D3D11 graphics API fast dehazing method, and uses its parallel processing capabilities to speed up the execution of the dehazing algorithm. The specific technical advantages are as follows: (1) Fast processing speed: By utilizing the parallel computing feature of the graphics API DirectX 3d11, namely the compute shader thread group feature, a simple mean filter is performed on a single image in the GPU to estimate the ambient light and global atmospheric light to implement a fast dehazing algorithm. This algorithm has a faster execution speed and the time required to process a 2560x1440 image is reduced from about 63ms to less than 1ms.

[0083] (2) Large number of loads: By utilizing the parallel computing features of the graphics API DirectX 3d11, the fast dehazing algorithm is transferred to the GPU for processing, greatly reducing the CPU burden. The number of high-definition (resolution 2560x1440) monitoring channels of the load is increased from 2 to 9.

[0084] (3) Support zero copy: Currently, many Windows computers support zero copy technology for hardware decoding and rendering, which means that during the video decoding and image rendering process, the data does not need to go through additional memory copy steps, thereby improving overall performance and response speed. The fast dehazing algorithm implemented based on the D3D11 graphics API also supports zero copy, that is, the video data after hardware decoding does not need to be copied back to the system memory and then the CPU is used to perform fast dehazing on the data. Instead, the fast dehazing is performed directly in the video memory, and hardware rendering is performed directly after the processing is completed. This avoids the need to copy the hardware decoded data back to the memory for processing due to reliance on CPU processing, further optimizing processing efficiency and time consumption.

[0085] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0086] In this embodiment, a method and device for image processing based on a graphical application program interface is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0087] Fig.11 is a structural block diagram of the image processing method and device based on the graphical application program interface of this embodiment. Fig.11 As shown, the device includes: a data conversion module 10 and a defogging calculation module 20.

[0088] The data conversion module 10 is used to convert the format of the received original image data based on the graphical application program interface to obtain a foggy texture image in RGB format.

[0089] The defogging calculation module 20 is used to calculate the ambient light texture and the global atmospheric light based on the mean filtering result of the foggy texture image; and restore the fog-free texture image from the foggy texture image based on the ambient light texture and the global atmospheric light.

[0090] In some of the embodiments, the data conversion module 10 is also used to map the received original image data in pixel format into texture data using a graphics application program interface; the texture data is input into the rendering pipeline of the graphics application program for pixel conversion to obtain a foggy texture image in RGB format.

[0091] In some of the embodiments, the data conversion module 10 is also used to map the texture data into three texture images: the three texture images include a luminance texture, a first chrominance texture, and a second chrominance texture; the three texture images are converted based on a color conversion formula, and the converted pixel points are filled into an image in RGB format to obtain a foggy texture image.

[0092] In some of the embodiments, the defogging calculation module 20 is also used to use the computing shader to allocate thread groups, and to parallelly calculate the minimum value of the three RGB color channels in each pixel of the foggy texture image, and form a dark channel texture based on the minimum value; use the parallel computing capability of the computing shader to perform mean filtering on the dark channel texture to obtain the mean filtered dark channel texture; use the parallel computing capability of the computing shader to respectively calculate the ambient light texture and the global atmospheric light based on the mean filtered dark channel texture.

[0093] In some of the embodiments, the defogging calculation module 20 is further used to obtain the ambient light texture based on the dark channel texture and the mean filtered dark channel texture; and to obtain the global atmospheric light based on the mean filtered dark channel texture and the foggy texture image.

[0094] In some of the embodiments, the dehazing calculation module 20 is further used to calculate the mean of all pixels in the mean filtered dark channel texture based on the parallel computing capability of the calculation shader to obtain the pixel mean; and to calculate the ambient light texture based on the pixel mean, the dark channel texture and the mean filtered dark channel texture.

[0095] In some of the embodiments, the raw image data is obtained from a memory or a video memory.

[0096] In some of the embodiments, the method further includes: copying the fog-free texture image to a screen memory for display.

[0097] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0098] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0099] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0100] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0101] In addition, in combination with the image processing method based on the graphical application program interface provided in the above embodiments, a storage medium can also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the image processing methods based on the graphical application program interface in the above embodiments is implemented.

[0102] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0103] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0104] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0105] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. An image processing method based on a graphical application program interface, characterized in that: The method comprises: Based on the graphics application program interface, the received original image data is converted into a format to obtain a foggy texture image in RGB format; Based on the mean filtering result of the foggy texture image, the ambient light texture and the global atmospheric light are calculated; based on the ambient light texture and the global atmospheric light, a fog-free texture image is restored from the foggy texture image.

2. The image processing method based on the graphical application program interface according to claim 1, characterized in that: Based on the graphics application program interface, the received raw image data is converted into a format to obtain a foggy texture image in RGB format, including: Mapping the received raw image data in pixel format into texture data using a graphics application program interface; The texture data is input into the rendering pipeline of the graphics application to perform pixel conversion to obtain a foggy texture image in RGB format.

3. The image processing method based on the graphical application program interface according to claim 2, characterized in that: Inputting the texture data into the rendering pipeline of the graphics application to perform pixel conversion to obtain a foggy texture image in RGB format, including: Mapping the texture data into three texture images: the three texture images include a luminance texture, a first chrominance texture, and a second chrominance texture; The three texture images are converted based on a color conversion formula, and the converted pixel points are filled into an image in RGB format to obtain a foggy texture image.

4. The image processing method based on the graphical application program interface according to claim 1, characterized in that: Based on the mean filtering result of the foggy texture image, the ambient light texture and the global atmospheric light are calculated, including: Using a compute shader to allocate a thread group, the minimum value of the three RGB color channels in each pixel of the foggy texture image is calculated in parallel, and a dark channel texture is formed based on the minimum value; Using the parallel computing capability of the computing shader, a mean filtering calculation is performed on the dark channel texture to obtain a mean filtered dark channel texture; The parallel computing capability of the computing shader is utilized to respectively calculate the computing environment light texture and the global atmospheric light based on the mean filtered dark channel texture.

5. The image processing method based on graphical application program interface according to claim 4, characterized in that: Utilizing the parallel computing capability of the computing shader, computing the computing environment light texture and the global atmospheric light based on the mean filtered dark channel texture, including: Obtaining an ambient light texture based on the dark channel texture and the mean filtered dark channel texture; Based on the mean filtered dark channel texture and the foggy texture image, a global atmospheric light is obtained.

6. The image processing method based on the graphical application program interface according to claim 5, characterized in that: Based on the dark channel texture and the mean filtered dark channel texture, an ambient light texture is obtained, comprising: Based on the parallel computing capability of the computing shader, the mean of all pixels in the mean-filtered dark channel texture is calculated to obtain a pixel mean; An ambient light texture is calculated based on the pixel mean, the dark channel texture and the mean-filtered dark channel texture.

7. The image processing method based on graphical application program interface according to claim 1, characterized in that: The original image data is obtained from the memory or the video memory.

8. The image processing method based on graphical application program interface according to claim 1, characterized in that: The method further comprises: The fog-free texture image is copied to screen memory for display.

9. An image processing device based on a graphical application program interface, characterized in that: The device comprises: A data conversion module is used to convert the format of the received original image data based on a graphical application program interface to obtain a foggy texture image in RGB format; The defogging calculation module is used to calculate the ambient light texture and the global atmospheric light based on the mean filtering result of the foggy texture image; and restore the fog-free texture image from the foggy texture image based on the ambient light texture and the global atmospheric light.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images