A method for dehazing video images and a medical endoscope camera system

Abstract

This invention discloses a method for dehazing video images and a medical endoscope camera system. The method processes received video image frames sequentially. When a frame of the video is received, a dehazing algorithm is executed on the current frame using multiple dehazing parameters from the previous frame to obtain and output the dehazed current frame image. Multiple dehazing parameters of the current frame image are also obtained for use when executing the dehazing algorithm on the next frame. In this way, the video image can be dehazed frame by frame, ultimately achieving dehazing of the entire video without physical dehazing.

Description

Technical Field

[0001] This invention relates to the medical field, and more particularly to a method for dehazing video images and a medical endoscope camera system. Background Technology

[0002] Medical endoscopic imaging systems, as auxiliary imaging tools for examination, diagnosis, and treatment, greatly facilitate medical personnel. Currently, the application scenarios of medical endoscopic imaging systems are becoming increasingly diverse and complex. In certain special surgical scenarios, smoke and water mist can reduce the clarity of the images displayed by the endoscopic system, obstructing the surgeon's view and hindering the safe and rapid conduct of the surgery. To address these issues, many endoscopic systems currently employ physical methods to defog, which can temporarily alleviate the interference of fog on the surgery to some extent, but it still cannot fundamentally solve the problem. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of the present invention provide a method for dehazing video images and a medical endoscope camera system, aiming to dehaze video images in a non-physical manner.

[0004] One embodiment provides a method for dehazing video images, comprising:

[0005] Receive video;

[0006] Upon receiving the i-th frame of the video, a dehazing algorithm is performed on the i-th frame based on multiple dehazing parameters of the (i-1)-th frame to obtain and output the dehazed i-th frame. Multiple dehazing parameters of the i-th frame are also obtained for use when performing the dehazing algorithm on the (i+1)-th frame. These multiple dehazing parameters include: guide filter parameters, the target position of a preset number of brightest pixels in the transmittance image of the (i-1)-th frame, and global stray noise.

[0007] In one embodiment of the method, the step of performing a dehazing algorithm on the i-th frame image based on multiple dehazing parameters of the (i-1)-th frame image to obtain and output the dehazed i-th frame image, and also obtaining multiple dehazing parameters of the i-th frame image, including:

[0008] The first step of the dehazing algorithm processes the i-th frame image, and during the processing, the first-stage dehazing parameters of the i-th frame image are obtained, so that subsequent steps can use the first-stage dehazing parameters of the i-th frame image to process the (i+1)-th frame image; wherein, the dehazing algorithm includes n steps, where n is an integer greater than or equal to 3;

[0009] The second step of the dehazing algorithm is executed based on the first-stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image. During the processing, the second-stage dehazing parameters of the i-th frame image are obtained so that subsequent steps can use the second-stage dehazing parameters of the i-th frame image to process the (i+1)th frame image.

[0010] Based on the (j-1)th stage dehazing parameters of the (i-1)th frame image, the j-th step of the dehazing algorithm is executed to continue processing the i-th frame image. During the processing, the j-th stage dehazing parameters of the i-th frame image are obtained so that subsequent steps can use the j-th stage dehazing parameters of the i-th frame image to process the (i+1)th frame image; where j is an integer and ∈ [2, n-1].

[0011] Based on the (n-1)th stage dehazing parameters of the (i-1)th frame image, the nth step of the dehazing algorithm is executed to continue processing the i-th frame image, resulting in the dehazed i-th frame image, which is then output.

[0012] In one embodiment of the method, the first-stage dehazing parameters of the i-th frame image include guided filter parameters calculated from the dark channel image of the i-th frame image; the first step of executing the dehazing algorithm processes the i-th frame image, and the first-stage dehazing parameters of the i-th frame image are obtained during the processing, including:

[0013] After traversing the i-th frame image using the minimum value filtering method in the dehazing algorithm, the dark channel image of the i-th frame image is generated;

[0014] Guided filtering parameters are calculated based on the dark channel image, and mean filtering is applied to the dark channel image. The mean-filtered dark channel image is then processed to obtain multiple intermediate process images for guided filtering.

[0015] In one embodiment of the method, the second-stage dehazing parameters of the i-th frame image include: the target positions of a preset number of brightest pixels in the transmittance image of the i-th frame image; the second step of the dehazing algorithm based on the first-stage dehazing parameters of the (i-1)-th frame image continues to process the i-th frame image, and the second-stage dehazing parameters of the i-th frame image are obtained during the processing, including:

[0016] Obtain the guided filtering parameters calculated from the dark channel image of the (i-1)th frame image. Based on these guided filtering parameters, perform guided filtering on the dark channel image of the i-th frame image, the dark channel image of the i-th frame image after mean filtering, and the intermediate process image of the i-th frame image to obtain the transmittance image of the i-th frame image.

[0017] The brightness of the pixels in the transmittance image is statistically analyzed to obtain the target positions of a preset number of the brightest pixels.

[0018] In one embodiment of the method, the third-stage dehazing parameters of the i-th frame image include global spurious emissions; the j-th step of the dehazing algorithm, which executes the j-1-th stage dehazing parameters of the (i-1)-th frame image, continues to process the i-th frame image, and the j-th stage dehazing parameters of the i-th frame image are obtained during the processing, including:

[0019] Obtain the target position of a preset number of brightest pixels in the transmittance image of the (i-1)th frame;

[0020] Global spurious emissions are obtained by acquiring the pixel values ​​of the pixels at these target locations in the i-th frame image and calculating their average value.

[0021] In one embodiment of the method, n = 4, and the step of performing the nth step of the dehazing algorithm based on the (n-1)th stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image, obtaining the dehazed i-th frame image and outputting it, includes:

[0022] Obtain the global spurious signals of the (i-1)th frame image;

[0023] Based on the global spurious emissions and the transmittance image of the i-th frame, the i-th frame is dehazed to obtain the dehazed i-th frame image and output it.

[0024] One embodiment provides a medical endoscope camera system, including a camera and a processor, wherein a data connection is established between the camera and the processor, and the camera is used to acquire video of a target object;

[0025] The processor is used for:

[0026] When the i-th frame of the video of the target object is received, the dehazing algorithm is executed on the i-th frame according to multiple dehazing parameters of the (i-1)-th frame to obtain and output the dehazed i-th frame. Multiple dehazing parameters of the i-th frame are also obtained for use when the dehazing algorithm is executed on the (i+1)-th frame. The multiple dehazing parameters of the (i-1)-th frame include: guide filter parameters, target positions of a preset number of brightest pixels in the transmittance image of the (i-1)-th frame, and global spurious emissions.

[0027] In one embodiment of the medical endoscope imaging system, the processor includes n processing modules, which include a first processing module, a second processing module, ..., an nth processing module;

[0028] The first processing module is used to execute the first step of the dehazing algorithm to process the i-th frame image when it receives the i-th frame image. During the processing, the first-stage dehazing parameters of the i-th frame image are obtained, and the first-stage dehazing parameters of the i-th frame image are output to the second processing module so that the second processing module can use the first-stage dehazing parameters of the i-th frame image to process the (i+1)-th frame image. The dehazing algorithm includes n steps, and the n steps are executed sequentially by the n processing modules, where n is an integer greater than or equal to 3.

[0029] The second processing module is used to execute the second step of the dehazing algorithm based on the first-stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image. During the processing, the second-stage dehazing parameters of the i-th frame image are obtained, and the second-stage dehazing parameters of the i-th frame image are output to the third processing module so that the third processing module can use the second-stage dehazing parameters of the i-th frame image to process the (i+1)th frame image.

[0030] The j-th processing module is used to execute the j-th step of the dehazing algorithm based on the (j-1)-th stage dehazing parameters of the (i-1)-th frame image to continue processing the i-th frame image. During the processing, the j-th stage dehazing parameters of the i-th frame image are obtained, and the j-th stage dehazing parameters of the i-th frame image are output to the (j+1)-th processing module so that the (j+1)-th processing module can use the j-th stage dehazing parameters of the i-th frame image to process the (i+1)-th frame image; where j is an integer and ∈ [2, n-1];

[0031] The nth processing module is used to execute the nth step of the dehazing algorithm based on the (n-1)th stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image, obtain the dehazed i-th frame image, and output it.

[0032] In one embodiment of the medical endoscope camera system, the processor is a field-programmable gate array (FPGA).

[0033] One embodiment provides a computer-readable storage medium including a program that can be executed by a processor to implement the method described above.

[0034] The video image dehazing method and medical endoscope camera system provided in this invention process the received video image frames sequentially. When a frame of video is received, a dehazing algorithm is executed on the current frame using multiple dehazing parameters from the previous frame to obtain and output the dehazed current frame image. Multiple dehazing parameters of the current frame image are also obtained for use when the dehazing algorithm is executed on the next frame. In this way, the video image can be dehazed frame by frame, and the entire video can be dehazed without physical dehazing. Attached Figure Description

[0035] Figure 1 A flowchart of an embodiment of the video image dehazing method provided by the present invention;

[0036] Figure 2 This is a structural block diagram of an embodiment of the defogging device provided by the present invention;

[0037] Figure 3 A structural block diagram of an embodiment of the medical endoscope imaging system provided by the present invention;

[0038] Figure 4 for Figure 1 In the flowchart of step 2 of an embodiment;

[0039] Figure 5 A schematic diagram of the processing of a frame of image from input to output in the medical endoscope imaging system provided by the present invention;

[0040] Figure 6 A simplified diagram of the image processing process in the medical endoscope imaging system provided by the present invention;

[0041] Figure 7 The timing diagram of parameters and pixel stream of the first 5 frames of images provided by the present invention in the medical endoscope imaging system. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] This invention addresses the complexity and unreliability of physical dehazing methods by providing a non-physical approach to dehaze individual frames of a video. It also considers transmission timeliness and display smoothness, eliminating the need for caching multiple video frames. The following examples illustrate this in detail.

[0044] like Figure 1 As shown, the video image dehazing method provided by the present invention includes the following steps:

[0045] Step 1: Receive video. The video can be captured by various video acquisition devices (such as various cameras, webcams, etc.). Since the dehazing method provided by this invention can perform real-time frame-by-frame dehazing of the video, this embodiment will use real-time reception of video image frames as an example for explanation.

[0046] Step 2: Upon receiving the i-th frame of the video, perform a preset dehazing algorithm on the i-th frame based on multiple dehazing parameters from the (i-1)-th frame (e.g., the frame preceding the i-th frame), obtaining and outputting the dehazed i-th frame. The multiple dehazing parameters for the (i-1)-th frame are obtained by performing the preset dehazing algorithm. These parameters include: steering filter parameters, the target positions of a preset number of brightest pixels in the transmittance image of the (i-1)-th frame, and global clutter. Traditional dehazing algorithms typically process single images, storing the image to be processed, then using the dehazing algorithm to process the image to obtain the corresponding dehazing parameters, and finally using these parameters to dehaze the stored image. If this conventional method is used for video, multiple video frames need to be cached. If this conventional method is used for real-time dehazing of transmitted video, each received video frame needs to be cached, and the dehazing algorithm needs to be used to process the video frame to obtain dehazing parameters. These parameters are then used to dehaze the cached video frame. This method is unusable for processors with insufficient cache (such as FPGAs). Even for processors with sufficient cache, this method requires receiving and dehazing multiple video frames before transmitting the dehazed video image, which can cause stuttering and unsmoothness in the dehazed video. In this embodiment, the multiple dehazing parameters of the (i-1)th frame are all the parameters required for image dehazing in the dehazing algorithm. Because all dehazing parameters in this invention are from previous frames, dehazing can be performed while receiving image frames (image frames are data streams, so they can be processed in parts). After processing, the dehazed image frame is transmitted to the backend, thus eliminating the need to cache multiple frames. The i-th frame can be any frame in the video.

[0047] Based on multiple dehazing parameters of the (i-1)th frame image, a preset dehazing algorithm is executed on the i-th frame image, and multiple dehazing parameters of the i-th frame image are also obtained for use when the dehazing algorithm is executed on the (i+1)th frame image (such as the next frame image of the i-th frame image). This process is repeated, realizing real-time acquisition and real-time dehazing of video images without the need to cache multiple frames of video images.

[0048] The above method can be applied to various devices with data processing capabilities, such as... Figure 2 As shown, this device, which can be called a defogging device 20, has a processor that can execute a preset program to implement the aforementioned defogging method. The input of the defogging device 20 can be communicatively connected to the video acquisition device 10, and the output of the defogging device 20 can be connected to the display 30. The video acquisition device 10 can acquire (e.g., film) video of the target object in real time and transmit the video data stream to the defogging device 20 in real time. The defogging device 20 then... Figure 1The method processes the video data stream in real time, thereby outputting the dehazed video data stream to the display 30 in real time. The processor can be of various types with data processing capabilities, such as CPU, MCU, FPGA (Field Programmable Gate Array), etc. Since it does not require buffering multiple frames of video data, the processor can use an FPGA with a small cache.

[0049] As can be seen from the background technology, the above-mentioned defogging method is very suitable for application in the medical field, especially in medical endoscopic imaging systems, thus eliminating the need for physical defogging and facilitating doctors to examine lesions and perform surgeries. A specific embodiment will be described in detail below.

[0050] like Figure 3 As shown, the medical endoscope camera system provided by the present invention includes a camera 40 and a processor 50. A data connection is established between the camera 40 and the processor 50.

[0051] Camera 40 is used to capture video of the target object. It can be a camera used with various types of endoscopes.

[0052] The processor 50, upon receiving the i-th frame of a video of the target object, performs a preset dehazing algorithm on the i-th frame based on multiple dehazing parameters of the (i-1)-th frame, obtaining and outputting the dehazed i-th frame. It also obtains multiple dehazing parameters of the i-th frame for use when performing the dehazing algorithm on the (i+1)-th frame. These dehazing parameters include: steering filter parameters, the target position of a preset number of brightest pixels in the transmittance image of the (i-1)-th frame, and global clutter. In other words, the processor 50 is capable of... Figure 1 The method shown specifically involves processor 50 comprising n processing modules. These n processing modules are: first processing module 510, second processing module 520, ..., nth processing module 5n0. These n processing modules cooperate to implement... Figure 1 Step 2 in the process is as follows: Figure 4 As shown, it includes the following steps:

[0053] Step 21: When the first processing module 510 receives the i-th frame of the video, it executes the first step of the dehazing algorithm to process the i-th frame. During the processing, it obtains the first-stage dehazing parameters of the i-th frame and outputs these parameters to the second processing module 520, so that the second processing module 520 can use them to process the (i+1)-th frame. The dehazing algorithm includes n steps, which are executed sequentially by the n processing modules, where n is an integer greater than or equal to 3. In other words, this invention divides the dehazing algorithm into n steps (stages) for sequential processing, which aligns with the processing characteristics of video data streams. A conventional dehazing algorithm can be used.

[0054] Please refer to the following for details. Figure 5 The first-stage dehazing parameters of the i-th frame image include the guided filter parameters calculated from the dark channel image of the i-th frame image. The first processing module 510 traverses the i-th frame image using the minimum value filtering method in the dehazing algorithm and generates the dark channel image (dark pixel stream of the dark channel image) of the i-th frame image. For example, the first processing module 510 generates an initial dark channel image. The size and resolution of the dark channel image are the same as those of the i-th frame image, and the positions of the pixels correspond one-to-one. The initial dark channel image is blank, and it is necessary to determine the pixel value of each pixel. A first window of a preset size is set with pixel A of the i-th frame image as the center (the first window frames multiple pixels). The minimum value in the RGB channels within the first window is found, and the pixel value of pixel A' corresponding to pixel A in the initial dark channel image is set to this minimum value. That is, the pixel with the smallest pixel value in the first window centered on a pixel of the i-th frame image is taken as the pixel value of the pixel at the center of the first window in the dark channel image. This process is performed on each pixel. After all the first windows have completed this operation, each pixel in the dark channel image has a pixel value, and thus the dark channel image is obtained.

[0055] The first processing module 510 calculates the guided filtering parameters based on the dark channel image and performs mean filtering on the dark channel image. The mean-filtered dark channel image is then processed to obtain multiple intermediate process images for guided filtering. Specifically, the dark channel image Jdark calculates the maximum and minimum values ​​of the entire dark channel image using a maximum / minimum value selector, thereby calculating a guided filtering parameter. In this embodiment, gradient guided filtering is used, so the calculated parameter is the gradient guided filtering algorithm control parameter epsilon (epsilon is used to calculate parameters a and b in gradient guided filtering). Simultaneously, mean filtering (replacing the box filter in the classic dehazing algorithm) is performed to output the mean_I pixel stream. Meanwhile, I (the dark channel image Jdark) is first multiplied by itself to obtain I^2, and then mean filtering is performed on the I^2 pixel stream to obtain the mean_I^2 image. Finally, the mean_I^2 and mean_I are used to calculate the process variable pixel stream Var_I of the gradient guided filtering algorithm. The first processing module 510 also uses different convolution kernel sizes to solve for Var_I` through the above steps. Then, it uses Var_I and Var_I` to solve for the process variable pixel stream chi_I of the gradient-guided filtering algorithm. Then, it uses the minimum value selector to calculate the minimum value min(chi_I) of the chi_I pixel stream of a frame, and solves for the mean value mean(chi_I) of the chi_I pixel stream of a frame by accumulating the values. After the above calculations, after the input of the i-th frame image ends and before the input of the (i+1)-th frame image, it outputs the calculation results of parameters such as epsilon, min(chi_I), and mean(chi_I) to the second processing module 520. Before calculating the gradient guided filter parameters such as epsilon, min(chi_I), and mean(chi_I), the input i-th frame image is processed and output sequentially as three intermediate process images: chi_I, Var_I, and Mean_I, as well as four pixel streams: Jdark, chi_I, Var_I, Mean_I, and Jdark. (The guided filter parameters can only be obtained after the input of the i-th frame is completed, while the pixel streams start to be output during the input of the i-th frame. That is, the first processing module 510 receives, processes, and outputs the pixel streams at the same time. The pixel streams will flow into the second processing module 520 earlier than the calculated guided filter parameters.)

[0056] The first processing module 510 performs mean filtering on one dark channel image. Specifically, a preset second window can be set for each pixel of the dark channel image. The second window frames multiple pixels. The second window is centered on the pixel, and the pixel value of the center pixel of the second window is replaced with the average pixel value of all pixels within the second window. After each pixel undergoes this processing, mean filtering is completed. Specifically, the second window can be horizontally scrolled (translated) sequentially from the first pixel of the dark channel image, traversing all pixels and replacing the pixel value of each pixel with the average pixel value within the corresponding second window, thus completing the mean filtering of the entire frame image. The size of the second window can be the same as the size of the first window. While classic dehazing algorithms like box filters produce good edges, they are complex and resource-intensive. This invention, however, uses mean filtering, which can be performed simultaneously with the generation of the dark channel image. As seen in the dark channel image generation process described above, the dark channel image can be generated sequentially starting from the first pixel. Once a pixel value is obtained, it is split into two paths and transmitted forward. One path is transmitted as a pixel stream to the second processing module 520, while the other path undergoes mean filtering. This pixel stream processing eliminates the need to buffer multiple frames of images; in fact, only a few rows of pixels need to be buffered, making it very suitable for FPGA platforms.

[0057] Step 22: The second processing module 520 has previously received and saved the first-stage dehazing parameters of the (i-1)th frame image. Based on these parameters, the second processing module 520 executes the second step of the dehazing algorithm to continue processing the i-th frame image. During this process, it obtains the second-stage dehazing parameters of the i-th frame image and outputs them to the third processing module 530. The third processing module 530 then uses these parameters to process the (i+1)th frame image. The second processing module 520 saves the first-stage dehazing parameters (such as gradient-guided filtering parameters) received from the i-th frame image. It can replace the first-stage dehazing parameters of the (i-1)th frame image with these parameters for use in dehazing the next frame.

[0058] The processor 50 uses a pipelined approach. Figure 4The method shown processes video image frames (pixel streams) without buffering multiple frames. Because the pixel stream transmitted from the first processing module 510 arrives at the second processing module 520 before the guided filtering parameters, the second processing module 520 can only process the pixel stream using the dehazing parameters from previous image frames. Specifically, in this step, the second processing module 520 obtains the guided filtering parameters calculated from the dark channel image of the (i-1)th frame, which are the guided filtering parameters transmitted backward by the first processing module 510 when processing the i-th frame. Based on these guided filtering parameters, guided filtering is applied to the dark channel image Jdark of the i-th frame, the dark channel image Mean_I of the i-th frame after mean filtering, and the intermediate process images (chi_I and Var_I) of the i-th frame to obtain the transmittance image Gdark of the i-th frame. In this embodiment, gradient-guided filtering is specifically used. That is, the three gradient-guided filtering parameters epsilon, min(chi_I), and mean(chi_I), along with the four pixel streams Mean_I, Jdark, Var_I, and chi_I, together complete the gradient-guided filtering. Conventional gradient-guided filtering can be used, and this invention does not limit it.

[0059] The second processing module 520 statistically analyzes the brightness of pixels in the transmittance image Gdark to obtain the target positions of a preset number of brightest pixels. The preset number can be set as needed, such as preset number = video image resolution / 1000. For example, the preset number for a 1080P video image is 1080*1920 / 1000 = 2073, and for 4K it is 3840*2160 / 1000 = 8294. The second-stage dehazing parameters for the i-th frame image include: the target positions of the preset number of brightest pixels in the transmittance image of the i-th frame image.

[0060] Step 2j: The j-th processing module 5j0 continues processing the i-th frame image by executing the j-th step of the dehazing algorithm based on the (j-1)-th stage dehazing parameters of the i-th frame image. During the processing, the j-th stage dehazing parameters of the i-th frame image are obtained, and the j-th stage dehazing parameters of the i-th frame image are output to the (j+1)-th processing module so that the (j+1)-th processing module can use the j-th stage dehazing parameters of the i-th frame image to process the (i+1)-th frame image; where j is an integer and ∈ [2, n-1]. When j is 2, this step is step 2 above; when j is 3, it is step 3. This embodiment uses j=3 as an example for explanation. The third stage dehazing parameters of the i-th frame image include global spurious emissions.

[0061] The third processing module 530 has previously received and saved the second-stage dehazing parameters of the (i-1)th frame image. The third processing module 530 then obtains the target positions of a preset number of brightest pixels in the transmittance image of the (i-1)th frame image; that is, the positions of these pixels in the transmittance image are obtained by the second processing module 520 during the processing of the previous image frames. The image frames, transmittance images, dark channel images, and intermediate process images all have the same image size, therefore the pixels in these images correspond one-to-one.

[0062] The third processing module 530 obtains the pixel values ​​of the target locations in the i-th frame image and calculates their average to obtain global spurious emissions. That is, it uses the target locations in the (i-1)-th frame image to find the corresponding pixels in the i-th frame image, obtains their pixel values, and accumulates these pixel values ​​to calculate the average. Since the time interval between two frames is very short, the pixel values ​​of the same pixel location in the two frames are very similar. Therefore, using the target locations of multiple pixels in the previous frame image to obtain the pixel values ​​of the corresponding pixels in the current frame image is acceptable in terms of accuracy. After receiving the second-stage dehazing parameters of the i-th frame image (such as the target locations of a preset number of brightest pixels), the third processing module 530 saves them. It can replace the second-stage dehazing parameters of the (i-1)-th frame image with the second-stage dehazing parameters of the i-th frame image for use in the dehazing of the next frame image.

[0063] Step 2n: The nth processing module 5n0 executes the nth step of the dehazing algorithm based on the (n-1)th stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image, obtaining the dehazed i-th frame image and outputting it. In this embodiment, based on the pixel stream transmission time, n is 4, and this step may specifically include:

[0064] The fourth processing module 540 has previously received and saved the third-stage dehazing parameters of the (i-1)th frame image. The fourth processing module 540 acquires the global spurious signals of the (i-1)th frame image; based on these global spurious signals and the transmittance image of the i-th frame image, it performs dehazing processing on the i-th frame image to obtain the dehazed i-th frame image and outputs it to the display 30. The fourth processing module 540 saves the third-stage dehazing parameters (such as global spurious signals) received from the i-th frame image. It can replace the third-stage dehazing parameters of the (i-1)th frame image with the third-stage dehazing parameters of the i-th frame image for use in dehazing the next frame image.

[0065] The above steps are completed before the (i+1)th frame image is received. After the (i+1)th frame image is received, step 2 above is repeated, that is, i is changed to i+1, so as to achieve real-time dehazing frame by frame, and the video image seen on the display 30 will not be stuttered.

[0066] The image sensor of camera 40 captures images and transmits them to display 30 via a link, one pixel at a time. Once the resolution of one frame is completed, that frame is displayed. Since one frame must be transmitted before the next, this invention adds a processor 50 to this link. As the pixel stream flows through the processor 50, the processor 50 performs the aforementioned processing on the pixel stream—specifically, dehazing the pixel stream as it passes through the link—before it is displayed on the screen. Transmitting one frame takes a very long time, typically in the millisecond range, while the processing (algorithm) added by this invention takes a very short time, typically in the microsecond range, such as 20ns * 1920. Therefore, it does not affect the transmission of video images and does not cause video stuttering.

[0067] In summary, the dehazing method provided by this invention replaces the complex boxfilter with mean filtering, simplifying computation while maintaining the desired effect. The processor can be an FPGA (Field-Programmable Gate Array). The FPGA-based method for dehazing medical endoscopic images has a simple architecture, avoiding the need for additional DDR memory for image caching during algorithm implementation. The caching requirement for just a few rows of pixels can be met by the FPGA's built-in RAM. Furthermore, a complete version of the dark channel prior-based dehazing algorithm can be implemented on the FPGA hardware, resulting in significant dehazing effects with minimal chip resource consumption, and efficient and stable operation on the FPGA hardware platform.

[0068] In this embodiment, as Figure 6 As shown, after a pixel stream is input into the processor, it flows sequentially through processing modules 1, 2, 3, and 4. The pixel stream is composed of the concatenated pixels of the image frame and flows continuously. The parameters in the diagram (corresponding to the parameters calculated in each step above) are numerical values ​​that can only be calculated after all the pixels in a frame have flowed through. That is, the parameters are updated after a pixel stream ends. For the first few frames, the initial parameters are unknown and can be defaulted to 0, or replaced with 1 or any other number. The results calculated from the initial parameter values ​​are incorrect. Therefore, when n=4, the first 3 frames cannot achieve the dehazing effect, while subsequent frames can. This will be explained in detail below.

[0069] When the pixel stream of the first frame image is input to the processor, it flows through various processing modules before finally reaching the output port. First, let's analyze the flow through the first processing module. After the entire pixel stream of the first frame image has passed through the first processing module, the correct parameter 'a' is obtained, which we'll call a1. However, before the first frame image is completely input, the value of parameter 'a' remains the default 0 (initial parameter value). Next, let's analyze the second processing module. When the pixel stream flows through the second processing module, the input pixel stream is correct, but parameter 'a' is the default 0. Therefore, when the pixel stream ends, the calculated parameter 'b' is incorrect. Similarly, parameter 'c' and the output image are also incorrect.

[0070] When the pixel stream of the second frame image is input to the processor, the analysis of the first processing module is the same as above. At the end, the correct parameter a2 is obtained, which will not be elaborated here. Analyzing the second processing module, since a1 was calculated at the end of the first frame image, the parameter b obtained by the second processing module at the end of the second frame image is correct, and is denoted as parameter b2. Analyzing the third processing module, since the parameter b calculated for the first frame image is incorrect, the parameter c calculated at the end of the second frame image is also incorrect. The output image is also incorrect (the second frame image without dehazing effect).

[0071] Therefore, it can be deduced that the correct output image was obtained when the 4th frame image was input.

[0072] As can be seen, the output image depends not only on the pixel stream of the current frame, but also on the parameter c of frame -1 (frame -1 is the previous frame, and so on). The parameter c of frame -1 depends on the parameter b of frame -2, and the parameter b of frame -2 depends on the parameter a of frame -3. Therefore, the output of the current image depends on the image data of the previous three frames. This also explains why the output of the first three frames when the image is first input has no dehazing effect (because there is no data from the first three frames).

[0073] Please refer to the following for details. Figure 5 and Figure 7The first frame image I1 of the video input to the first processing module is processed by minimum value filtering to calculate the dark channel image pixel stream Jdark1 of the first frame image I1. The output dark channel image Jdark1 is processed by a maximum and minimum value selector to calculate the maximum and minimum values ​​of the entire dark channel image and thus calculate the gradient-guided filtering algorithm control parameter epsilon1. At the same time, mean filtering is performed to output the mean_I1 pixel stream. Meanwhile, I1 is first multiplied by itself to obtain I1^2, and then mean filtering is performed on the I1^2 pixel stream to obtain the mean_I1^2 image. Then, the process variable pixel stream Var_I1 of the gradient-guided filtering algorithm is calculated using mean_I1^2 and mean_I1. Using different convolution kernel sizes, Var_I1` is calculated through the above steps. Then, the pixel stream chi_I1, the process variable of the gradient-guided filtering algorithm, is calculated using Var_I1 and Var_I1`. Next, the minimum value min(chi_I)1 of the chi_I1 pixel stream for a frame is calculated using a minimum value selector, and the mean value mean(chi_I)1 of the chi_I1 pixel stream for a frame is calculated by accumulating these values. After the above calculations, after the input of the first frame image I1 and before the input of the second frame image I2, the calculation results of parameters such as epsilon1, min(chi_I)1, and mean(chi_I)1 are output to the second processing module. Before calculating parameters such as epsilon1, min(chi_I)1, and mean(chi_I)1, the first frame image I1 has already output four pixel streams: chi_I1, Var_I1, Mean_I1, and Jdark1 (the parameters can only be obtained after the image input is complete, while the pixel streams begin to be output during the image input process and will flow into the second processing module earlier than the calculated parameters). These pixel streams will flow into the second processing module sequentially. At this time, the parameters epsilon, min(chi_I), and mean(chi_I) in the second processing module are the default initial parameters (the initial parameters are inappropriate and "incorrect," see...). Figure 7 (The dashed arrows indicate that these four pixel streams, along with the unupdated default epsilon, min(chi_I), and mean(chi_I) parameters, are used in a gradient-guided filtering operation to obtain the transmittance map Gdark1 pixel stream of the first frame image I1. However, since the epsilon, min(chi_I), and mean(chi_I) parameters have not been updated, the calculated Gdark1 is not the "correct" transmittance image Gdark (see...). Figure 7 (The dashed arrow), and so on, calculate the target position (coordinate set 1) of the preset number of "error" brightest pixels, global spurious A1, and then flow through the 4th processing module to output the undehazed image.

[0074] After the second frame image I2 arrives, like the first frame image I1, it first flows through the first processing module. After the input of the second frame image I2 is finished and before the input of the third frame image, the first processing module calculates parameters such as epsilon2, min(chi_I)2, and mean(chi_I)2, and outputs four pixel streams: chi_I2, Var_I2, Mean_I2, and Jdark2. These four pixel streams flow to the second frame image I2, and the three parameters epsilon1, min(chi_I)1, and mean(chi_I)1 calculated from the first frame image I1 are used to calculate the transmittance image Gdark2 of the second frame image I2 using a gradient-guided filtering algorithm. At this point, the calculated Gdark2 is the "correct" transmittance image Gdark. After calculating the Gdark2 pixel stream, histogram statistics are performed on the Gdark2 pixel stream. After the second frame image I2 is input and before the third frame image is input, the coordinate set 2 of the first preset number of brightest pixels is obtained by sorting. After the second frame image I2 is delayed for a period of time (preset duration), it is input to the third processing module. At this time, the input parameter coordinate set 1 is "incorrect", so the calculated global spurious A2 is also "incorrect". Finally, it flows through the fourth processing module and outputs an incorrect image that has not been dehazed.

[0075] After the arrival of the third frame image I3, similar to the first and second frames, it first flows through the first processing module. After the input of the third frame image is finished and before the input of the fourth frame image, the first processing module calculates parameters such as epsilon3, min(chi_I)3, and mean(chi_I)3, and outputs four pixel streams: chi_I3, Var_I3, Mean_I3, and Jdark3. These four pixel streams flow to the second processing module, where they are combined with the three parameters epsilon2, min(chi_I)2, and mean(chi_I)2 calculated from the second frame image and processed by a gradient-guided filtering algorithm to calculate the transmittance image Gdark3 of the third frame image. After calculating the Gdark3 pixel stream, histogram statistics are performed on the Gdark3 pixel stream. After the input of the third frame image is finished and before the input of the fourth frame image, the coordinate set 3 corresponding to the first preset number of brightest Gdark3 points is obtained by sorting. The third frame image is input to the third processing module after a certain delay. It outputs the pixel values ​​at the corresponding positions in coordinate set 2. These pixel values ​​are summed and averaged to obtain the global spurious A3. The A3 calculated at this time is the "correct" A3. The Gdark3 pixel stream flows through the fourth processing module and interacts with the erroneous global spurious A2 and the third frame image to output an erroneous image that has not been dehazed.

[0076] After the arrival of the fourth frame image I4, similar to the first, second, and third frames, it first flows through the first processing module. At the end of the fourth frame image input, parameters such as epsilon4, min(chi_I)4, and mean(chi_I)4 are calculated, and four pixel streams—chi_I4, Var_I4, Mean_I4, and Jdark4—are output sequentially. These four pixel streams flow to the second processing module, where, together with the three parameters epsilon3, min(chi_I)3, and mean(chi_I)3 calculated from the original third frame input image, a gradient-guided filtering algorithm is used to calculate the transmittance image Gdark4 of the fourth frame image. After calculating the Gdark4 pixel stream, histogram statistics are performed on the Gdark4 pixel stream. Before the arrival of the fifth frame image, the coordinate set 4 corresponding to the pre-set number of brightest Gdark4 points is obtained by sorting. The fourth frame image is input to the third processing module after a certain delay. The output coordinate set 3 outputs the pixel values ​​at the corresponding positions. These pixel values ​​are summed and averaged to obtain the global spurious signal A4. At this point, the calculated A4 is also the "correct" A4. Finally, the Gdark4 pixel stream flows through the fourth processing module and interacts with the "correct" global spurious signal A3 calculated from the third frame image and the fourth frame image to output the fourth frame dehazed image. In other words, the fourth frame of the output image begins to achieve the dehazing effect. Subsequent image frames, because the parameters of each processing module have been updated (there are no initial parameters) and are correct, all output image frames will achieve the dehazing effect.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dehazing video images, characterized in that, include: The system receives video, simultaneously receives image frames from the video, performs dehazing processing on each image frame, and then transmits the dehazed image frames to the backend to achieve frame-by-frame dehazing. After receiving the i-th frame of the video and before receiving the (i+1)-th frame, a dehazing algorithm is performed on the i-th frame based on multiple dehazing parameters of the (i-1)-th frame to obtain and output the dehazed i-th frame. Multiple dehazing parameters of the i-th frame are also obtained for use when performing the dehazing algorithm on the (i+1)-th frame. These multiple dehazing parameters include: steering filter parameters, target positions of a preset number of brightest pixels in the transmittance image of the (i-1)-th frame, and global clutter. The process of performing the dehazing algorithm on the i-th frame based on these multiple dehazing parameters, obtaining and outputting the dehazed i-th frame, and obtaining the multiple dehazing parameters of the i-th frame, includes: The first step of the dehazing algorithm processes the i-th frame image, and during the processing, the first-stage dehazing parameters of the i-th frame image are obtained, so that subsequent steps can use the first-stage dehazing parameters of the i-th frame image to process the (i+1)-th frame image; wherein, the dehazing algorithm includes n steps, where n is an integer greater than 3; The second step of the dehazing algorithm is executed based on the first-stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image. During the processing, the second-stage dehazing parameters of the i-th frame image are obtained so that subsequent steps can use the second-stage dehazing parameters of the i-th frame image to process the (i+1)th frame image. Based on the (j-1)th stage dehazing parameters of the (i-1)th frame image, the j-th step of the dehazing algorithm is executed to continue processing the i-th frame image. During the processing, the j-th stage dehazing parameters of the i-th frame image are obtained so that subsequent steps can use the j-th stage dehazing parameters of the i-th frame image to process the (i+1)th frame image; where j is an integer and ∈ [2, n-1]. Based on the (n-1)th stage dehazing parameters of the (i-1)th frame image, the nth step of the dehazing algorithm is executed to continue processing the i-th frame image, resulting in the dehazed i-th frame image, which is then output.

2. The method according to claim 1, characterized in that, The first-stage dehazing parameters of the i-th frame image include guided filter parameters calculated from the dark channel image of the i-th frame image; the first step of executing the dehazing algorithm processes the i-th frame image, and the first-stage dehazing parameters of the i-th frame image are obtained during the processing, including: After traversing the i-th frame image using the minimum value filtering method in the dehazing algorithm, the dark channel image of the i-th frame image is generated; Guided filtering parameters are calculated based on the dark channel image, and mean filtering is applied to the dark channel image. The mean-filtered dark channel image is then processed to obtain multiple intermediate process images for guided filtering.

3. The method according to claim 2, characterized in that, The second-stage dehazing parameters of the i-th frame image include: the target positions of a preset number of brightest pixels in the transmittance image of the i-th frame image; the second step of the dehazing algorithm based on the first-stage dehazing parameters of the (i-1)-th frame image continues to process the i-th frame image, and the second-stage dehazing parameters of the i-th frame image are obtained during the processing, including: Obtain the guided filtering parameters calculated from the dark channel image of the (i-1)th frame image. Based on these guided filtering parameters, perform guided filtering on the dark channel image of the i-th frame image, the dark channel image of the i-th frame image after mean filtering, and the intermediate process image of the i-th frame image to obtain the transmittance image of the i-th frame image. The brightness of the pixels in the transmittance image is statistically analyzed to obtain the target positions of a preset number of the brightest pixels.

4. The method according to claim 3, characterized in that, The third-stage dehazing parameters of the i-th frame image include global spurious emissions; the j-th step of the dehazing algorithm, which is executed based on the (j-1)-th stage dehazing parameters of the (i-1)-th frame image, continues to process the i-th frame image, and the j-th stage dehazing parameters of the i-th frame image are obtained during the processing, including: Obtain the target position of a preset number of brightest pixels in the transmittance image of the (i-1)th frame; Global spurious emissions are obtained by acquiring the pixel values ​​of the pixels at these target locations in the i-th frame image and calculating their average value.

5. The method according to claim 4, characterized in that, When n=4, the step of performing the nth step of the dehazing algorithm based on the (n-1)th stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image, obtaining the dehazed i-th frame image and outputting it, includes: Obtain the global spurious signals of the (i-1)th frame image; Based on the global spurious emissions and the transmittance image of the i-th frame, the i-th frame is dehazed to obtain the dehazed i-th frame image and output it.

6. A medical endoscope camera system, comprising a camera and a processor, wherein a data connection is established between the camera and the processor, and the camera is used to acquire video of a target object; Its features are, The processor is used for: The system receives a video of the target object, and while receiving image frames from the video, it performs dehazing processing on the image frames. After processing, it transmits the dehazed image frames to the backend to achieve frame-by-frame dehazing. After receiving the i-th frame of the video of the target object and before receiving the (i+1)-th frame, a dehazing algorithm is performed on the i-th frame based on multiple dehazing parameters of the (i-1)-th frame to obtain and output the dehazed i-th frame. Multiple dehazing parameters of the i-th frame are also obtained for use when the dehazing algorithm is executed on the (i+1)-th frame. These multiple dehazing parameters include: steering filter parameters, the target position of a preset number of brightest pixels in the transmittance image of the (i-1)-th frame, and global stray noise. The processor includes n processing modules, which include: a first processing module, a second processing module, ..., an nth processing module; The first processing module is used to execute the first step of the dehazing algorithm to process the i-th frame image when it receives the i-th frame image. During the processing, the first-stage dehazing parameters of the i-th frame image are obtained, and the first-stage dehazing parameters of the i-th frame image are output to the second processing module so that the second processing module can use the first-stage dehazing parameters of the i-th frame image to process the (i+1)-th frame image. The dehazing algorithm includes n steps, and the n steps are executed sequentially by the n processing modules, where n is an integer greater than 3. The second processing module is used to execute the second step of the dehazing algorithm based on the first-stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image. During the processing, the second-stage dehazing parameters of the i-th frame image are obtained, and the second-stage dehazing parameters of the i-th frame image are output to the third processing module so that the third processing module can use the second-stage dehazing parameters of the i-th frame image to process the (i+1)th frame image. The j-th processing module is used to execute the j-th step of the dehazing algorithm based on the (j-1)-th stage dehazing parameters of the (i-1)-th frame image to continue processing the i-th frame image. During the processing, the j-th stage dehazing parameters of the i-th frame image are obtained, and the j-th stage dehazing parameters of the i-th frame image are output to the (j+1)-th processing module so that the (j+1)-th processing module can use the j-th stage dehazing parameters of the i-th frame image to process the (i+1)-th frame image; where j is an integer and ∈ [2, n-1]; The nth processing module is used to execute the nth step of the dehazing algorithm based on the (n-1)th stage dehazing parameters of the (i-1)th frame image to continue processing the i-th frame image, obtain the dehazed i-th frame image, and output it.

7. The medical endoscope imaging system as described in claim 6, characterized in that, The processor is a field-programmable gate array (FPGA).

8. A computer-readable storage medium, characterized in that, Includes a program that can be executed by a processor to implement the method as described in any one of claims 1-5.

Images