Video defogging method and device based on three-dimensional point cloud matching and real scene reconstruction

Abstract

The application discloses a video defogging method and device based on three-dimensional point cloud matching and real scene reconstruction. The method comprises the following steps: selecting two pictures of different time periods of the same scene in a video, taking the former as a fog image and the latter as a reference image, and obtaining three-dimensional point clouds in the images; matching the two three-dimensional point clouds, finding the projection positions of the three-dimensional points matched successfully in the corresponding images, and obtaining camera poses by using the three-dimensional points matched successfully and the projection positions; calculating the transmissivity of the scene by using a depth image; and performing defogging on the images by using the obtained transmissivity. The method can effectively remove the fog in the video and restore the images of the real scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a video defogging method and device based on three-dimensional point cloud matching and real scene reconstruction. BACKGROUND

[0002] Under the influence of haze, video monitoring, remote sensing, automatic driving and many other practical applications are easily threatened, and advanced computer vision tasks such as detection and identification are difficult to complete. Therefore, defogging has become an increasingly important technology, and finding a simple and effective image defogging method is crucial for subsequent research in computer vision. SUMMARY

[0003] Therefore, in order to solve the above problems, the present application provides a video defogging method and device based on three-dimensional point cloud matching and real scene reconstruction.

[0004] According to a first aspect of the present application, a video defogging method based on three-dimensional point cloud matching and real scene reconstruction is provided, comprising the following steps:

[0005] A three-dimensional point cloud acquisition step selects two frames of images at different time periods under the same scene of a video, the former being a haze image and the latter being a reference image. Three-dimensional point clouds are extracted from the haze image, and three-dimensional point clouds of the same region in the reference image are extracted as a matching reference. Extracting three-dimensional point clouds from the haze image includes extracting depth information from the haze image to obtain original three-dimensional point clouds.

[0006] A matching step matches the two three-dimensional point clouds, finds the projection position of the matching successful three-dimensional point in its corresponding image, and obtains the camera pose using the matching successful three-dimensional point and its projection position.

[0007] A transmittance calculation step projects the reference image to the depth image plane through the intrinsic and extrinsic parameters of the camera to obtain a depth image, and calculates the transmittance of the scene through the depth image. The intrinsic and extrinsic parameters of the camera are obtained according to the camera pose.

[0008] A defogging step uses the obtained transmittance to defog the image to obtain a defogged image.

[0009] In some embodiments, the three-dimensional point clouds of the haze image and the reference image are obtained by a camera in the three-dimensional point cloud acquisition step.

[0010] In some embodiments, matching the two three-dimensional point clouds to find the projection position of the matching successful three-dimensional point in its corresponding image, and obtaining the camera pose using the matching successful three-dimensional point and the projection position, comprises:

[0011] The SIFT algorithm is used to construct a Gaussian pyramid to perform scale transformation on the haze image and the reference image; the Gaussian difference algorithm is used to detect feature points in the haze image and the reference image; the main direction of each feature point is calculated, and the gradient information of a local image is used to describe the features of each feature point in the two images; the feature points in the reference image and the haze image are matched, for each feature point in the reference image, the nearest feature point in the haze image is found, and then it is judged whether the matching is successful according to the set threshold; the projection position of the successfully matched feature point in the corresponding image is obtained; the mapping relationship between the camera coordinate system and the image coordinate system is established, and the conversion matrix between the camera coordinate system and the image coordinate system is calculated according to the successfully matched feature points and the projection position of the successfully matched feature points in the corresponding image; the pose of the camera is obtained through the conversion matrix; the pose of the camera is adjusted by minimizing the re-projection error, so that the projection error between the successfully matched feature points is minimized. The conversion matrix is calculated by the pnp algorithm.

[0012] In some embodiments, the reference image is projected onto the depth image plane through the intrinsic and extrinsic parameters of the camera to obtain a depth image, and the transmittance of the scene is calculated through the depth image, including:

[0013] Each pixel point in the reference image is mapped onto the depth image plane according to the projection matrix through the intrinsic and extrinsic parameters of the camera, and the depth value of the mapped pixel point is taken as the depth value of the corresponding pixel point in the depth image, to obtain a depth image; the beta coefficient is calculated using the Koschmieder atmospheric optical model; the transmittance is calculated through the relationship between the transmittance and the distance.

[0014] In some embodiments, the calculation formula of the Koschmieder atmospheric optical model is:

[0015] beta = (1 / d0) * ln(I0 / I1)

[0016] Wherein, beta is the beta coefficient, d0 is a reference distance under a standard atmospheric condition; I0 is the brightness value obtained when the camera observes the same scene without haze; I1 is the brightness value obtained when the camera observes the same scene with haze.

[0017] In some embodiments, the calculation formula for calculating the transmittance through the relationship between the transmittance and the distance is:

[0018] t(x) = exp(-beta * d(x))

[0019] Wherein, t(x) is the transmittance, beta is the beta coefficient, and d(x) is the distance between the camera and the object.

[0020] In some embodiments, the foggy image is dehazed by using the obtained transmittance to obtain a dehazed image, comprising:

[0021] The dark channel of the foggy image is obtained; the 0.1% pixels with the highest brightness in the dark channel image are selected, and the pixel area is recorded; the value of the point with the highest brightness in the corresponding position of the pixel area in the original foggy image is taken as the A value; the foggy image is dehazed by using the obtained transmittance to obtain a dehazed image.

[0022] In some embodiments, the formula of the dark channel of the foggy image is as follows:

[0023]

[0024] Wherein, J c represents each channel of the color image, Ω(x) represents a window centered on the pixel x, and c represents the channel of the image.

[0025] In some embodiments, the formula for dehazing the image by using the obtained transmittance is as follows:

[0026] J(x)=(I(x)-A(1-t(x))) / t(x)

[0027] Wherein t(x) is the transmittance, A is the global atmospheric light value, I(x) is the foggy image, and J(x) is the dehazed image.

[0028] The cameras are all the same stereo camera; the stereo camera has high precision and accuracy in obtaining three-dimensional point cloud, and can capture and process images at real-time or near real-time speed, so that three-dimensional point cloud data can be quickly generated.

[0029] According to the second aspect of the application, a video dehazing device based on three-dimensional point cloud matching and real scene reconstruction is provided, comprising:

[0030] The three-dimensional point cloud acquisition module is used for selecting two frames of pictures at different time periods of the same scene of the video, taking the former as a foggy image and the latter as a reference image, extracting three-dimensional point cloud from the foggy image, and extracting three-dimensional point cloud of the same region in the reference image as a matching reference;

[0031] The matching module is used for matching the two three-dimensional point clouds, finding the projection position of the three-dimensional point matched successfully in the corresponding image, and obtaining the camera pose by using the three-dimensional point matched successfully and the projection position;

[0032] The transmittance calculation module is used for projecting the reference image to the depth image plane by the intrinsic and extrinsic parameters of the camera to obtain a depth image, and calculating the transmittance of the scene by the depth image; the intrinsic and extrinsic parameters of the camera are obtained according to the camera pose;

[0033] The defogging module is configured to defog the image using the obtained transmittance to obtain a defogged image.

[0034] According to a third aspect of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of the above embodiments when executing the computer program.

[0035] According to a fourth aspect of the present application, a computer readable storage medium is provided, having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of the above embodiments.

[0036] The present application provides a new video defogging method, which can effectively remove the haze in the video and restore the image of the real scene. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of some embodiments of the video defogging method based on three-dimensional point cloud matching and real scene reconstruction of the present application;

[0038] Figure 2 is a structural schematic diagram of some embodiments of the video defogging device based on three-dimensional point cloud matching and real scene reconstruction of the present application;

[0039] Figure 3 is an internal structural diagram of a computer device for implementing some embodiments of the present application. DETAILED DESCRIPTION

[0040] Embodiments of the present application will be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present application are shown. This present application may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0041] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0042] The terminology used herein (including the technical and scientific terminology) has the same meaning as is commonly understood by one of ordinary skill in the art to which this disclosure belongs unless specifically defined otherwise. The terminology used herein should be interpreted as having a meaning that is consistent with its meaning in the context of the specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0043] Figure 1 Flowcharts showing some embodiments of the video defogging method based on three-dimensional point cloud matching and real scene reconstruction of the present application are shown.

[0044] As shown in Figure 1 The method comprises:

[0045] The three-dimensional point cloud obtaining step S102 selects two frames of pictures at different time periods under the same scene of a video, the former being a haze image and the latter being a reference image, extracts a three-dimensional point cloud from the haze image, and extracts a three-dimensional point cloud of the same region in the reference image as a matching reference;

[0046] The matching step S104 matches the two three-dimensional point clouds, finds the projection position of the three-dimensional point that has successfully matched in its corresponding image, and obtains the camera pose by using the three-dimensional point that has successfully matched and its projection position;

[0047] The transmittance calculating step S106 projects the reference image to the depth image plane by using the intrinsic and extrinsic parameters of the camera to obtain a depth image, and calculates the transmittance of the scene by using the depth image; the intrinsic and extrinsic parameters of the camera are obtained according to the camera pose;

[0048] The specific calculation method of the intrinsic and extrinsic parameters is as follows: after matching, the corresponding feature point pairs are found, a fundamental matrix is obtained, the fundamental matrix is converted into an essential matrix, the essential matrix is decomposed, and the intrinsic and extrinsic matrices of the camera are calculated by using the known corresponding projection points of the three-dimensional point in the two images; for the possible errors during the calculation, a non-linear optimization method or a re-projection error optimization method is used to improve the accuracy.

[0049] The defogging step S108 uses the obtained transmittance to defog the image to obtain a defogged image.

[0050] The three-dimensional point cloud obtaining step S102 specifically comprises:

[0051] The two frames of pictures at different time periods under the same scene of a video are selected, the former being a haze image and the latter being a reference image; and the three-dimensional point clouds in the haze image and the reference image are obtained by using a stereo camera.

[0052] The matching step S104 specifically comprises:

[0053] A Gaussian pyramid is constructed to scale the haze image (to-be-matched image) and the reference image; a Gaussian difference algorithm is used to detect key points in the two images; a main direction of each key point is calculated, and then gradient information of a local image is used to describe features of each key point in the two images; feature points in the reference image and the to-be-matched image are matched. For each feature point in the reference image, a nearest feature point in the to-be-matched image is found, and then whether the matching is successful is judged according to a certain threshold; some three-dimensional points with successful matching are selected, and their projection positions in the image are obtained; a mapping relationship between a camera coordinate system and an image coordinate system is established, and a conversion matrix between the camera coordinate system and the image coordinate system is calculated (pnp algorithm calculation) according to the three-dimensional points and their projection positions in the image; a pose of the camera is obtained through the conversion matrix; the pose of the camera is adjusted by minimizing a re-projection error, so that the projection error between the matching points is minimized.

[0054] The transmittance calculation step S106 specifically comprises:

[0055] Each pixel point in the reference image is mapped to a depth image plane according to a projection matrix through the intrinsic and extrinsic parameters of the camera, and a depth value of the mapped pixel point is taken as a depth value of a corresponding pixel point in the depth image, so as to obtain the depth image; a beta coefficient is calculated by using a Koschmieder atmospheric optical model; and the transmittance is calculated through the relationship between the transmittance and the distance.

[0056] Further, the Koschmieder atmospheric optical model formula is as follows:

[0057] beta=(1 / d0)*ln(I0 / I1)

[0058] Wherein, d0 is a reference distance under a standard atmospheric condition, and is usually taken as 10 kilometers; I0 is a brightness value obtained when a camera observes the same scene in the case of no haze; and I1 is a brightness value obtained when the camera observes the same scene in the case of existing haze.

[0059] Further, the relationship formula between the transmittance and the distance is as follows:

[0060] t(x)=exp(-beta*d(x))

[0061] Wherein, beta is an attenuation coefficient, which can be calculated according to the theory of atmospheric optics, and d(x) is a distance between the camera and an object.

[0062] The haze removing step S108 specifically comprises:

[0063] Dark channel of the fog image is acquired; 0.1% pixels with the highest brightness are selected from the dark channel image; the value of the point with the highest brightness in the original fog image is searched and taken as the value of A; and the fog is removed from each frame of image.

[0064] Further, the dark channel formula of the fog image is as follows:

[0065]

[0066] In the formula, J C represents each channel of the color image, and Omega (x) represents a window centered at pixel x.

[0067] The formula of the defogged image is as follows:

[0068] J (x) = (I (x) - A (1-t (x))) / t (x)

[0069] where t (x) is the transmittance, A is the global atmospheric light value, I (x) is the fog image, and J (x) is the defogged image.

[0070] Figure 2 The structural schematic diagram of some embodiments of the video defogging device based on three-dimensional point cloud matching and real scene reconstruction of the application is shown.

[0071] As Figure 2 shown, the video defogging device based on three-dimensional point cloud matching and real scene reconstruction in the embodiments comprises:

[0072] The three-dimensional point cloud acquisition module 100 is configured to select two frames of pictures at different time periods of the same scene of the video, take the former as a fog image and the latter as a reference image, extract three-dimensional point clouds from the fog image, and extract three-dimensional point clouds of the same region in the reference image as a matching reference;

[0073] The matching module 200 is configured to match the two three-dimensional point clouds, find the projection positions of the three-dimensional points with successful matching in their corresponding images, and obtain the camera pose by using the three-dimensional points with successful matching and their projection positions.

[0074] The transmittance calculation module 300 is configured to project the reference image to the depth image plane by using the intrinsic and extrinsic parameters of the camera to obtain a depth image, and calculate the transmittance of the scene by using the depth image; the intrinsic and extrinsic parameters of the camera are obtained according to the camera pose.

[0075] The defogging module 400 is configured to defog the image by using the obtained transmittance to obtain a defogged image. In some embodiments, the conversion rule further comprises: converting the flow interface into a sub-flow or starting an event.

[0076] The specific limitation of the device for video defogging based on three-dimensional point cloud matching and real scene reconstruction can refer to the limitation of the method for video defogging based on three-dimensional point cloud matching and real scene reconstruction, which will not be repeated here. Each module in the device for video defogging based on three-dimensional point cloud matching and real scene reconstruction can be realized by software, hardware, and a combination thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0077] The computer device can be a terminal, and an internal structure diagram thereof can be as shown in Figure 3 The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the method for video defogging based on three-dimensional point cloud matching and real scene reconstruction. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc. Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0078] The present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method for video defogging based on three-dimensional point cloud matching and real scene reconstruction.

[0079] Those skilled in the art can understand that all or part of the processes in the above-mentioned method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0080] So far, the embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

[0081] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the present application. Those skilled in the art should understand that the above embodiments can be modified or some technical features can be replaced equivalently without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A video defogging method based on three-dimensional point cloud matching and real scene reconstruction, characterized in that, Includes the following steps: The steps for acquiring 3D point clouds are as follows: Select two frames of images from different time periods in the same scene of the video. The former is used as the haze image and the latter is used as the reference image. Extract 3D point clouds from the haze image and extract 3D point clouds of the same area from the reference image as matching references. The matching step involves matching two 3D point clouds, finding the projection position of the successfully matched 3D point in its corresponding image, and obtaining the camera pose using the successfully matched 3D point and its projection position. The transmittance calculation step involves projecting a reference image onto a depth image plane using the camera's intrinsic and extrinsic parameters to obtain a depth image, and then calculating the scene's transmittance from the depth image. The camera's intrinsic and extrinsic parameters are calculated based on the camera's pose. The dehazing step uses the obtained transmittance to dehaze the image, resulting in a dehazed image. The process of matching two 3D point clouds, finding the projection position of the successfully matched 3D point in its corresponding image, and obtaining the camera pose using the successfully matched 3D point and its projection position includes: A Gaussian pyramid is constructed to scale the haze image and the reference image; the difference of Gaussians algorithm is used to detect feature points in the haze image and the reference image; the principal direction of each feature point is calculated, and the gradient information of the local image is used to describe the features of each feature point in the two images; feature points in the reference image and the haze image are matched. For each feature point in the reference image, the nearest feature point in the haze image is found, and the matching is determined according to a set threshold; the projection position of the successfully matched feature point in its corresponding image is obtained; the mapping relationship between the camera coordinate system and the image coordinate system is established, and the transformation matrix between the camera coordinate system and the image coordinate system is calculated based on the successfully matched feature points and their projection positions in their corresponding images; the camera pose is obtained through the transformation matrix; the camera pose is adjusted by minimizing the reprojection error to minimize the projection error between the successfully matched feature points.

2. The video dehazing method based on 3D point cloud matching and real scene reconstruction according to claim 1, characterized in that, The process of projecting a reference image onto a depth image plane using the camera's intrinsic and extrinsic parameters to obtain a depth image, and then calculating the scene's transmittance from the depth image, includes: Using the camera's intrinsic and extrinsic parameters, each pixel in the reference image is mapped onto the depth image plane according to the projection matrix, and the depth value of the mapped pixel is used as the depth value of the corresponding pixel in the depth image to obtain the depth image; the beta coefficient is calculated using Koschmieder's atmospheric optical model; and the transmittance is calculated through the relationship between transmittance and distance.

3. The video dehazing method based on 3D point cloud matching and real scene reconstruction according to claim 2, characterized in that, The calculation formula for Koschmieder's atmospheric optical model is as follows: beta = (1 / d0) * ln(I0 / I1) Where beta is the beta coefficient, d0 is a reference distance under standard atmospheric conditions, I0 is the brightness value obtained by the camera when observing the same scene in the absence of fog, and I1 is the brightness value obtained by the camera when observing the same scene in the presence of fog.

4. The video dehazing method based on 3D point cloud matching and real scene reconstruction according to claim 2, characterized in that, The formula for calculating transmittance based on the relationship between transmittance and distance is as follows: t(x)=exp(-beta*d(x)) Where t(x) is the transmittance, beta is the beta coefficient, and d(x) is the distance between the camera and the object.

5. The video dehazing method based on 3D point cloud matching and real scene reconstruction according to claim 1, characterized in that, The process of dehazing the image using the obtained transmittance to obtain a dehazed image includes: Obtain the dark channel of the hazy image; select the top 0.1% of pixels from the dark channel image according to their brightness and record the pixel region; find the value of the point with the highest brightness in the corresponding position of the pixel region in the original hazy image and use it as the global atmospheric light value A; use the obtained transmittance to dehaze the image and obtain the dehazed image.

6. The video dehazing method based on 3D point cloud matching and real scene reconstruction according to claim 5, characterized in that, The formula for the dark channel of the hazy image is as follows: where J c represents each channel of a color image, Ω(x) represents a window centered at pixel x, and c represents a channel of the image.

7. The video dehazing method based on 3D point cloud matching and real scene reconstruction according to claim 5, characterized in that, The formula for dehazing the image using the obtained transmittance is as follows: J(x)=(I(x)-A(1-t(x))) / t(x) Where t(x) is the transmittance, A is the global atmospheric light value, I(x) is the foggy image, and J(x) is the defogging image.

8. An apparatus for video defogging based on three-dimensional point cloud matching and real scene reconstruction, characterized in that, include: The 3D point cloud acquisition module is used to select two frames of images from different time periods in the same scene of a video. The former is used as a haze image and the latter is used as a reference image. The module extracts 3D point clouds from the haze image and extracts 3D point clouds of the same area from the reference image as a matching reference. The matching module is used to match two 3D point clouds, find the projection position of the successfully matched 3D point in its corresponding image, and obtain the camera pose using the successfully matched 3D point and its projection position. The transmittance calculation module is used to project a reference image onto a depth image plane using the camera's intrinsic and extrinsic parameters to obtain a depth image, and then calculate the transmittance of the scene using the depth image; the camera's intrinsic and extrinsic parameters are calculated based on the camera pose. The dehazing module is used to dehaze the image using the obtained transmittance to obtain a dehazed image. The process of matching two 3D point clouds, finding the projection position of the successfully matched 3D point in its corresponding image, and obtaining the camera pose using the successfully matched 3D point and its projection position includes: A Gaussian pyramid is constructed to scale the haze image and the reference image; the difference of Gaussians algorithm is used to detect feature points in the haze image and the reference image; the principal direction of each feature point is calculated, and the gradient information of the local image is used to describe the features of each feature point in the two images; feature points in the reference image and the haze image are matched. For each feature point in the reference image, the nearest feature point in the haze image is found, and the matching is determined according to a set threshold; the projection position of the successfully matched feature point in its corresponding image is obtained; the mapping relationship between the camera coordinate system and the image coordinate system is established, and the transformation matrix between the camera coordinate system and the image coordinate system is calculated based on the successfully matched feature points and their projection positions in their corresponding images; the camera pose is obtained through the transformation matrix; the camera pose is adjusted by minimizing the reprojection error to minimize the projection error between the successfully matched feature points.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images