An underwater image super-resolution reconstruction method and an underwater robot terminal
By combining optical and depth cameras, and utilizing the Canny algorithm and joint bilateral filters to reconstruct underwater images, the problems of underwater image blurring and noise were solved, achieving high-resolution underwater image reconstruction and improving image quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-23
- Publication Date
- 2026-03-27
AI Technical Summary
Underwater images are of poor quality, especially in environments with low visibility, uneven lighting, and scattering from suspended particles. The images acquired by imaging devices are blurry, have low contrast, and suffer from noise and bright spots. Existing methods are too demanding for real-time robot deployment and are difficult to effectively recover high-resolution images.
Underwater images and depth images are captured using an optical camera and a depth camera, respectively. The edge contours of the depth images are extracted using the Canny algorithm, and high-resolution images are reconstructed using a joint bilateral filter. The smooth contours of the depth images are then used as a guide for super-resolution reconstruction.
It achieves high-resolution reconstruction of underwater images under complex underwater lighting conditions, avoids texture copying errors, reduces jagged boundaries, preserves edge characteristics, and improves the resolution and quality of underwater images.
Smart Images
Figure CN115131207B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of underwater imaging, in particular to an underwater image super-resolution reconstruction method and an underwater robot terminal. BACKGROUND
[0002] Underwater image super-resolution and restoration is an important research problem, correcting and restoring distorted optical images to preserve pixel intensity, classic methods use hand-crafted filters to improve local contrast and enhance color stability, especially for defogging, color correction, water removal, etc. The inspiration of these methods comes from human visual perception theory, and mainly focuses on restoring background illumination and brightness reproduction. However, these methods are usually too computationally demanding for real-time robotic deployment, and dense depth and optical measurements are not always available in practical applications.
[0003] Underwater optical detection mainly relies on underwater images taken by underwater robots, and the quality of underwater images is closely related to the imaging system carried by underwater robots. Due to low visibility, uneven illumination, selective absorption of water to light and scattering of suspended particles in underwater environment, the underwater images obtained by imaging equipment are often dark and low in contrast. Backscattering in suspended particle scattering causes image blurring, the original color of the object surface is covered, and the local area of the object is not clear. In addition, due to the lack of light in deep water, artificial light sources are often added, which often leads to the existence of bright spots in the images taken in underwater environment, and the underwater degraded images also have noise and other problems. SUMMARY
[0004] The technical problem to be solved by the present application is to provide an underwater image super-resolution reconstruction method and an underwater robot terminal, which realizes the super-resolution reconstruction of underwater images, thereby enhancing the resolution of underwater images to obtain high-resolution underwater pictures.
[0005] To solve the above technical problems, the technical scheme adopted by the present application is:
[0006] An underwater image super-resolution reconstruction method, comprising the steps of:
[0007] S1, controlling an optical camera and a depth camera to respectively shoot original underwater images and depth images of the same target at the same time;
[0008] S2, using a Canny algorithm to extract the edge contour of the depth image;
[0009] S3, after denoising the original underwater image and aligning and correcting it with the edge contour of the depth image, a paired picture with a smooth contour is obtained;
[0010] S4, guided by the smooth contour, reconstructing high-resolution underwater image texture by using a joint bilateral filter to obtain a high-resolution underwater picture.
[0011] To solve the above technical problems, another technical solution adopted by the present application is:
[0012] An underwater robot terminal comprises an optical camera, a depth camera, a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
[0013] S1, controlling the optical camera and the depth camera to respectively capture original underwater images and depth images of the same target at the same time;
[0014] S2, extracting the edge contour of the depth image by using a Canny algorithm;
[0015] S3, after denoising the original underwater image and aligning and correcting it with the edge contour of the depth image, obtaining a paired picture with a smooth contour;
[0016] S4, guided by the smooth contour, reconstructing high-resolution underwater image texture by using a joint bilateral filter to obtain a high-resolution underwater picture.
[0017] The present application has the beneficial effect that the edge contour of the depth image is used to guide the super-resolution reconstruction of the underwater image, and the super-resolution problem is converted from texture guidance to edge detection based on the depth image, specifically, after denoising the original underwater image and aligning and correcting it with the edge contour, a paired picture is obtained by modifying part of the edge, a smooth contour is obtained, and then the smooth contour is used as a guide to apply a joint bilateral filter to reconstruct the super-resolution image texture, thereby realizing the super-resolution reconstruction of the underwater image, enhancing the resolution of the underwater image, and obtaining a high-resolution underwater picture; the smooth contour guide not only helps to avoid texture copy errors introduced by direct texture prediction, but also reduces jagged boundaries and preserves the characteristics of the edge, and focuses on solving the ghost problem caused by directly applying the optical image for super-resolution reconstruction and enhancement in a complex underwater lighting environment. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 The present application is a kind of underwater image super-resolution reconstruction method for the overall flow chart of the embodiment;
[0019] Figure 2 The present application is a kind of super-resolution reconstruction flow chart for the embodiment;
[0020] Figure 3 The present application is a kind of underwater robot terminal structure schematic diagram for the embodiment
[0021] Figure 4 A basic interaction procedure flow chart of an underwater robot terminal according to an embodiment of the present application;
[0022] Figure 5 A schematic diagram of a depth camera according to an embodiment of the present application;
[0023] Figure 6 Final effect of a high-resolution underwater picture obtained by an underwater image super-resolution reconstruction method according to an embodiment of the present application.
[0024] Label explanation:
[0025] 1. An underwater robot terminal; 2. An optical camera; 3. A depth camera; 4. A memory; 5. A processor. DETAILED DESCRIPTION
[0026] To explain the technical content, achieved purposes and effects of the present application in detail, the following will be described in detail in combination with embodiments and the accompanying drawings.
[0027] The most critical idea of the present application is that a depth camera is mounted on an underwater robot terminal, the super-resolution problem is converted from texture guidance to edge detection based on a depth map, and the super-resolution reconstruction of an underwater image captured by an optical camera is realized through the contour texture guidance of a depth image, so that a high-resolution underwater picture is obtained.
[0028] Please refer to Figure 1 , Figure 2 and Figure 6 , an underwater image super-resolution reconstruction method, comprising the steps of:
[0029] S1, controlling an optical camera and a depth camera to capture an original underwater image and a depth image of the same target at the same time;
[0030] S2, extracting the edge contour of the depth image by using a Canny algorithm;
[0031] S3, after denoising the original underwater image and aligning and correcting it with the edge contour of the depth image, a paired picture with a smooth contour is obtained;
[0032] S4, using a joint bilateral filter to reconstruct high-resolution underwater image texture guided by the smooth contour, a high-resolution underwater picture is obtained.
[0033] From the above description, the beneficial effects of the present application are that the edge contour based on the depth image guides the super-resolution reconstruction of the underwater image, converts the super-resolution problem from texture guidance to edge detection based on the depth image, specifically, after denoising and aligning the original underwater image with the edge contour, the paired image is formed by correcting part of the edge, the smooth contour is obtained, then the smooth contour is used as the guide, the joint bilateral filter is applied to reconstruct the super-resolution image texture, the super-resolution reconstruction of the underwater image is realized, and thus the resolution of the underwater image is enhanced to obtain a high-resolution underwater picture.
[0034] Further, the S1 further comprises:
[0035] The original underwater image and the depth image are registered in a world coordinate system, specifically:
[0036] The intrinsic matrix and distortion parameters of the depth camera are determined by a camera calibration method, the correspondence between the preset points in the depth image and the world coordinate system and the image coordinate system is determined according to the intrinsic matrix and the distortion parameters, the image coordinates of the depth image are converted to the world coordinates, and the registration of the depth image and the original underwater image is performed in combination with the RGB values at the positions corresponding to the world coordinates of the preset points in the original underwater image, and the registration coordinate calculation is as formula (1):
[0037]
[0038] Wherein, (X, Y, Z) is the coordinate of the preset point in the world coordinate system, (u, v) is the coordinate of the preset point in the image coordinate system, f u and f v are the size of the unit pixel of the imaging sensor of the depth camera in the u-axis and v-axis of the image coordinate, M1 is the intrinsic matrix, and M2 is the distortion parameter matrix containing the rotation matrix R and the translation vector t.
[0039] From the above description, since the original underwater image (i.e. RGB image) and the depth image captured by the optical camera and the depth camera respectively may have parallax, the two images need to be registered in the coordinate system, the two-dimensional image coordinates of the depth image are converted to three-dimensional world coordinates, and then the RGB image is combined to register the registered image, so as to correct the parallax of the depth image and the RGB image, in order to correct the subsequent contour texture.
[0040] Further, the S2 specifically comprises:
[0041] S21, define I(i, j) represents the pixel value of the point with image coordinates (i, j) in the depth image, and the gradient change rate of the pixel point I is obtained by the first derivative of the gray level along the two-dimensional direction, as shown in formula (2):
[0042]
[0043] Where f u and f v are the size of the unit pixel of the imaging sensor of the depth camera on the u-axis and v-axis of the image coordinates, respectively;
[0044] S22, the pixel point I is brought into the above formula (2) and is discretely calculated by the Isotropic Sobel template in the Canny algorithm, as shown in formula (3):
[0045]
[0046] In formula (3), Where G(i,j) is the Gaussian smoothing kernel of the pixel point I, and the amplitude r(i, j) and the direction a(i, j) of the edge profile of the depth image are obtained by the calculation of formula (3), as shown in formula (4):
[0047]
[0048] S23, the threshold of the edge profile is obtained by non-maximum suppression, and the same amplitude calculation is made according to the upper, lower, left and right four pixels of the point pixel I to obtain r1, r2, r3 and r4, then the threshold in two directions is calculated as formula (5):
[0049]
[0050] When r(i,j)>A1 and r(i,j)>A2 are satisfied, a maximum value r max (i,j) is obtained.
[0051] S24, traverse all pixel points of the depth image, and obtain the edge profile according to all maximum values.
[0052] From the above description, it can be seen that the Canny algorithm is a multi-level edge detection algorithm, and the edge profile of the depth image is detected by the Canny algorithm, which can ensure that the final obtained edge profile will not be changed due to the properties of the original depth image, and at the same time can significantly reduce the data size and calculation consumption of the edge profile, effectively improve the efficiency and precision of edge detection.
[0053] Further, the S3 is specifically:
[0054] S31, denoising the original underwater image and the depth image;
[0055] S32, defining a low-resolution depth map of the depth image as D, and a high-resolution texture contour of the original underwater image as C, upsampling D to the same resolution as C, specifically:
[0056] Let c=[c i ]、 d'=[d i ],wherein c, d and d' are the texture contour map, the low-resolution depth map and the depth map after expanding the resolution respectively, and all element data in c, d and d' are normalized to the interval [0, 1], and for vectors c and d', the following formula (6) is introduced:
[0057] c'=arg min{e(c')}=E f +aE s (6);
[0058] wherein c' is the corrected texture contour, E f is a data item, indicating the difference between the corrected contour and the original contour, E s is a regular term, indicating the constraint property of the edge of the depth image, and the constant value a is the influence factor of E f and E s balance;
[0059] S33, solving formula (6) by using IRLS algorithm to obtain the corrected smooth contour C'.
[0060] As can be seen from the above description, the original underwater image and the depth image are first denoised to realize image enhancement, and meanwhile, since the resolution of the depth image is smaller than that of the RGB image, and the texture contour generated by the RGB image must contain a large number of false object contours of the same depth, it is necessary to align and correct the contour texture of the RGB image by using the contour texture of the denoised depth image, and then pair them to obtain a picture with high-resolution contour texture, so as to greatly reduce the false contours of the original underwater image.
[0061] Further, the S4 is specifically:
[0062] The smooth contour is substituted into the following formula (7) as a guide:
[0063]
[0064] Wherein, F is a high-resolution RGB image, P is a low-resolution RGB image, q is an adjustable normalization coefficient, C' is the smooth contour obtained by step S3, and the different pixel point coordinates (i, j) of the smooth contour are determined according to the two filter sliding windows of the corresponding joint bilateral filter according to formula (7), and the distance ||i-j|| between pixels and the directional derivative C' i,j And the weighted operation is performed through Gaussian kernel functions f1 and f2, and the Gaussian kernel functions are as follows formula (8):
[0065]
[0066] Wherein, σ1 and σ2 are standard deviations, and β and λ are 2-3.
[0067] From the above description, the smooth contour obtained after correction is used for texture guidance, and the joint bilateral filter is applied to perform super-resolution texture reconstruction, which not only helps to avoid texture copy errors caused by direct texture introduction, but also reduces the sawtooth effect and preserves the edge effect, realizes image resolution enhancement in complex underwater environment, and avoids the ghost problem caused by direct application of optical image for super-resolution reconstruction and image enhancement.
[0068] Please refer to Figures 3 to 5 An underwater robot terminal comprises an optical camera, a depth camera, a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the following steps when executing the computer program:
[0069] S1, control the optical camera and the depth camera to shoot the original underwater image and the depth image of the same target at the same time;
[0070] S2, the edge contour of the depth image is extracted by using the Canny algorithm;
[0071] S3, after denoising the original underwater image and aligning and correcting the edge contour of the depth image, a paired picture with a smooth contour is obtained;
[0072] S4, using the smooth contour as a guide, a high-resolution underwater image texture is reconstructed by using a joint bilateral filter to obtain a high-resolution underwater picture.
[0073] From the above description, the beneficial effects of the present application are that: based on the same technical concept, cooperating with the above-mentioned underwater image super-resolution reconstruction method, an underwater robot terminal is provided, the edge contour of the depth image is used to guide the super-resolution reconstruction of the underwater image, the super-resolution problem is converted from texture guidance to edge detection based on the depth image, specifically, after the original underwater image is denoised and aligned with the edge contour, part of the edge is corrected to form a paired picture, a smooth contour is obtained, then the smooth contour is used as a guide, a joint bilateral filter is applied to reconstruct the image texture of the super-resolution, the super-resolution reconstruction of the underwater image is realized, and the resolution of the underwater image is enhanced to obtain a high-resolution underwater picture;The smoothed contour guide not only helps to avoid texture copy errors introduced by direct texture prediction, but also reduces jagged boundaries and preserves the characteristics of the edge, and focuses on solving the ghost problem caused by directly applying optical images for super-resolution reconstruction and enhancement in a complex underwater lighting environment.
[0074] Further, the S1 further comprises:
[0075] The original underwater image and the depth image are registered in a world coordinate system, specifically:
[0076] The intrinsic matrix and distortion parameters of the depth camera are determined by a camera calibration method, the correspondence relationship between the preset point in the depth image and the world coordinate system and the image coordinate system is determined according to the intrinsic matrix and the distortion parameters, the image coordinates of the depth image are converted to the world coordinates, and the registration of the depth image and the original underwater image is performed in combination with the RGB value of the position corresponding to the world coordinates of the preset point in the original underwater image, and the registration coordinate calculation is as formula (1):
[0077]
[0078] Wherein, (X, Y, Z) is the coordinate of the preset point in the world coordinate system, (u, v) is the coordinate of the preset point in the image coordinate system, f u And f v The size of the unit pixel of the imaging sensor of the depth camera on the u-axis and v-axis of the image coordinate, respectively, M1 is the intrinsic matrix, and M2 is the distortion parameter matrix containing the rotation matrix R and the translation vector t.
[0079] From the above description, since the original underwater image (i.e. RGB image) and the depth image photographed by the optical camera and the depth camera respectively may have parallax, it is necessary to register the two images in the coordinate system, the two-dimensional image coordinates of the depth image are converted to three-dimensional world coordinates, and then the RGB image is used to register the registered image, thereby correcting the parallax of the depth image and the RGB image. In order to correct the contour texture in the subsequent.
[0080] Further, the S2 is specifically:
[0081] S21, define I(i, j) represents the pixel value of the point with image coordinates (i, j) in the depth image, and the gradient change rate of the pixel point I is obtained by the first derivative of the gray along the two-dimensional direction, as shown in formula (2):
[0082]
[0083] Wherein, f u and f v The size of the unit pixel of the imaging sensor of the depth camera on the u axis and the v axis of the image coordinate;
[0084] S22, the pixel point I is brought into the above formula (2) and is calculated by the Isotropic Sobel template in the Canny algorithm, as shown in formula (3):
[0085]
[0086] In formula (3), Wherein, G(i,j) is the Gaussian smoothing kernel of the pixel point I, and the amplitude r(i, j) and the direction a(i, j) of the edge profile of the depth image are obtained by the calculation of formula (3), as shown in formula (4):
[0087]
[0088] S23, the threshold value of the edge profile is obtained by non-maximum suppression, and the same amplitude calculation is made according to the upper, lower, left and right four pixels of the point pixel point I, r1, r2, r3 and r4 are obtained, and the threshold values in two directions are calculated as formula (5):
[0089]
[0090] When r(i,j)>A1 and r(i,j)>A2 are satisfied, a maximum value r max (i,j) is obtained.
[0091] S24, all pixel points of the depth image are traversed, and the edge profile is obtained according to all the maximum values.
[0092] From the above description, it can be seen that the Canny algorithm is a multi-stage edge detection algorithm, which can ensure that the edge profile obtained finally will not be changed due to the properties of the original depth image, and at the same time can significantly reduce the data size and calculation consumption of the edge profile, effectively improve the efficiency and precision of edge detection.
[0093] Further, the S3 is specifically:
[0094] S31, denoising the original underwater image and the depth image;
[0095] S32, defining a low-resolution depth image of the depth image as D, and a high-resolution texture contour of the original underwater image as C, upsampling D to the same resolution as C, specifically:
[0096] Let c=[c i ]、 d'=[d i ],wherein c, d and d' are texture contour, low-resolution depth image and depth image after expanding resolution respectively, and normalize all element data in c, d and d' to [0, 1] interval, and introduce the following formula (6) for vectors c and d':
[0097] c'=arg min{e(c')}=E f +aE s (6);
[0098] wherein c' is the corrected texture contour, E f is a data item, indicating the difference between the corrected contour and the original contour, E s is a regular term, indicating the constraint property of the edge of the depth image, and constant value a is the influence factor of E f and E s balance.
[0099] S33, solving formula (6) by using IRLS algorithm to obtain the corrected smooth contour C'.
[0100] As can be seen from the above description, the original underwater image and the depth image are first denoised to realize image enhancement, and meanwhile, since the resolution of the depth image is smaller than that of the RGB image, and the texture contour generated by the RGB image must contain a large number of false object contours of the same depth, it is necessary to align and correct the contour texture of the RGB image by using the contour texture of the denoised depth image, and then pair them to obtain a picture with high-resolution contour texture, so as to greatly reduce the false contours of the original underwater image.
[0101] Further, the S4 is specifically:
[0102] Substitute the following formula (7) with the smooth contour as a guide:
[0103]
[0104] Wherein, F is a high-resolution RGB image, P is a low-resolution RGB image, q is an adjustable normalization coefficient, C' is the smooth contour obtained by step S3, and the different pixel point coordinates (i, j) of the smooth contour are determined according to the two filter sliding windows of the corresponding joint bilateral filter according to formula (7), and the distance ||i-j|| between the pixels and the directional derivative C' i,j And the weighted operation is carried out through Gaussian kernel functions f1 and f2, and the Gaussian kernel functions are as follows formula (8):
[0105]
[0106] Wherein, sigma1 and sigma2 are standard deviations, beta and lambda are 2-3.
[0107] It can be known from the above description that the smooth contour obtained after the correction is used for texture guidance, and then the joint bilateral filter is applied to perform super-resolution texture reconstruction, which not only helps to avoid texture copy errors caused by direct texture introduction, but also reduces the sawtooth effect and retains the edge effect, realizes image resolution enhancement in a complex underwater environment, and avoids the ghost problem caused by direct application of optical images for super-resolution reconstruction and image enhancement.
[0108] The underwater image super-resolution reconstruction method and the underwater robot terminal provided by the application are suitable for super-resolution reconstruction of underwater images shot in a complex underwater lighting environment, realize enhancement of underwater image resolution, and are described below in combination with specific embodiments.
[0109] Please refer to Figure 1 And Figure 6 Embodiment one of the application is:
[0110] An underwater image super-resolution reconstruction method, as Figure 1 Described, comprising the steps of:
[0111] S1, control the optical camera and the depth camera to shoot the original underwater image and the depth image of the same target at the same time;
[0112] S2, the edge contour of the depth image is extracted by using the Canny algorithm;
[0113] S3, after denoising and aligning the original underwater image with the edge contour of the depth image, the paired picture with the smooth contour is obtained;
[0114] S4, guided by the smooth contour, the high-resolution underwater image texture is reconstructed by using the joint bilateral filter, and the high-resolution underwater picture is obtained.
[0115] That is, in the embodiment, the edge contour based on the depth image guides the super-resolution reconstruction of the underwater image, converts the super-resolution problem from texture guidance to edge detection based on the depth image, specifically, after denoising and aligning the original underwater image with the edge contour, the paired image is formed by correcting part of the edge, the smooth contour is obtained, then the smooth contour is used as the guide, the joint bilateral filter is applied to reconstruct the super-resolution image texture, the super-resolution reconstruction of the underwater image is realized, so that the resolution of the underwater image is enhanced to obtain a high-resolution underwater image; wherein the smooth contour guide not only helps to avoid the texture copy error caused by direct texture introduction, but also reduces the jagged boundary and retains the characteristics of the edge, and focuses on solving the ghost problem caused by directly applying the optical image for super-resolution reconstruction and enhancement in a complex underwater lighting environment.
[0116] Meanwhile, the technical scheme of the embodiment is effectively applied to the super-resolution reconstruction of the underwater image to enhance the image resolution, and the depth information guided super-resolution recovery method is compared with the following multiple RGB image based deep learning models:
[0117] (1) image super-resolution analysis network SRCNN;
[0118] (2) residual super-resolution architecture SRResNet;
[0119] (3) super-resolution image restoration SRGAN.
[0120] The above models are currently the mainstream methods in the super-resolution analysis of underwater images in the RGB color space, and all have good performance. Under the same test data set, the performance of the models is quantitatively evaluated under the 2-fold, 3-fold and 4-fold super-resolution analysis settings, and the evaluation index parameters are: peak signal-to-noise ratio (PSNR), structural similarity measure (SSIM) and underwater image quality measure (UIQM). Among them, the peak signal-to-noise ratio (PSNR) and the structural similarity measure (SSIM) quantify the reconstruction quality and structural similarity of the generated image (relative to the high-resolution reference image), and the underwater image quality measure (UIQM) evaluates the image quality according to the color, sharpness and contrast.
[0121] The high-resolution underwater image effect finally obtained by the embodiment is shown in Figure 6 , wherein the upper left is the original underwater image taken by an optical camera, the lower left is the depth image taken by a depth camera, and the right side is the high-resolution underwater image after 2-fold super-resolution enhancement. The effect has a competitive advantage in terms of PSNR and SSIM, and generally has better performance, and also has a better UIQM score.
[0122] Please refer to Figure 2 , embodiment two of the present application is:
[0123] A method for super-resolution reconstruction of underwater images, based on the above embodiment one, in this embodiment, step S1 also includes:
[0124] The original underwater image and the depth image are registered in the world coordinate system, specifically:
[0125] The intrinsic matrix and distortion parameters of the depth camera are determined by camera calibration, and the correspondence between the preset point in the depth image and the world coordinate system and the image coordinate system is determined according to the intrinsic matrix and distortion parameters. The image coordinates of the depth image are converted to the world coordinates, and the RGB values of the positions corresponding to the world coordinates of the preset point in the original underwater image are combined to register the depth image and the original underwater image. The registration coordinate calculation is as formula (1):
[0126]
[0127] Where (X, Y, Z) is the coordinate of the preset point in the world coordinate system, (u, v) is the coordinate of the preset point in the image coordinate system, f u and f v are the size of the unit pixel of the imaging sensor of the depth camera in the u-axis and v-axis of the image coordinate, M1 is the intrinsic matrix, M2 is the distortion parameter matrix containing the rotation matrix R and the translation vector t, and T represents the transpose.
[0128] In this embodiment, the depth camera is selected as a Kinect depth camera, and the classic Zhang calibration method is used. A black and white checkerboard pattern A3 size template is printed, the depth camera is used to capture the test image, and the related functions in the Kinect SDK (Software Development Kit) are used to capture the depth data stream. The internal parameters of the depth camera are calculated by applying formula (1).
[0129] It is worth noting that this embodiment is to calibrate the depth camera to register with the optical camera. In other equivalent embodiments, the optical camera can also be calibrated and registered with the depth camera. That is, only one of the two cameras needs to be calibrated, and the viewing angle of one camera can be registered with the viewing angle of the other camera.
[0130] That is, in this embodiment, since the original underwater images (i.e. RGB images) and depth images captured by the optical camera and the depth camera may have parallax, the two images need to be registered in the coordinate system. By converting the two-dimensional image coordinates of the depth image to three-dimensional world coordinates, and then combining the RGB image to register the registered image, the parallax of the depth image and the RGB image is corrected to facilitate subsequent contour texture correction.
[0131] wherein, the step S2 is specifically:
[0132] S21, define I(i, j) represents the pixel value of the point with image coordinates (i, j) in the depth image, and the gradient change rate of the pixel point I is obtained by the first derivative of the gray along the two-dimensional direction, as shown in formula (2):
[0133]
[0134] S22, the pixel point I is brought into the above formula (2) and is discretely calculated by the Isotropic Sobel template in the Canny algorithm, as shown in formula (3):
[0135]
[0136] In formula (3), Wherein, G(i,j) is the Gaussian smoothing kernel of the pixel point I, the amplitude r(i, j) and the direction a(i, j) of the edge contour of the depth image are obtained by the calculation of formula (3), as shown in formula (4):
[0137]
[0138] S23, the threshold of the edge contour is obtained by using the non-maximum suppression, the same amplitude calculation is made according to the upper, lower, left and right four pixels of the point pixel point I, r1, r2, r3 and r4 are obtained, then the threshold in two directions is calculated as formula (5):
[0139]
[0140] When r(i,j)>A1 and r(i,j)>A2 are satisfied, a maximum value r max (i,j) is obtained.
[0141] S24, all pixel points of the depth image are traversed, and the edge contour is obtained according to all the maximum values.
[0142] Wherein, the Canny algorithm is a multi-stage edge detection algorithm, the edge contour of the depth image is detected by the Canny algorithm, which can ensure that the final obtained edge contour will not change the properties of the original depth image, and at the same time can significantly reduce the data size and calculation consumption of the edge contour, effectively improve the efficiency and precision of the edge detection; in addition, the non-maximum suppression (Non-Maximum Suppression, NMS) means suppressing elements that are not maximum values, which can be understood as local maximum search, in this embodiment, that is, the threshold is obtained from the edge contour of the depth image.
[0143] In the step S3, the original underwater image and the depth image are denoised to enhance the original underwater image and the depth image.
[0144] S31, denoising the original underwater image and the depth image to enhance the original underwater image and the depth image.
[0145] S32, defining a low-resolution depth image of the depth image as D, and a high-resolution texture contour of the original underwater image as C, upsampling D to the same resolution as C, specifically:
[0146] Let c = [c i ], d' = [d i ], where c, d and d' are the texture contour, the low-resolution depth image and the depth image after the resolution is expanded, respectively, and all element data in c, d and d' are normalized to the interval [0, 1], and for the vectors c and d', the following formula (6) is introduced:
[0147] c' = arg min{e(c')} = E f +aE s (6);
[0148] where c' is the corrected texture contour, E f is a data item representing the difference between the corrected contour and the original contour, E s is a regularization term representing the constraint property of the edge of the depth image, and the constant value a is an influence factor balancing E f and E s . Through experimental verification, when a >= 8, the false contour can be greatly reduced while maintaining the details of the edge contour.
[0149] S33, using the IRLS (iterative reweighted least squares) algorithm to solve formula (6) to obtain the corrected smooth contour C'.
[0150] That is, the original underwater image and the depth image are first denoised to enhance the image, and at the same time, since the resolution of the depth image is smaller than that of the RGB image, and the texture contour generated by the RGB image necessarily contains a large number of false object contours of the same depth, it is necessary to use the contour texture of the denoised depth image to correct and pair the contour texture of the RGB image to obtain a picture with high-resolution contour texture, so as to greatly reduce the false contour of the original underwater image.
[0151] Finally, the step S4 is specifically:
[0152] Using the smooth contour as a guide, substitute into the following formula (7):
[0153]
[0154] Wherein, F is a high-resolution RGB image, P is a low-resolution RGB image, q is an adjustable normalization coefficient, C' is a smooth contour obtained by step S3, and the different pixel point coordinates (i, j) of the smooth contour are determined according to two filter sliding windows of the corresponding joint bilateral filter according to formula (7), and the distance ||i-j|| between pixels and the directional derivative C' are calculated i,j And the weighted operation is performed through Gaussian kernel functions f1 and f2, and the Gaussian kernel functions are as follows formula (8):
[0155]
[0156] Wherein, sigma1 and sigma2 are standard deviations, in the example, the values are related to the size of the filter window, beta and lambda are empirical values of 2-3 according to the super-resolution ratio.
[0157] That is, the texture is guided by the modified smooth contour, and the joint bilateral filter is applied to perform super-resolution texture reconstruction, which not only helps to avoid texture copy errors caused by direct texture introduction, but also reduces sawtooth effects and preserves edge effects, realizes image resolution enhancement in a complex underwater environment, and avoids the ghost problem caused by directly applying optical images for super-resolution reconstruction and image enhancement.
[0158] The underwater image super-resolution reconstruction method can achieve 2-16 times of super-resolution recovery of underwater images, and automatically obtain the enhancement of the underwater images, so as to obtain high-resolution underwater pictures.
[0159] Please refer to Figures 3 to 5 , the third embodiment of the present application is:
[0160] An underwater robot terminal 1, as shown in Figure 3 , comprises an optical camera 2, a depth camera 3, a memory 4, a processor 5, and a computer program stored on the memory 4 and executable on the processor 5, and the processor 5 realizes the steps in the above-mentioned embodiment one or embodiment two when executing the computer program.
[0161] Wherein, in the embodiment, the depth camera 3 adopts a Kinect depth camera, and the structure is as shown in Figure 4 , and the processor 5 of the underwater robot terminal 1 also executes a basic interaction program as shown in Figure 5 , to realize complete underwater image shooting and super-resolution reconstruction operation.
[0162] In summary, the application provides an underwater image super-resolution reconstruction method and an underwater robot terminal, which is based on edge contour of a depth image to guide super-resolution reconstruction of the underwater image, converts the super-resolution problem from texture guidance to edge detection based on the depth image, specifically, after denoising and aligning with the edge contour, the original underwater image is corrected to form a paired picture, and a smooth contour is obtained, then the smooth contour is used as a guide to apply a joint bilateral filter to reconstruct the super-resolution image texture, so as to realize super-resolution reconstruction of the underwater image, thereby enhancing the resolution of the underwater image to obtain a high-resolution underwater picture; wherein the smooth contour guide not only helps to avoid texture copy errors introduced by direct texture prediction, but also reduces jagged boundaries and preserves the characteristics of the edge, and focuses on solving the ghost problem caused by directly applying optical images for super-resolution reconstruction and enhancement in a complex underwater lighting environment.
[0163] The above description is only an embodiment of the application, and does not limit the patent scope of the application, and any equivalent transformation or direct or indirect application in the related technical field based on the content of the specification and drawings is also included in the patent protection scope of the application.
Claims
1. A method for super-resolution reconstruction of underwater images, characterized in that, The method comprises the steps of: S1, controlling an optical camera and a depth camera to respectively capture an original underwater image and a depth image of a same target at a same time; S2, extracting an edge contour of the depth image by using a Canny algorithm; the S2 specifically comprises: S21, defining I(i, j) as a pixel value of a point with an image coordinate of (i, j) in the depth image, and a gradient change rate of the pixel point I is obtained by a first-order derivative of a gray scale along a two-dimensional direction, as shown in a formula (2); S22, bringing the pixel point I into the above formula (2) and performing discrete calculation by using an Isotropic Sobel template in the Canny algorithm, as shown in a formula (3); In formula (3), Wherein, G(i,j) is a Gaussian smoothing kernel of the pixel point I, the amplitude r(i,j) and the direction a(i,j) of the edge profile of the depth image are obtained through the calculation of formula (3), as follows formula (4): S23, obtaining a threshold value of the edge contour by using non-maximum suppression, and obtaining r1, r2, r3 and r4 by performing the same amplitude calculation on four pixels above, below, left and right of the point pixel I, and then calculating the threshold values in two directions as shown in a formula (5); When r(i,j) > A1 and r(i,j) > A2, a maximum value r is obtained max (i,j); S24, traversing all pixel points of the depth image, and obtaining the edge contour according to all maximum values; S3, performing denoising on the original underwater image, and correcting the original underwater image by aligning with the edge contour of the depth image, to obtain a smooth contour of the corrected original underwater image; S4, taking the smooth contour as a guide, and substituting into a formula (7) as follows: Wherein, F is a high-resolution underwater image, P is a low-resolution original underwater image, q is an adjustable normalization coefficient, C' is the smooth profile obtained from step S3, and the different pixel point coordinates (i, j) of the smooth profile are determined according to the two filter sliding windows of the corresponding joint bilateral filter according to formula (7), and the distance ||i-j|| between the pixels and the directional derivative C' i,j and the weighted operation is performed through Gaussian kernel functions f1 and f2, and the Gaussian kernel functions are as follows formula (8): wherein, sigma1 and sigma2 are standard deviations of respective items, and beta and lambda are 2-3.
2. An underwater robotic terminal, characterized by, The method comprises the steps of: S1, controlling an optical camera and a depth camera to respectively capture an original underwater image and a depth image of a same target at a same time; S2, extracting an edge contour of the depth image by using a Canny algorithm; the S2 specifically comprises: S21, defining I(i, j) as a pixel value of a point with an image coordinate of (i, j) in the depth image, and a gradient change rate of the pixel point I is obtained by a first-order derivative of a gray scale along a two-dimensional direction, as shown in a formula (2); S22, bringing the pixel point I into the above formula (2) and performing discrete calculation by using an Isotropic Sobel template in the Canny algorithm, as shown in a formula (3); In formula (3), Wherein, G(i,j) is a Gaussian smoothing kernel of the pixel point I, the amplitude r(i,j) and the direction a(i,j) of the edge profile of the depth image are obtained by calculation of formula (3), as follows formula (4): S23, obtaining a threshold value of the edge contour by using non-maximum suppression, and obtaining r1, r2, r3 and r4 by performing the same amplitude calculation on four pixels above, below, left and right of the point pixel I, and then calculating the threshold values in two directions as shown in a formula (5); When r(i,j) > A1 and r(i,j) > A2, a maximum value r is obtained max (i,j); S24, traversing all pixel points of the depth image, and obtaining the edge contour according to all maximum values; S3, performing denoising on the original underwater image, and correcting the original underwater image by aligning with the edge contour of the depth image, to obtain a smooth contour of the corrected original underwater image; S4, taking the smooth contour as a guide, and substituting into a formula (7) as follows: Wherein, F is a high-resolution underwater image, P is a low-resolution original underwater image, q is an adjustable normalization coefficient, C' is the smooth profile obtained from step S3, and the different pixel point coordinates (i, j) of the smooth profile are determined according to the two filter sliding windows of the corresponding joint bilateral filter according to formula (7), and the distance ||i-j|| between the pixels and the directional derivative C' i,j and the weighted operation is performed through Gaussian kernel functions f1 and f2, and the Gaussian kernel functions are as follows formula (8): wherein, sigma1 and sigma2 are standard deviations of respective items, and beta and lambda are 2-3.