A method for restoring degraded image information based on multi-sensor fusion

Abstract

The present invention discloses a method for restoring degraded image information based on multi-sensor fusion. The present invention comprises: step 1: calibration of internal and external parameters of camera and inertial sensor; step 2: calculation of joint degradation parameters based on degradation model: defining relative displacement as a vector, reading image distance, obtaining depth and motion, and calculating vector based on similar triangles, and then projecting to imaging plane and decomposing into length parameter and angle parameter; step 3: de-motion blur; step 4: rolling shutter effect compensation. The present invention reduces input parameters when dealing with this problem and improves the integration of the solution. The processing results are better than general methods in terms of indicators such as MSE, PSNR, and SSIM.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for restoring degraded image information based on multi-sensor fusion. Background Art

[0002] CMOS rolling shutter type cameras are currently the most widely used in various devices including smart phones, security equipment, and robots due to their lower cost and better performance advantages compared to other types of cameras. Its physical properties and structure enable this type of camera to have a better performance when shooting static objects, but also due to the characteristics of its own structure, this type of camera will have motion blur in the form of loss of detail information and rolling shutter effect in the form of structural distortion in some dynamic scenes such as when the camera itself or the target object is in motion. These two phenomena can be summarized as image degradation problems. The present invention aims to restore the information lost in the degradation process while ensuring that the camera hardware and shooting quality remain unchanged, and to obtain a clear and undistorted image. In response to the two degradation problems that occur simultaneously in this scene, the original image is processed through a multi-sensor method and an image processing algorithm to achieve the goal.

[0003] Consider the image degradation scenario when taking photos with a smartphone during exercise. Although the clear image is degraded and information is lost due to the motion during the shooting process, the inertial sensor in the phone also records the acceleration, angular acceleration and other information of the device itself during the motion. Based on the inertial sensor information, the originally unknown motion information becomes a priori constraint in the restoration process to restore the lost details. Summary of the invention

[0004] The purpose of the present invention is to provide a method for restoring degraded image information of a high dynamic scene CMOS rolling shutter camera combined with inertial navigation information in view of the shortcomings of the prior art. The present invention uses an inertial device matched with the camera, takes image data and inertial navigation information as input, and designs a method for restoring degraded image information based on multi-sensor fusion.

[0005] The method mainly includes four stages: calibration of internal and external parameters of the camera and inertial sensor, calculation of joint degradation parameters based on the degradation model, de-motion blur and rolling shutter effect compensation.

[0006] Step 1: Calibrate the internal and external parameters of the camera and inertial sensor: Determine the relative position relationship T of the sensor through the sensor calibration method imu2cam , which facilitates unifying sensor data into the same coordinate system;

[0007] Step 2: Calculate the joint degradation parameters based on the degradation model: Since motion is relative, the cause of image degradation can be considered as the relative displacement that occurs during the exposure time. The relative displacement is defined as a vector texposure ; Read the image distance f from the camera file, use frame matching and triangulation to get the depth depth, use the inertial navigation information integration to get the motion motion, and calculate the vector t based on similar triangles exposure , then projected onto the imaging plane and decomposed into length parameter length and angle parameter angle;

[0008] Step 3: De-motion blur: The degradation parameters calculated in step 2 are used as input to construct the blur kernel h. The grayscale image of the blurred image is converted into an exposure image according to the photoelectric conversion function, and then restored based on the Wiener filtering method to obtain a clear image.

[0009] Step 4: Rolling shutter effect compensation: Based on the same degradation parameters, calculate the displacement of the corner points in the image affected by the rolling shutter effect. After considering the influence of perspective, the new coordinates of the corner points in the image coordinate system are obtained as (ug, vg). The image not affected by the rolling shutter effect can be restored through perspective transformation.

[0010] The advantages and beneficial results of the method of the present invention are:

[0011] 1. The present invention only requires the device to have both a camera and an inertial navigation device, without the need for additional hardware to increase costs. Currently, many electronic devices in various fields, from smart phones to robots, almost meet this condition, and have wide applicability;

[0012] 2. The present invention considers the two main phenomena of motion blur and rolling shutter effect in degradation, and proposes a degradation model to normalize the two, so that the input parameters for dealing with the problem are reduced and the solution integration is improved. The processing results are better than general methods in terms of MSE, PSNR, SSIM and other indicators.

[0013] 3. The method of the present invention improves the effect of the image restoration method by adding a photoelectric conversion function processing method. The boundaries of the processed image are clearer, so that more feature points can be extracted, which better meets the requirements of image information processing tasks such as VSLAM and semantic segmentation.

[0014] 4. The method of the present invention incorporates motion and perspective factors into the model of the rolling shutter effect, so that the structured elements of the processed image are increased, the external environment structure can be more accurately characterized, and the influence of image quality on the accuracy of the processing results of the visual measurement task is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the geometric relationship between the variables in the calculation method of step 2 in this method.

[0016] Figure 2 is a flow chart of step 3 in the method.

[0017] Figure 3 is a flow chart of step 4 in the method. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0019] like Figure 1-Figure 3 As shown, the specific steps of the present invention are as follows:

[0020] Step 1: Calibrate the internal and external parameters of the camera and inertial sensor. The specific implementation is as follows:

[0021] Use a device with a camera and an inertial sensor to record a calibration data file with a certain movement on the calibration board, and calculate the relative position relationship T of the sensor. imu2cam :

[0022]

[0023] Among them, R imu2cam is the rotation representation of the posture between the two transformations, t imu2cam It is a translation representation of its position.

[0024] The calculation of the joint degradation parameters based on the degradation model described in step 2 is specifically implemented as follows:

[0025] like Figure 1 As shown, in the degradation model, the vector t exposure , image distance f, depth depth, motion motion have the formula:

[0026]

[0027] The vector t exposure Place it in the image coordinate system by multiplying it by the known coordinate system direction axis unit vector img x 、img y Decompose to get the projection proj of the vector in the direction of the corresponding coordinate axis x 、proj y :

[0028]

[0029] Then the length parameter length is:

[0030]

[0031] The angle parameter angle is:

[0032]

[0033] The de-motion blurring described in step 3 is specifically implemented as follows:

[0034] Reference Figure 2 In the process shown, the CMOS photoelectric conversion function is defined as LE (grey), then the CMOS electro-optical conversion function is obtained by exchanging the XY axis as EL (exp), and the grayscale image g_g (x, y) of the blurred image and the exposure image g_e (x, y) satisfy:

[0035] g_e(x,y)=EL(g_g(x,y))(6)

[0036] Based on the length parameter length and angle parameter angle described in step 2, the blur kernel h(x, y) is constructed, and its conjugate is h * (x,y), the square of the absolute value is |h(x,y)| 2 , assuming that the noise is n(x,y), then perform Wiener filtering and use LE(x,y) conversion to obtain the deblurred clear image f(x,y):

[0037]

[0038] Rolling shutter effect compensation as described in step 4;

[0039] Reference Figure 3 The specific steps of the process are as follows:

[0040] 4-1. During one shooting process, the camera will scan at a speed v scan Along with img x The camera photoelectric conversion array is scanned in the opposite direction of the axis. If the camera moves at a speed of v cam A displacement of length length is generated along the angle parameter angle direction, then:

[0041]

[0042] 4-2. Based on the conditions of step 4-1, assume that there is a point FP in the scene and the camera internal parameter matrix is ​​K. Consider that the coordinates of the point fp in the image after perspective mapping of FP before distortion are (u g ,v g ), the depth data is h e ; The coordinates of the point fp' in the image after distortion are (u f ,v f ), the image depth data at this time is h' e If the motion is decomposed into rotational representation R and translational representation t, the above variables satisfy the formula:

[0043]

[0044] By substituting in specific values, the coordinates of the distortion front point fp can be solved, and the perspective transformation based on the coordinates can complete the compensation of the image rolling effect.

[0045] Experimental data support:

[0046] The present invention has the ability to eliminate both degradation manifestations of images captured by multi-sensor devices in high dynamic scenes, and the calibration plate is a standard test and calibration tool commonly used for visual sensors. Therefore, the experiment uses a smartphone to collect corresponding data from the calibration plate in a moving state, and then shows the effects before and after different algorithm processing and proves the effectiveness and advancement of the invention through objective indicators such as MSE, PSNR, and SSIM.

[0047] Table 1 shows the processing effects of various advanced deblurring algorithms and the deblurring module in the proposed algorithm on image motion blur. The larger the index, the better.

[0048] Table 1 Objective index evaluation of motion blur restoration methods

[0049]

[0050]

[0051] From a subjective perspective, all kinds of motion blur algorithms have a certain ability to restore blurred images, but for the ringing produced during restoration, the method of the present invention is closer to the physical essence of real blur, so the restoration effect is better than other algorithms. From the quantitative perspective of objective indicators, the method of the present invention is better than the existing advanced algorithms in terms of PSNR, SSIM and the number of good matching of feature points, showing the effectiveness of the algorithm and its performance better than the existing advanced algorithms.

[0052] Table 2 shows the processing effects of various advanced rolling compensation algorithms and the anti-rolling compensation module in the algorithm of the present invention on the rolling effect of images. The original image when captured and the image to be tested are used as inputs when calculating the indicators. Except for the last item, the larger the better.

[0053] Table 2 Objective indicators of rolling shutter compensation algorithm

[0054]

[0055] From a subjective perspective, both the method of the present invention and the existing advanced methods can compensate for the rolling shutter effect, but the processing effect of the present invention is obviously closer to the real original image. Quantitatively, the present invention is superior to the existing methods in terms of PSNR and SSIM indicators. Especially in the indicator of the absolute error of the angle between the angle and the original image, the error can be reduced by an order of magnitude. The comparative advantages of the three indicator data prove the effectiveness of the algorithm of the present invention and show performance superior to that of the existing algorithm.

[0056] Table 3 shows the objective indicators of the comparison between the clear original image and the actually captured blurred image, the image with only motion deblurring, the image with only rolling shutter compensation, and the image with both rolling shutter compensation and motion deblurring (method in this paper).

[0057] Table 3 Objective indicators of algorithm comprehensive experiment

[0058]

[0059]

[0060] The results show that each module has a certain improvement effect on the various indicators of the blurred image, but it will also have an adverse effect on other indicators. Motion blur removal emphasizes providing more visual features, but the SSIM indicator will be significantly reduced. Compared with the results of the two treatments at the same time, the decline of the SSIM indicator has been significantly improved, and the "number of good matches of feature points" indicator has been further significantly improved. Rolling blind compensation emphasizes correcting visual information to improve accuracy, so its angular error with the reference clear image has been improved to a certain extent, but its "number of good matches of feature points" indicator has been slightly reduced. Compared with the results of the two treatments at the same time, although SSIM has decreased, it has brought a significant improvement in the "number of good matches of feature points" indicator, and the absolute angle error indicator has been reduced by nearly half again. Therefore, in general, the modules in the method not only improve some indicators of the blurred image, but also combine the two to produce a certain degree of complementary effect. It can be considered that this method is an advanced method with a high improvement in the comprehensive indicators of degraded images.

[0061] The above only describes some exemplary embodiments of the present invention by way of illustration. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for restoring degraded image information based on multi-sensor fusion, characterized in that: The steps include: Step 1: Calibrate the internal and external parameters of the camera and inertial sensor: Determine the relative position and posture relationship of the sensor through the sensor calibration method to unify the sensor data into the same coordinate system; Step 2: Calculate the joint degradation parameters based on the degradation model: Since motion is relative, the cause of image degradation can be considered as the relative displacement that occurs during the exposure time. The relative displacement is defined as a vector t exposure ; Read the image distance f from the camera file, use frame matching and triangulation to get the depth depth, use the inertial navigation information integration to get the motion motion, and calculate the vector t based on similar triangles exposure , then projected onto the imaging plane and decomposed into length parameter length and angle parameter angle; Step 3: De-motion blurring: the degradation parameters calculated in step 2 are used as input to construct a blur kernel. The grayscale image of the blurred image is converted into an exposure image according to the photoelectric conversion function, and then restored based on the Wiener filtering method to obtain a clear image. Step 4: Rolling shutter effect compensation: Based on the same degradation parameters, the displacement of the corner points affected by the rolling shutter effect in the image is calculated. After considering the influence of perspective, the new coordinates of the corner points in the image coordinate system are obtained. The image not affected by the rolling shutter effect can be restored through perspective transformation. Step 4 is implemented as follows: 4-1. During one shooting process, the camera will scan at a speed v scan Along with img x The camera photoelectric conversion array is scanned in the opposite direction of the axis, with a total of cols columns of pixels; if the camera moves at a speed of v cam A displacement of length length is generated along the angle parameter angle direction, then: 4-2. Based on the conditions of step 4-1, assume that there is a point FP in the scene, and the camera internal parameter matrix is ​​K; consider that the coordinates of the point fp in the image mapped to the perspective of FP before distortion are (u g ,v g ), the depth data is h e ; The coordinates of the point fp' in the image after distortion are (u f ,v f ), the image depth data at this time is h' e ; If the motion is decomposed into rotational representation R and translational representation t, the above variables satisfy the formula: By substituting in specific values, the coordinates of the distortion front point fp can be solved, and the perspective transformation based on the coordinates can complete the compensation of the image rolling effect.

2. The method for restoring degraded image information based on multi-sensor fusion according to claim 1, characterized in that: Step 1 is implemented as follows: Use a device with a camera and an inertial sensor to record a calibration data file with a certain movement on the calibration board, and calculate the relative position relationship T of the sensor. imu2cam : Among them, R imu2cam is the rotation representation of the posture between the two transformations, t imu2cam It is a translation representation of its position.

3. The method for restoring degraded image information based on multi-sensor fusion according to claim 1, characterized in that: Step 2 is implemented as follows: In the degradation model, the vector t exposure , image distance f, depth depth, motion motion have the formula: The vector t exposure Place it in the image coordinate system by multiplying it by the known coordinate system direction axis unit vector img x 、img y Decompose to get the projection proj of the vector in the direction of the corresponding coordinate axis x 、proj y : Then the length parameter length is: The angle parameter angle is:

4. The method for restoring degraded image information based on multi-sensor fusion according to claim 3 is characterized in that: Step 3 is implemented as follows: Define the CMOS photoelectric conversion function as LE (grey), then swap the XY axis to get the CMOS electro-optical conversion function as EL (exp), and the grayscale image g_g (x, y) of the blurred image and the exposure image g_e (x, y) satisfy: g_e(x,y)=EL(g_g(x,y)) Based on the length parameter length and angle parameter angle described in step 2, the blur kernel h(x, y) is constructed and its conjugate is h * (x,y), the square of the absolute value is |h(x,y)| 2 ; Assuming the noise is n(x,y), perform Wiener filtering and use LE(x,y) conversion to obtain the deblurred clear image f(x,y):

5. A system for restoring degraded image information based on multi-sensor fusion, characterized in that: The method for restoring degraded image information based on multi-sensor fusion as claimed in claim 1 comprises: Camera and inertial sensor internal and external parameter calibration module: Determine the relative position and posture relationship of the sensor through the sensor calibration method, so as to unify the sensor data into the same coordinate system; Joint degradation parameter calculation module: Since motion is relative, the cause of image degradation can be considered as the relative displacement that occurs during the exposure time. The relative displacement is defined as a vector t exposure ; Read the image distance f from the camera file, use frame matching and triangulation to get the depth depth, use the inertial navigation information integration to get the motion motion, and calculate the vector t based on similar triangles exposure , then projected onto the imaging plane and decomposed into length parameter length and angle parameter angle; De-motion blur module: The degradation parameters calculated by the joint degradation parameter calculation module are used as input to construct the blur kernel, and the grayscale image of the blurred image is converted into an exposure image according to the photoelectric conversion function, and then a clear image is obtained after restoration based on the Wiener filtering method; Rolling effect compensation module: Based on the same degradation parameters, the displacement of the corner points in the image affected by the rolling effect is calculated, and the new coordinates of the corner points in the image coordinate system are obtained after considering the influence of perspective. The image not affected by the rolling effect can be restored through perspective transformation.

Images