Image interpolation method and device based on RGB-D image and multi-camera system

Through the image interpolation method based on RGB-D images and multi-camera system, the problem of view angle switching lag caused by insufficient number of cameras in multi-camera systems is solved, and efficient multi-view video generation under a small number of cameras is achieved, improving fluency and Calculate the speed.

CN112488918BActive Publication Date: 2024-11-26PLEX VR DIGITAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011355759.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-11-27
Publication Date
2024-11-26
Estimated Expiration
2040-11-27

AI Technical Summary

Technical Problem

In multi-camera systems, due to the small number of acquisition cameras, the viewing experience is stuttered when switching perspectives.

Method used

The image interpolation method based on RGB-D images and multi-camera systems is adopted, and image interpolation is realized through camera calibration, position calculation, image fusion and pixel completion, and the interpolated image of a new camera is generated, reducing the dependence on the number of cameras.

Benefits of technology

It realizes the formation of multi-view video under a small number of cameras, improves the fluency of viewing angle switching, saves shooting costs, and improves data transmission speed and calculation speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112488918B_ABST
    Figure CN112488918B_ABST
Patent Text Reader

Abstract

The present invention discloses an image interpolation method and device based on RGB-D images and a multi-camera system, wherein the method comprises: performing camera calibration on each camera in the multi-camera system; clarifying the interpolation position of a new camera according to the location information of each camera in the multi-camera system, and calculating the camera pose of the new camera according to the camera calibration data; calculating a plurality of initial interpolation images having a one-to-one correspondence with the designated images collected by each camera in the multi-camera system according to the projection relationship of the cameras and the pose information of each camera; performing image fusion on each of the initial interpolation images to obtain a fused interpolation image; performing pixel completion on the fused interpolation image to finally obtain an interpolation image associated with the new camera. The present invention solves the problem that multi-view video is prone to visual freeze when switching viewing angles due to too few cameras being deployed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image interpolation method, and in particular to an image interpolation method and device based on RGB-D and a multi-camera system. Background Art

[0002] Nowadays, multi-camera systems have been widely used in the fields of 3D reconstruction, motion capture, and multi-viewpoint video shooting. Multi-camera systems use multiple different cameras, light sources, storage devices, etc. to simultaneously track and shoot one or more targets. The resulting multi-view videos can better display the characteristics of the targets and greatly improve the audience's visual experience. However, multi-view videos can usually only be viewed from the perspective of the original acquisition camera. When the number of acquisition cameras is sparse, the perspective switching causes a large change in the content, which makes the user's viewing experience lag. Summary of the invention

[0003] The present invention aims to solve the problem that multi-viewing video is prone to viewing freezes when switching viewing angles due to too few acquisition cameras, and proposes an image interpolation method and device based on RGB-D images and a multi-camera system.

[0004] To achieve this object, the present invention adopts the following technical solutions:

[0005] An image interpolation method based on an RGB-D image and a multi-camera system is provided, the steps comprising:

[0006] 1) Perform camera calibration for each camera in a multi-camera system;

[0007] 2) according to the position information of each camera in the multi-camera system, the interpolation position of the new camera is determined, and the camera pose of the new camera is calculated according to the camera calibration data of step 1);

[0008] 3) calculating a plurality of initial interpolation images having a one-to-one correspondence with the designated images captured by each camera in the multi-camera system according to the projection relationship of the cameras and the position information of each camera;

[0009] 4) performing image fusion on each of the initial interpolation images to obtain a fused interpolation image;

[0010] 5) Perform pixel completion on the fused interpolated image to finally obtain an interpolated image associated with the new camera.

[0011] Preferably, in step 2), the camera pose of the new camera includes a camera intrinsic parameter matrix, a camera translation vector and a camera rotation matrix, and the camera intrinsic parameter matrix of the new camera is calculated by the following formula (1):

[0012] K'=(1-λ)K1+λK2 Formula (1)

[0013] In formula (1), K' represents the camera intrinsic parameter matrix of the new camera;

[0014] λ is used to represent the interpolated position of the new camera, λ is the ratio of the distance from the new camera to the left camera to the total distance between the left and right cameras, 0λ≤1;

[0015] K1 represents the intrinsic parameter matrix of the left camera arranged on the left hand side of the new camera;

[0016] K2 represents the intrinsic parameter matrix of the right camera arranged on the right hand side of the new camera.

[0017] Preferably, the camera translation vector of the new camera is calculated by the following formula (2):

[0018] T'=(10λ)T1+λT2 Formula (2)

[0019] In formula (2), T' represents the camera translation vector of the new camera;

[0020] T1 represents the camera translation vector of the left camera;

[0021] T2 represents the camera translation vector of the right camera.

[0022] Preferably, the specific steps of calculating the camera rotation matrix of the new camera include:

[0023] 2.1) calculating a first relative rotation matrix of the right camera relative to the left camera through the camera rotation matrices of the left camera and the right camera;

[0024] 2.2) Convert the first relative rotation matrix into a first relative rotation vector, wherein the first relative rotation vector is a rotation axis r=r x ,r y ,r z ] T and the rotation angle θ represents;

[0025] 2.3) calculating the product of the rotation angle θ and the ratio λ as the rotation angle θ' of the new camera relative to the left camera, the rotation angle θ' and the rotation axis r which is the same as the first relative rotation vector are used to represent a second relative rotation vector of the new camera relative to the left camera;

[0026] 2.4) converting the second relative rotation vector into a second relative rotation matrix;

[0027] 2.5) Reversely calculate the camera rotation matrix of the new camera based on the second relative rotation matrix and the camera rotation matrix of the left camera.

[0028] Preferably, the process of calculating the camera rotation matrix of the new camera is expressed by the following formula (3):

[0029]

[0030] In formula (3), R' represents the camera rotation matrix of the new camera;

[0031] M v2r represents converting the first relative rotation matrix into the first relative rotation vector;

[0032] M r2v represents converting the second relative rotation vector into the second relative rotation matrix;

[0033] R1 represents the camera rotation matrix of the left camera from the camera coordinate system to the world coordinate system;

[0034] R2 represents the camera rotation matrix of the right camera from the camera coordinate system to the world coordinate system.

[0035] Preferably, in step 3), the specific steps of calculating the initial interpolation image include:

[0036] 3.1) Establish the projection matrix of each camera;

[0037] 3.2) According to all pixel coordinates and depth values ​​on the specified image acquired by a specified camera, a three-dimensional discrete point S is obtained by back-projection using the established camera projection matrix;

[0038] 3.3) calculating pixel coordinates on the image to be generated according to the pose information of the designated camera and the new camera and according to the camera projection matrix of the new camera;

[0039] 3.4) According to the correspondence between the coordinates of the pixel points on the designated image and the image to be generated, fill the pixel values ​​and depth values ​​on the designated image into the corresponding pixel points on the image to be generated, so as to obtain the initial interpolated image having a correspondence with the designated image;

[0040] 3.5) Repeat steps 3.2) to 3.4) until a plurality of initial interpolation images having a one-to-one correspondence with the designated images captured by all cameras in the multi-camera system are calculated.

[0041] Preferably, in step 3.3), the pixel coordinates on the image to be generated are calculated by the following formula (4):

[0042]

[0043] In formula (4), u' represents the coordinate of the pixel on the image to be generated on the x-axis;

[0044] v' represents the coordinate of the pixel on the image to be generated on the y-axis;

[0045] d' represents the depth value corresponding to the pixel at the u', v' coordinate position;

[0046] The x and y in formula (4) are calculated by the following formula (5):

[0047]

[0048] In formula (5), u1 and v1 represent the pixel coordinate positions on the specified image, u1 represents the coordinate of the pixel on the specified image on the x-axis, and v1 represents the coordinate of the pixel on the specified image on the y-axis;

[0049] P1 represents the camera projection matrix of the specified camera;

[0050] P' represents the camera projection matrix of the new camera;

[0051] d1 represents the depth value corresponding to the pixel at the coordinate position of u1 and v1.

[0052] Preferably, when there are multiple pixel points projected from the same designated image to the same coordinate position on the image to be generated, the pixel value of the pixel with the smallest depth value d' is retained as the pixel value of the pixel point at the coordinate position on the image to be generated.

[0053] Preferably, in step 4), the method for performing image fusion on each of the initial interpolation images is:

[0054] 4.1) determining whether the pixel values ​​of the pixel points at the same position on each of the initial interpolation images are all empty,

[0055] If yes, jump to step 5) to enter the image completion process;

[0056] If not, go to step 4.2);

[0057] 4.2) determining whether the number of the initial interpolation images whose pixel values ​​at the same position are non-empty is 1,

[0058] If yes, assigning a non-empty pixel value to the pixel at the same position on the fused interpolated image;

[0059] If not, go to step 4.3);

[0060] 4.3) Calculate the difference in depth values ​​of pixels with non-empty pixel values ​​at the same position between the initial interpolation images, and select a corresponding pixel value assignment method according to the threshold judgment result through a threshold judgment method, and assign the pixel value on the initial interpolation image to the fused interpolation image.

[0061] Preferably, in step 4.3), the specific method of assigning the pixel value on the initial interpolation image to the fused interpolation image is:

[0062] If the absolute value of the difference between the depth values ​​of the pixel points at the same position on the right image captured by the right camera and the left image captured by the left camera is less than or equal to the set threshold ∈, the pixel values ​​of the left image and the right image at the same position are weighted averaged and assigned to the corresponding pixel points of the fused interpolated image;

[0063] If the difference between the pixel values ​​at the same position on the right image and the left image is greater than the threshold ∈, assigning the pixel value at the same position on the left image to the corresponding pixel point of the fused interpolated image;

[0064] If the difference between the pixel values ​​at the same position on the left image and the right image is less than the threshold ∈, the pixel value at the same position on the right image is assigned to the corresponding pixel point of the fused interpolated image.

[0065] Preferably, the step of performing pixel completion on the fused interpolated image specifically includes:

[0066] 5.1) Generate a window W with the empty pixel as the center;

[0067] 5.2) Calculate the average pixel value of all non-empty pixels in the window W;

[0068] 5.3) Filling the average pixel value to the central pixel point determined in step 5.1);

[0069] 5.4) Repeat steps 5.1) to 5.3) until pixel completion of all empty pixels on the fused interpolated image is completed.

[0070] The present invention also provides an image interpolation device based on RGB-D images and a multi-camera system, the image interpolation device comprising:

[0071] A camera calibration module, used to calibrate each camera in a multi-camera system;

[0072] A new camera pose calculation module, connected to the camera calibration module, for determining the position of the new camera according to the position information of each camera in the multi-camera system, and calculating the camera pose of the new camera according to the camera calibration data;

[0073] an initial interpolation image calculation module, connected to the new camera posture calculation module, for calculating a plurality of initial interpolation images having a one-to-one correspondence with the designated images acquired by each camera in the multi-camera system according to the projection relationship of the cameras and the posture information of each camera;

[0074] An image fusion module, connected to the initial interpolation image calculation module, for performing image fusion on each of the initial interpolation images to obtain a fused interpolation image;

[0075] The image completion module is connected to the image fusion module and is used to perform pixel completion on the fused interpolation image to finally obtain an interpolation image associated with the new camera.

[0076] The present invention has the following beneficial effects:

[0077] 1. Image interpolation can be performed at any linear position between cameras, so that the shooting effect of multiple cameras can be achieved with only a small number of cameras, saving shooting costs;

[0078] 2. Using a small number of cameras, a multi-view video can be formed as if it were viewed from a dense perspective. The video perspective switching is smoother and there is no lag. In addition, the number of images is reduced, which is conducive to improving the data transmission speed of the multi-camera system.

[0079] 3. The parallel computing method is used to calculate the pixel value of each pixel point on the interpolation image, which improves the calculation speed of the interpolation image. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0081] Figure 1 is a step diagram of an image interpolation method based on RGB-D image and multi-camera system provided by one embodiment of the present invention;

[0082] Figure 2 This is a step-by-step diagram of how to calculate the camera rotation matrix of a new camera;

[0083] Figure 3 is a specific method step diagram for calculating the initial interpolation image;

[0084] Figure 4 is a step diagram of a method for performing image fusion on each of the initial interpolation images;

[0085] Figure 5It is a schematic diagram for calculating the position of the new camera;

[0086] Figure 6 is a schematic diagram for calculating the initial interpolation image;

[0087] Figure 7 is a step diagram of a method for performing pixel completion on the fused interpolation image;

[0088] Figure 8 It is a schematic diagram of the internal logical structure of an image interpolation device based on RGB-D images and a multi-camera system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0090] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0091] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as a limitation on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0092] In the description of the present invention, unless otherwise clearly specified and limited, if the term "connection" or the like appears to indicate the connection relationship between components, the term should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0093] An image interpolation method based on RGB-D and a multi-camera system is provided in an embodiment of the present invention. Figure 1 As shown, the steps include:

[0094] 1) Perform camera calibration on each camera in the multi-camera system to obtain the camera's intrinsic and extrinsic parameters. The intrinsic parameter matrix K is represented by the following 3×3 matrix:

[0095]

[0096] Where: f x Indicates the focal length of the camera along the x-axis, in pixels;

[0097] f y Indicates the focal length of the camera along the y-axis in pixels;

[0098] c x is the coordinate of the principal point of the image along the x-axis, in pixels;

[0099] c y is the coordinate of the principal point of the image along the y-axis, in pixels.

[0100] The external parameter matrix is ​​a 3×4 matrix R|T composed of a 3×3 rotation matrix R and a 3×1 translation vector T;

[0101] 2) According to the position information of each camera in the multi-camera system, the interpolation position of the new camera is determined, and the camera position of the new camera is calculated according to the camera calibration data of step 1);

[0102] The camera position specifying method of the new camera adopted by the present invention is:

[0103] like Figure 5 As described, within the camera trajectory, taking any two adjacent cameras as an example, one is recorded as the left camera and the other is recorded as the right camera, the new camera is interpolated at the position between the line segment connecting the left camera and the right camera. The interpolation position of the new camera is represented by the ratio λ, and the specific setting position of the new camera is calculated as the ratio of the distance from the new camera to the left camera to the total distance of the left and right cameras, and the ratio is represented by λ. When the new camera is located at the position of the left camera, λ=0; when the new camera is located at the position of the right camera, λ=1. Therefore, when the new camera is located between the positions of the left and right cameras, 0≤λ≤1.

[0104] The camera pose of the new camera includes the camera intrinsic parameter matrix, camera translation vector and camera rotation matrix. The camera translation vector and camera rotation matrix of the new camera constitute the external parameter matrix of the new camera. The camera intrinsic parameter matrix of the new camera is calculated by the following formula (1):

[0105] K'=(1-λ)K1+λK2 Formula (1)

[0106] In formula (1), K' represents the camera intrinsic parameter matrix of the new camera;

[0107] λ is used to represent the interpolated position of the new camera. λ is the ratio of the distance from the new camera to the left camera to the total distance between the left and right cameras, 0≤λ≤1;

[0108] K1 represents the intrinsic parameter matrix of the left camera set on the left hand side of the new camera;

[0109] K2 represents the intrinsic parameter matrix of the right camera set on the right hand side of the new camera.

[0110] The camera translation vector of the new camera is calculated using the following formula (2):

[0111] T'=1-λT1+λT2 formula (2)

[0112] In formula (2), T' represents the camera translation vector of the new camera;

[0113] T1 represents the camera translation vector of the left camera;

[0114] T2 represents the camera translation vector of the right camera.

[0115] like Figure 2 As shown, the calculation process of the camera rotation matrix of the new camera specifically includes the following steps:

[0116] 2.1) Calculate the first relative rotation matrix of the right camera relative to the left camera through the camera rotation matrices of the left camera and the right camera;

[0117] 2.2) The first relative rotation matrix is ​​converted into a first relative rotation vector. The first relative rotation vector is the rotation axis r = [r x ,r y ,r z ] T and the rotation angle θ represents;

[0118] 2.3) calculating the product of the rotation angle θ and the ratio λ as the rotation angle θ' of the new camera relative to the left camera, and the rotation angle θ' and the same rotation axis r as the first relative rotation vector are used to represent the second relative rotation vector of the new camera relative to the left camera;

[0119] 2.4) converting the second relative rotation vector into a second relative rotation matrix;

[0120] 2.5) Based on the second relative rotation matrix and the camera rotation matrix of the left camera, the camera rotation matrix of the new camera is calculated in reverse.

[0121] The above process of calculating the camera rotation matrix of the new camera can be expressed by the following formula (3):

[0122]

[0123] In formula (3), R' represents the camera rotation matrix of the new camera;

[0124] M v2r It represents the conversion of the first relative rotation matrix into the first relative rotation vector. The process of converting the first relative rotation matrix into the first relative rotation vector can be expressed by the following formula (10):

[0125]

[0126] M r2v It represents that the second relative rotation vector is converted into the second relative rotation matrix. The process of converting the second relative rotation vector into the second relative rotation matrix can be expressed by the following formula (11):

[0127]

[0128] R1 represents the camera rotation matrix of the left camera from the camera coordinate system to the world coordinate system;

[0129] R2 represents the camera rotation matrix of the right camera from the camera coordinate system to the world coordinate system;

[0130] I is the 3×3 identity matrix.

[0131] Please continue to refer to Figure 1 , the image interpolation method based on RGB-D image and multi-camera system provided by the present invention also includes:

[0132] 3) According to the projection relationship of the cameras and the position information of each camera, a plurality of initial interpolation images having a one-to-one correspondence with the specified images collected by each camera in the multi-camera system are calculated;

[0133] like Figure 3 and Figure 6 As shown, the specific steps for calculating the initial interpolation image include:

[0134] 3.1) Establish the projection matrix of each camera; the projection matrix P of each camera is calculated by the following formula (12):

[0135]

[0136] In formula (12), K represents the intrinsic parameter matrix of the camera;

[0137] R represents the rotation matrix of the camera from the world coordinate system to the camera coordinate system;

[0138] T represents the translation vector of the camera from the world coordinate system to the camera coordinate system;

[0139] The transformation between the camera coordinate system and the world coordinate system can be calculated by the following formula (13):

[0140]

[0141] In formula (13), R w2c Represents the rotation matrix from the world coordinate system to the camera coordinate system;

[0142] T w2c Represents the translation vector from the world coordinate system to the camera coordinate system;

[0143] R c2w Represents the rotation matrix from the camera coordinate system to the world coordinate system;

[0144] T c2w Represents the translation vector from the camera coordinate system to the world coordinate system.

[0145] 3.2) According to all pixel coordinates and depth values ​​on the specified image acquired by the specified camera, a three-dimensional discrete point S is obtained by back-projection using the established camera projection matrix;

[0146] 3.3) According to the pose information of the specified camera and the new camera, and according to the camera projection matrix of the new camera, the pixel coordinates on the image to be generated (i.e., the initial interpolation image) are calculated;

[0147] 3.4) According to the correspondence between the coordinates of the pixel points on the specified image and the image to be generated, the pixel values ​​and depth values ​​on the specified image are filled into the corresponding pixel points of the image to be generated, so as to obtain an initial interpolation image having a correspondence with the specified image;

[0148] 3.5) Repeat steps 3.2 to 3.4 until multiple initial interpolation images having a one-to-one correspondence with the specified images captured by all cameras in the multi-camera system are calculated.

[0149] The following example takes the setting of the new camera between the left camera and the right camera as an example, and combines Figure 6 , the calculation process of the initial interpolation image is explained:

[0150] First, the image captured by the left camera is recorded as the left image (i.e., the designated image). According to all pixel coordinates and depth values ​​on the left image, the projection matrix is ​​used to back-project to obtain the three-dimensional discrete point S. Then, according to the projection matrix of the new camera, the pixel coordinates on the image to be generated (interpolated image) are projected using the posture relationship between the left camera and the new camera. Then, the pixel values ​​on the left image are filled into the corresponding pixel points of the image to be generated. If multiple pixels on the left image are projected to the same pixel position on the image to be generated, only the pixel value with the smallest depth value after projection is retained to obtain the initial interpolated RGB image I l , and get the initial interpolated depth image D lFinally, the same interpolation method is used to obtain the initial interpolated RGB image I according to the back projection and projection of the right image collected by the right camera. r and the initial interpolated depth image D r .

[0151] In the above step 3.3), the pixel coordinates on the image to be generated are calculated by the following formula (4):

[0152]

[0153] In formula (4), u' represents the coordinate of the pixel on the image to be generated on the x-axis;

[0154] v' represents the coordinate of the pixel on the image to be generated on the y-axis;

[0155] d' represents the depth value corresponding to the pixel at the u', v' coordinate position;

[0156] The x and y in formula (4) are calculated by the following formula (5):

[0157]

[0158] In formula (5), u1 and v1 represent the pixel coordinate positions on the specified image, u1 represents the coordinate of the pixel on the specified image on the x-axis, and v1 represents the coordinate of the pixel on the specified image on the y-axis;

[0159] P1 represents the camera projection matrix of the specified camera;

[0160] P' represents the camera projection matrix of the new camera;

[0161] d1 represents the depth value corresponding to the pixel at the coordinate position of u1 and v1.

[0162] Please continue to refer to Figure 1 , the image interpolation method based on RGB-D image and multi-camera system provided by the present invention also includes:

[0163] Step 4) performing image fusion on each initial interpolation image to obtain a fused interpolation image;

[0164] Specifically, Figure 4 As shown, the specific steps of fusing the initial interpolation images include:

[0165] 4.1) Determine whether the pixel values ​​of the pixels at the same position on each initial interpolation image are all empty,

[0166] If yes, then enter the image completion process;

[0167] If not, go to step 4.2);

[0168] 4.2) Determine whether the number of initial interpolation images whose pixel values ​​at the same position are non-empty is 1,

[0169] If yes, assign the non-empty pixel value to the pixel at the same position on the fused interpolated image;

[0170] If not, go to step 4.3);

[0171] 4.3) Calculate the difference in depth values ​​of pixels with non-empty pixel values ​​at the same position between the initial interpolation images, and select the corresponding pixel value assignment method according to the threshold judgment result through the threshold judgment method, and assign the pixel value on the initial interpolation image to the fused interpolation image.

[0172] In step 4.3), the specific method of assigning the pixel values ​​on the initial interpolation image to the fused interpolation image is:

[0173] If the absolute value of the difference in depth value of the pixel at the same position on the right image captured by the right camera and the left image captured by the left camera is less than or equal to the set threshold ∈, the pixel values ​​at the same position of the left image and the right image are weighted averaged and assigned to the corresponding pixel of the fused interpolated image;

[0174] If the difference between the pixel values ​​at the same position on the right image and the left image is greater than the threshold ∈, the pixel value at the same position on the left image is assigned to the corresponding pixel point of the fused interpolated image;

[0175] If the difference between the pixel values ​​at the same position on the left image and the right image is less than the threshold ∈, the pixel value at the same position on the right image is assigned to the corresponding pixel point of the fused interpolated image.

[0176] Specifically, the present invention obtains the initial interpolation image I from the left image and the right image according to the following three criteria: l and I r The pixel values ​​at the same position are merged:

[0177] If at the same position, the initial interpolation image I l The pixel value on is not empty, the initial interpolation image I r If the pixel value on is empty, the initial interpolation image I l The pixel value at the position on is assigned to the fused interpolated image. The fusion process can be expressed by the following formula (6):

[0178] I'(i,j)=I l (i,j,ifI l (i,j)≠0andI r (i,j)=0 Formula (6)

[0179] In formula (6), I'(i,j) represents the fused interpolated image;

[0180] i, j represent the coordinate position of the pixel point on the initial interpolation image or the fused interpolation image.

[0181] If at the same position, the initial interpolation image I r The pixel value on is not empty, the initial interpolation image I l If the pixel value on is empty, the initial interpolation image I r The pixel value at the position on is assigned to the fused interpolated image. The fusion process can be expressed by the following formula (7):

[0182] I'(i,j)=I r (i,j),ifI r (i,j)≠0andI l (i,j)=0 Formula (7)

[0183] If at the same position, the initial interpolation image I l and the initial interpolation image I r If the pixel values ​​on are not empty, the difference of the depth values ​​of the pixels at the same position is calculated, and the corresponding pixel value assignment method is selected according to the threshold judgment result through the threshold judgment method, and the pixel values ​​on the initial interpolation image are assigned to the fused interpolation image. The specific interpolation process can be expressed by the following formula (8):

[0184]

[0185] In formula (8), D r (i, j) represents the initial interpolated depth image in the right image;

[0186] D l (i, j) represents the initial interpolated depth image on the left image;

[0187] I l (i, j) represents the initial interpolated RGB image formed by the projection of the left image;

[0188] I r (i, j) represents the initial interpolated RGB image formed by the projection of the right image.

[0189] In step 5), when it is determined that the pixel values ​​of the pixels at the same position on each initial interpolation image are all empty, Figure 7 As shown, the step of completing the pixels at the corresponding positions on the fused interpolated image specifically includes:

[0190] 5.1) Generate a window W with the empty pixel as the center;

[0191] 5.2) Calculate the average pixel value of all non-empty pixels in window W;

[0192] 5.3) Fill the average pixel value to the central pixel point determined in step 5.1);

[0193] 5.4) Repeat steps 5.1) to 5.3) until pixel completion of all empty pixels on the fused interpolated image is completed.

[0194] The above pixel completion process can be expressed by the following formula (9):

[0195]

[0196] In formula (9), I(i,j) represents the completed fused interpolated image;

[0197] Δx, Δy represent the offsets in the x and y directions of the window W relative to the center pixel;

[0198] card(W) is the number of valid pixels in window W.

[0199] I'(i,j) represents the incomplete fused interpolated image.

[0200] The present invention also provides an image interpolation device based on RGB-D image and multi-camera system, such as Figure 8 As shown, the device comprises:

[0201] A camera calibration module, used to calibrate each camera in a multi-camera system;

[0202] A new camera pose calculation module is connected to the camera calibration module and is used to determine the position of the new camera according to the position information of each camera in the multi-camera system, and calculate the camera pose of the new camera according to the camera calibration data;

[0203] An initial interpolation image calculation module, connected to the new camera posture calculation module, is used to calculate a plurality of initial interpolation images having a one-to-one correspondence with the specified images collected by each camera in the multi-camera system according to the projection relationship of the cameras and the posture information of each camera;

[0204] An image fusion module, connected to the initial interpolation image calculation module, is used to fuse the initial interpolation images to obtain a fused interpolation image;

[0205] The image completion module is connected to the image fusion module and is used to complete the pixels of the fused interpolated image to finally obtain the interpolated image associated with the new camera.

[0206] It should be noted that the above specific implementations are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art should understand that various modifications, equivalent substitutions, changes, etc. can be made to the present invention. However, as long as these changes do not deviate from the spirit of the present invention, they should be within the scope of protection of the present invention. In addition, some terms used in the specification and claims of this application are not restrictive, but are only for the convenience of description.

Claims

1. An image interpolation method based on RGB-D image and multi-camera system, characterized in that the steps include: 1) Perform camera calibration for each camera in a multi-camera system; 2) according to the position information of each camera in the multi-camera system, the interpolation position of the new camera is determined, and the camera pose of the new camera is calculated according to the camera calibration data of step 1); 3) calculating a plurality of initial interpolation images having a one-to-one correspondence with the designated images captured by each camera in the multi-camera system according to the projection relationship of the cameras and the position information of each camera; 4) performing image fusion on each of the initial interpolation images to obtain a fused interpolation image; 5) Performing pixel completion on the fused interpolated image to finally obtain an interpolated image associated with the new camera; in, In step 2), the camera pose of the new camera includes a camera intrinsic parameter matrix, a camera translation vector and a camera rotation matrix. The camera intrinsic parameter matrix of the new camera is calculated by the following formula (1): K'=1-λ)K1+λK2 Formula (1) In formula (1), K' represents the camera intrinsic parameter matrix of the new camera; λ is used to represent the interpolated position of the new camera, and λ is the ratio of the distance from the new camera to the left camera to the total distance between the left and right cameras, 0≤λ≤1; K1 represents the intrinsic parameter matrix of the left camera arranged on the left hand side of the new camera; K2 represents the intrinsic parameter matrix of the right camera arranged on the right hand side of the new camera; In step 3), the specific steps of calculating the initial interpolation image include: 3.1) Establish the projection matrix of each camera; 3.2) According to all pixel coordinates and depth values ​​on the specified image acquired by a specified camera, a three-dimensional discrete point S is obtained by back-projection using the established camera projection matrix; 3.3) calculating pixel coordinates on the image to be generated according to the pose information of the designated camera and the new camera, using the three-dimensional discrete points, and according to the camera projection matrix of the new camera; 3.4) According to the correspondence between the coordinates of the pixel points on the designated image and the image to be generated, fill the pixel values ​​and depth values ​​on the designated image into the corresponding pixel points on the image to be generated, so as to obtain the initial interpolated image having a correspondence with the designated image; 3.5) Repeat steps 3.2) to 3.4) until a plurality of initial interpolation images having a one-to-one correspondence with the designated images captured by all cameras in the multi-camera system are calculated.

2. The image interpolation method based on RGB-D image and multi-camera system according to claim 1, characterized in that: The camera translation vector of the new camera is calculated by the following formula (2): T′=(1-λ)T1+λT2 Formula (2) In formula (2), T′ represents the camera translation vector of the new camera; T1 represents the camera translation vector of the left camera; T2 represents the camera translation vector of the right camera.

3. The image interpolation method based on RGB-D image and multi-camera system according to claim 1, characterized in that: The specific steps of calculating the camera rotation matrix of the new camera include: 2.1) calculating a first relative rotation matrix of the right camera relative to the left camera through the camera rotation matrices of the left camera and the right camera; 2.2) Converting the first relative rotation matrix into a first relative rotation vector, wherein the first relative rotation vector is a rotation axis r=[r x , r y , r z ] T and the rotation angle θ represents; 2.3) calculating the product of the rotation angle θ and the ratio λ as the rotation angle θ′ of the new camera relative to the left camera, the rotation angle θ′ and the rotation axis r which is the same as the first relative rotation vector are used to represent a second relative rotation vector of the new camera relative to the left camera; 2.4) converting the second relative rotation vector into a second relative rotation matrix; 2.5) Reversely calculate the camera rotation matrix of the new camera based on the second relative rotation matrix and the camera rotation matrix of the left camera.

4. The image interpolation method based on RGB-D image and multi-camera system according to claim 3, characterized in that: The process of calculating the camera rotation matrix of the new camera is expressed by the following formula (3): In formula (3), R′ represents the camera rotation matrix of the new camera; M v2r represents converting the first relative rotation matrix into the first relative rotation vector; M r2v represents converting the second relative rotation vector into the second relative rotation matrix; R1 represents the camera rotation matrix of the left camera from the camera coordinate system to the world coordinate system; R2 represents the camera rotation matrix of the right camera from the camera coordinate system to the world coordinate system.

5. The image interpolation method based on RGB-D image and multi-camera system according to claim 1, characterized in that: In step 3.3), the pixel coordinates on the image to be generated are calculated by the following formula (4): In formula (4), u′ represents the coordinate of the pixel on the image to be generated on the x-axis; v′ represents the coordinate of the pixel on the image to be generated on the y-axis; d′ represents the depth value corresponding to the pixel at the u′ and v′ coordinate positions; The x and y in formula (4) are calculated by the following formula (5): In formula (5), u1 and v1 represent the pixel coordinate positions on the specified image, u1 represents the coordinate of the pixel on the specified image on the x-axis, and v1 represents the coordinate of the pixel on the specified image on the y-axis; P1 represents the camera projection matrix of the specified camera; P′ represents the camera projection matrix of the new camera; d1 represents the depth value corresponding to the pixel at the coordinate position of u1 and v1.

6. The image interpolation method based on RGB-D image and multi-camera system according to claim 5, characterized in that: When there are multiple pixel points projected from the same designated image to the same coordinate position on the image to be generated, the pixel value of the pixel with the smallest depth value d′ is retained as the pixel value of the pixel point at the coordinate position on the image to be generated.

7. The image interpolation method based on RGB-D image and multi-camera system according to claim 1, characterized in that: In step 4), the method of performing image fusion on each of the initial interpolation images is: 4.1) determining whether the pixel values ​​of the pixel points at the same position on each of the initial interpolation images are all empty, If yes, jump to step 5) to enter the image completion process; If not, go to step 4.2); 4.2) determining whether the number of the initial interpolation images whose pixel values ​​at the same position are non-empty is 1, If yes, assigning a non-empty pixel value to the pixel at the same position on the fused interpolated image; If not, go to step 4.3); 4.3) Calculate the difference in depth values ​​of pixels with non-empty pixel values ​​at the same position between the initial interpolation images, and select a corresponding pixel value assignment method according to the threshold judgment result through a threshold judgment method, and assign the pixel value on the initial interpolation image to the fused interpolation image.

8. The image interpolation method based on RGB-D image and multi-camera system according to claim 7, characterized in that: In step 4.3), the specific method of assigning the pixel value on the initial interpolation image to the fused interpolation image is: If the absolute value of the difference between the depth values ​​of the pixel points at the same position on the right image captured by the right camera and the left image captured by the left camera is less than or equal to the set threshold ∈, the pixel values ​​of the left image and the right image at the same position are weighted averaged and assigned to the corresponding pixel points of the fused interpolated image; If the difference between the pixel values ​​at the same position on the right image and the left image is greater than the threshold ∈, assigning the pixel value at the same position on the left image to the corresponding pixel point of the fused interpolated image; If the difference between the pixel values ​​at the same position on the left image and the right image is less than the threshold ∈, the pixel value at the same position on the right image is assigned to the corresponding pixel point of the fused interpolated image.

9. The image interpolation method based on RGB-D image and multi-camera system according to claim 7, characterized in that: The step of performing pixel completion on the fused interpolation image specifically includes: 5.1) Generate a window W with the empty pixel as the center; 5.2) Calculate the average pixel value of all non-empty pixels in the window W; 5.3) Filling the average pixel value to the central pixel point determined in step 5.1); 5.4) Repeat steps 5.1) to 5.3) until pixel completion of all empty pixels on the fused interpolated image is completed.

10. An image interpolation device based on RGB-D images and a multi-camera system, capable of implementing the image interpolation method according to any one of claims 1 to 9, characterized in that: The image interpolation device comprises: A camera calibration module, used to calibrate each camera in a multi-camera system; A new camera pose calculation module, connected to the camera calibration module, for determining the position of the new camera according to the position information of each camera in the multi-camera system, and calculating the camera pose of the new camera according to the camera calibration data; an initial interpolation image calculation module, connected to the new camera posture calculation module, for calculating a plurality of initial interpolation images having a one-to-one correspondence with the designated images acquired by each camera in the multi-camera system according to the projection relationship of the cameras and the posture information of each camera; An image fusion module, connected to the initial interpolation image calculation module, for performing image fusion on each of the initial interpolation images to obtain a fused interpolation image; The image completion module is connected to the image fusion module and is used to perform pixel completion on the fused interpolation image to finally obtain an interpolation image associated with the new camera.