Three-dimensional image reconstruction method, device and equipment

Abstract

The application provides a three-dimensional image reconstruction method, device and equipment, comprising: acquiring a plurality of color image pairs of a to-be-scanned object; the color image pair comprises a left color image photographed by a left eye camera of a binocular camera and a right color image photographed by a right eye camera; determining a plurality of corresponding parallax image pairs and initial depth image pairs of the plurality of color image pairs, and determining a target depth image pair based on the plurality of parallax image pairs and the plurality of initial depth image pairs; performing three-dimensional reconstruction based on the target depth image pair and the color image pair corresponding to the target depth image pair to obtain a three-dimensional image of the to-be-scanned object. The three-dimensional image reconstruction method, device and equipment provided by the application are used for improving the quality of the three-dimensional image.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus and device for three-dimensional image reconstruction. Background Technology

[0002] 3D reconstruction technology is widely used in fields such as Virtual Reality System (VR), Augmented Reality (AR), and film and games.

[0003] In 3D reconstruction technology, 3D images are obtained by performing 3D reconstruction based on multiple sets of color images and multiple sets of depth maps. Related technologies involve depth estimation from multiple sets of color images to obtain multiple sets of depth maps.

[0004] In the aforementioned related technologies, multiple sets of depth maps are obtained through depth estimation, resulting in low matching accuracy between multiple sets of color images and multiple sets of depth maps, leading to poor 3D reconstruction results and consequently poor 3D image quality. Summary of the Invention

[0005] This invention provides a three-dimensional image reconstruction method, apparatus, and device to address the shortcomings of existing technologies where the matching accuracy between multiple sets of color images and multiple sets of depth images is low, resulting in poor three-dimensional reconstruction effects and consequently poor three-dimensional image quality. This invention aims to improve the quality of three-dimensional images.

[0006] In a first aspect, the present invention provides a three-dimensional image reconstruction method, comprising:

[0007] Acquire multiple color image pairs of the object to be scanned; the color image pairs include a left color image captured by the left eye camera of the binocular camera and a right color image captured by the right eye camera;

[0008] Determine the disparity image pairs and initial depth image pairs corresponding to each of the plurality of color image pairs, and determine the target depth image pairs based on the plurality of disparity image pairs and the plurality of initial depth image pairs;

[0009] Three-dimensional reconstruction is performed based on the target depth image pair and the corresponding color image pair to obtain a three-dimensional image of the object to be scanned.

[0010] According to a three-dimensional image reconstruction method provided by the present invention, determining the disparity image pairs and initial depth image pairs corresponding to each of the plurality of color image pairs includes:

[0011] Perform the following operations on each color image pair:

[0012] Based on the intrinsic and extrinsic parameters of the binocular camera, stereo correction is performed on the color image pair to obtain a target color image pair. The target color image pair is then processed by a preset matching algorithm to obtain the disparity image pair and the initial depth image pair corresponding to the color image pair.

[0013] According to a three-dimensional image reconstruction method provided by the present invention, the step of determining a target depth image pair based on multiple disparity image pairs and multiple initial depth image pairs includes:

[0014] For each disparity image pair, a disparity constant value for the disparity image pair is determined based on the disparity value of each pixel in the disparity image pair.

[0015] An intermediate depth image pair is determined from the plurality of initial depth image pairs, wherein the disparity constant value of the disparity image pair corresponding to the intermediate depth image pair is less than or equal to a preset threshold.

[0016] Based on the intermediate depth image pair, the target depth image pair is determined.

[0017] According to a three-dimensional image reconstruction method provided by the present invention, determining the target depth image pair based on the intermediate depth image pair includes:

[0018] For each intermediate depth image in the intermediate depth image pair, the following operations are performed:

[0019] The intermediate depth image is divided into multiple non-overlapping image blocks; for each image block, an image block similar to the image block is determined in a preset region corresponding to the image block, wherein the preset region is a region including the image block or an adjacent region of the image block;

[0020] Based on the position of each image block in the intermediate depth image, image blocks similar to the image blocks are stitched together to obtain the target depth image corresponding to the intermediate depth image.

[0021] The target depth image pair includes the target depth image corresponding to the intermediate depth image of the intermediate depth image pair.

[0022] According to a three-dimensional image reconstruction method provided by the present invention, the step of determining an image block similar to the image block in a preset region corresponding to the image block includes:

[0023] For each reference image block in the preset region, determine the depth difference between two pixels whose positions match in the image block and the reference image block;

[0024] The sum of the depth differences between the two pixels that are matched at the location is determined as the patch difference value between the image patch and the reference image patch;

[0025] The reference image block corresponding to the smallest tile difference value in the preset area is determined as an image block similar to the image block.

[0026] According to a three-dimensional image reconstruction method provided by the present invention, the method further includes:

[0027] The three-dimensional image is optically encoded to obtain encoded information;

[0028] The encoded information is loaded into the light field display so that the light field display performs light field display based on the encoded information.

[0029] In a second aspect, the present invention also provides a three-dimensional image reconstruction apparatus, comprising:

[0030] The image acquisition module is used to acquire multiple color image pairs of the object to be scanned; the color image pairs include a left color image captured by the left eye camera of the binocular camera and a right color image captured by the right eye camera;

[0031] A stereo matching module is used to determine the disparity image pairs and initial depth image pairs corresponding to each of the plurality of color image pairs;

[0032] A depth denoising module is used to determine a target depth image pair based on multiple disparity image pairs and multiple initial depth image pairs;

[0033] The 3D reconstruction module is used to perform 3D reconstruction based on the target depth image pair and the corresponding color image pair to obtain a 3D image of the object to be scanned.

[0034] According to a three-dimensional image reconstruction apparatus provided by the present invention, the stereo matching module includes:

[0035] The parameter acquisition and stereo correction unit is specifically used to perform stereo correction on the color image pair based on the intrinsic and extrinsic parameters of the binocular camera to obtain the target color image pair;

[0036] The stereo matching unit is specifically used to process the target color image pair using a preset matching algorithm to obtain the disparity image pair and the initial depth image pair corresponding to the color image pair.

[0037] According to a three-dimensional image reconstruction apparatus provided by the present invention, the depth denoising module is specifically used for:

[0038] For each disparity image pair, a disparity constant value for the disparity image pair is determined based on the disparity value of each pixel in the disparity image pair.

[0039] An intermediate depth image pair is determined from the plurality of initial depth image pairs, wherein the disparity constant value of the disparity image pair corresponding to the intermediate depth image pair is less than or equal to a preset threshold.

[0040] Based on the intermediate depth image pair, the target depth image pair is determined.

[0041] According to a three-dimensional image reconstruction apparatus provided by the present invention, the depth denoising module is specifically used for:

[0042] For each intermediate depth image in the intermediate depth image pair, the following operations are performed:

[0043] The intermediate depth image is divided into multiple non-overlapping image blocks; for each image block, an image block similar to the image block is determined in a preset region corresponding to the image block, wherein the preset region is a region including the image block or an adjacent region of the image block;

[0044] Based on the position of each image block in the intermediate depth image, image blocks similar to the image blocks are stitched together to obtain the target depth image corresponding to the intermediate depth image.

[0045] The target depth image pair includes the target depth image corresponding to the intermediate depth image of the intermediate depth image pair.

[0046] According to a three-dimensional image reconstruction apparatus provided by the present invention, the depth denoising module is specifically used for:

[0047] For each reference image block in the preset region, determine the depth difference between two pixels whose positions match in the image block and the reference image block;

[0048] The sum of the depth differences between the two pixels that are matched at the location is determined as the patch difference value between the image patch and the reference image patch;

[0049] The reference image block corresponding to the smallest tile difference value in the preset area is determined as an image block similar to the image block.

[0050] According to the present invention, a three-dimensional image reconstruction apparatus further includes:

[0051] The light field encoding module is used to perform light field encoding on the three-dimensional image to obtain encoded information; and to load the encoded information onto the light field display so that the light field display performs light field display based on the encoded information.

[0052] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the three-dimensional image reconstruction methods described above.

[0053] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional image reconstruction method as described above.

[0054] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the three-dimensional image reconstruction methods described above.

[0055] The present invention provides a three-dimensional image reconstruction method, apparatus and device, which, after obtaining multiple initial depth image pairs corresponding to multiple disparity image pairs, determines a target depth image pair based on the multiple disparity image pairs and the multiple initial depth image pairs; and performs three-dimensional reconstruction based on the target depth image pair and the color image pair corresponding to the target depth image pair, so that the matching accuracy between the multiple color image pairs and the multiple target depth image pairs is high, thereby improving the effect of three-dimensional reconstruction and thus improving the quality of the three-dimensional image. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating the three-dimensional image reconstruction method provided by the present invention;

[0058] Figure 2 This is a schematic diagram of a binocular camera provided by the present invention;

[0059] Figure 3 This is a schematic diagram of the geometric model of binocular camera imaging provided by the present invention;

[0060] Figure 4 This is a schematic flowchart of the method for obtaining a disparity image pair and an initial depth image pair corresponding to a color image pair provided by the present invention.

[0061] Figure 5 This is a schematic flowchart of the method for determining target depth image pairs provided by the present invention;

[0062] Figure 6This is a schematic flowchart of the method for determining an image block similar to an image block in a preset area provided by the present invention;

[0063] Figure 7 This is a schematic diagram of the preset area provided by the present invention;

[0064] Figure 8 This is a schematic diagram of the structure of the three-dimensional image reconstruction device of the present invention;

[0065] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] In this invention, the term "comprising" and its variations can refer to a non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this invention, "at least one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0068] In existing technologies, depth estimation is usually achieved using stereo matching algorithms. Based on multiple sets of color images, multiple sets of depth maps are obtained through depth estimation. However, when using stereo matching algorithms to achieve depth estimation, the matching accuracy between multiple sets of color images and multiple sets of depth maps is low, resulting in poor 3D reconstruction results and consequently poor 3D image quality.

[0069] To improve the quality of 3D images, this invention provides a 3D image reconstruction method. In this method, after obtaining multiple initial depth image pairs corresponding to multiple disparity image pairs, a target depth image pair is determined based on the multiple disparity image pairs and the multiple initial depth image pairs. Based on the target depth image pair and the color image pair corresponding to the target depth image pair, 3D reconstruction is performed, which makes the matching accuracy between the multiple color image pairs and the multiple target depth image pairs higher, improves the effect of 3D reconstruction, and thus improves the quality of 3D images.

[0070] The three-dimensional image reconstruction method provided by the present invention will be described below with reference to specific embodiments.

[0071] Figure 1 This is a schematic flowchart of the three-dimensional image reconstruction method provided by the present invention. Figure 1 As shown, the method includes:

[0072] Step 101: Obtain multiple color image pairs of the object to be scanned; the color image pairs include the left color image captured by the left eye camera of the binocular camera and the right color image captured by the right eye camera.

[0073] Optionally, the three-dimensional image reconstruction method provided by the present invention can be executed by an electronic device or a three-dimensional image reconstruction device installed in an electronic device. The three-dimensional image reconstruction device can be implemented through a combination of software and / or hardware.

[0074] Optionally, the object to be scanned can be a single person or object, an indoor scene, or multiple people or objects, an outdoor scene, etc. There is no limitation on the object to be scanned here.

[0075] Optionally, the binocular camera can take multiple shots of the object to be scanned from multiple shooting angles to obtain multiple color image pairs.

[0076] Optionally, the R channel of the RGB values ​​of the left and right color images has a range of [0, 255], the G channel has a range of [0, 255], and the B channel has a range of [0, 255].

[0077] The following is combined Figure 2 The model of the binocular camera used in this invention will be described.

[0078] Figure 2 This is a schematic diagram of a binocular camera provided by the present invention. Figure 2 As shown, the binocular camera includes a horizontally placed left-eye camera and a right-eye camera. The distance between the left-eye and right-eye cameras is the baseline of the binocular camera. The left pixel is the image point of the target point (e.g., a location of the object to be scanned) on the imaging plane, and the right pixel is the image point of the target point on the imaging plane.

[0079] Figure 3 This is a schematic diagram of the geometric model of binocular camera imaging provided by the present invention. Figure 3 As shown, s is the imaging plane, O L O is the center of the left aperture of the left-eye camera. R y is the center of the right aperture of the right eye camera, f is the focal length (i.e., the distance from the binocular camera to the imaging plane), and y is the imaging plane.

[0080] P is the target point, PL Let P be the imaging point of P on the imaging plane (corresponding to the left pixel). R Let U be the imaging point of P on the imaging plane (corresponding to the right pixel). L For P L The coordinates U on the imaging plane R For P R The coordinates on the imaging plane, z is the vertical distance from P to the stereo camera.

[0081] Step 102: Determine the disparity image pairs and initial depth image pairs corresponding to each of the multiple color image pairs, and determine the target depth image pairs based on the multiple disparity image pairs and the multiple initial depth image pairs.

[0082] There is a one-to-one correspondence between color image pairs and disparity image pairs, and a one-to-one correspondence between disparity image pairs and initial depth image pairs.

[0083] Optionally, a target depth image pair is determined based on multiple disparity image pairs and multiple initial depth image pairs using a preset denoising algorithm.

[0084] Step 103: Perform 3D reconstruction based on the target depth image pair and the corresponding color image pair to obtain a 3D image of the object to be scanned.

[0085] Optionally, a 3D reconstruction process is performed on the target depth image pair and the corresponding color image pair using a truncated signed distance function (TSDF) algorithm to obtain a 3D image of the object to be scanned.

[0086] Alternatively, the TSDF algorithm can be implemented using Compute Unified Device Architecture (CUDA) technology and Ray Casting Algorithm.

[0087] In this invention, multiple color image pairs are identified, each corresponding to a disparity image pair and an initial depth image pair. A target depth image pair is then determined based on these multiple disparity image pairs and initial depth image pairs, improving the quality of the target depth image pairs and resulting in higher matching accuracy between the multiple color image pairs and the multiple target depth image pairs. Furthermore, based on the target depth image pairs and their corresponding color image pairs, three-dimensional reconstruction is performed to obtain a three-dimensional image of the object to be scanned, resulting in better three-dimensional reconstruction performance and thus improving the quality of the three-dimensional image.

[0088] Figure 4This is a schematic flowchart of the method provided by the present invention for obtaining disparity image pairs and initial depth image pairs corresponding to color image pairs. Figure 4 As shown, the method includes:

[0089] Step 401: Based on the intrinsic and extrinsic parameters of the binocular camera, perform stereo correction on the color image pair to obtain the target color image pair.

[0090] Optionally, the intrinsic parameters of the binocular camera include the focal length of the left eye camera and the focal length of the right eye camera, the distortion coefficient, the center of the left and right apertures, and the distortion value.

[0091] Optionally, the distortion coefficients include radial distortion coefficients, tangential distortion coefficients, and other distortion type coefficients.

[0092] Optionally, the extrinsic parameters of the binocular camera include rotation matrix and translation vector, etc.

[0093] Optionally, the intrinsic and extrinsic parameters of the stereo camera can be obtained through the following steps one through three.

[0094] The first step is to use a binocular camera to take multiple pictures of the black and white grid calibration board to obtain multiple sets of images of the black and white grid calibration board.

[0095] The second step is to use a preset toolbox to calibrate multiple sets of black and white grid calibration board images to obtain multiple sets of black and white grid calibration images.

[0096] Optionally, the default toolbox can be the MATLAB toolbox, etc.

[0097] The third step is to obtain the intrinsic and extrinsic parameters of the stereo camera based on multiple sets of black and white grid calibration images.

[0098] There is a one-to-one correspondence between multiple color image pairs and multiple target color image pairs.

[0099] Each target color image pair includes a left target color image corresponding to the left color image and a right target color image corresponding to the right color image.

[0100] Step 402: Process the target color image pair using a preset matching algorithm to obtain the disparity image pair and the initial depth image pair corresponding to the color image pair.

[0101] Optionally, the preset matching algorithm can be the Slanted PatchMatch algorithm. The Slanted PatchMatch algorithm is obtained through a local stereo matching algorithm.

[0102] Optionally, the disparity image pairs and the initial depth image pairs corresponding to the color image pairs can be obtained through the following steps one through five.

[0103] The first step is to map the pixel values ​​of the target color image pairs to the feature space based on a deep learning model.

[0104] The second step is to obtain the RGB values ​​of each left pixel in the left target color image and the RGB values ​​of each right pixel in the right target color image in the feature space.

[0105] The third step is to determine the matching right pixel in the right target color image based on the RGB values ​​of the left pixel and the RGB values ​​of each right pixel for each left pixel, so as to obtain the matching result.

[0106] Optionally, the method for determining the right pixel that matches the left pixel includes finding the right pixel that matches the left pixel using an epipolar constraint method.

[0107] The matching results include the correspondence between each left pixel in all left target color images and its matching right pixel. Here, all left target color images refer to the left target color images in the target color image pair.

[0108] The fourth step is to perform stereo matching on the target color image pairs based on the matching results, and obtain the disparity image pairs corresponding to the color image pairs.

[0109] A parallax image pair consists of a left parallax image and a right parallax image.

[0110] The fifth step is to obtain the initial depth image pairs corresponding to the color image pairs based on the disparity image pairs.

[0111] In this invention, stereo correction is performed on the color image pairs based on the intrinsic and extrinsic parameters of the binocular camera to obtain the target color image pairs, which can compensate for distortion and thus improve the quality of the obtained target color image pairs. The target color image pairs are processed by a preset matching algorithm to obtain the disparity image pairs and initial depth image pairs corresponding to the color image pairs. This enables fast and optimal matching of the left target color image and the right target color image, thereby improving the speed and quality of the multiple initial depth image pairs obtained, and thus improving the quality of the three-dimensional image.

[0112] Figure 5 This is a schematic flowchart of the method for determining target depth image pairs provided by the present invention. Figure 5 As shown, the method includes:

[0113] Step 501: For each disparity image pair, determine the disparity constant value of the disparity image pair based on the disparity value of each pixel in the disparity image pair.

[0114] Optionally, the disparity constant value of the disparity image pair can be the standard deviation or variance of the disparity image pair.

[0115] For example, when the disparity constant is the standard deviation, the mean of the disparity image pair is determined based on the disparity value of each pixel, and the standard deviation of the disparity image pair is determined based on the mean of the disparity image pair.

[0116] Step 502: Determine an intermediate depth image pair from multiple initial depth image pairs. The disparity constant value of the disparity image pair corresponding to the intermediate depth image pair is less than or equal to a preset threshold.

[0117] Step 503: Determine the target depth image pair based on the intermediate depth image pair.

[0118] Alternatively, the target depth image pair can be determined using the following methods 11 or 12.

[0119] Method 11: Determine the intermediate depth image pair as the target depth image pair.

[0120] Method 12, for each intermediate depth image in the intermediate depth image pair, performs the following operations:

[0121] The intermediate depth image is divided into multiple non-overlapping image blocks; for each image block, an image block similar to the image block is determined in the preset region corresponding to the image block. The preset region is the region that includes the image block or the adjacent region of the image block.

[0122] Based on the position of each image patch in the intermediate depth image, image patches similar to the image patches are stitched together to obtain the target depth image corresponding to the intermediate depth image.

[0123] The target depth image pair includes the target depth image corresponding to the intermediate depth image of the intermediate depth image pair.

[0124] It should be noted that, Figure 5 The method provided in the embodiments is a detailed process of a preset denoising algorithm.

[0125] exist Figure 5 In this embodiment, an intermediate depth image pair is determined from multiple initial depth image pairs, and a target depth image pair is determined based on the intermediate depth image pair. This can remove noise from the initial depth images, thereby improving the quality of the target depth image pair and thus improving the quality of the 3D image.

[0126] Based on method 12 above, the following combines... Figure 6 The embodiment describes the process of determining an image block similar to the image block in a preset area corresponding to the image block.

[0127] Figure 6 This is a schematic flowchart of a method for determining image blocks similar to image blocks within a preset area, provided by the present invention. Figure 6 As shown, the method includes:

[0128] Step 601: For each reference image block in the preset area, determine the depth difference between two pixels whose positions match in the image block and the reference image block.

[0129] Figure 7 This is a schematic diagram of the preset area provided by the present invention. For example... Figure 7 As shown, for example, the preset region corresponding to image block A includes: image block A and reference image blocks 1 to 8. Reference image blocks 1 to 8 are image blocks within the preset region.

[0130] For example, image block A includes pixels PA1 and PA2, and reference image block 2 includes pixels P21 and P22. PA1 and P21 are two pixels at the same location in image block A and reference image block 2, and PA2 and P22 are two pixels at another same location in image block A and reference image block 2.

[0131] It should be noted that, Figure 7 This explanation uses a preset region of size 3×3 centered on image block A as an example. For example, the preset region could also include reference image blocks 5, 7, and 8.

[0132] Step 602: The sum of the depth differences between two pixels with matching positions is determined as the patch difference value between the image patch and the reference image patch.

[0133] Step 603: Determine the reference image block corresponding to the smallest tile difference value in the preset area as an image block similar to the image block.

[0134] For example, in Figure 7 Based on the above, the tile difference value between reference image block 1 and image block A is 18, the tile difference value between reference image block 2 and image block A is 16, the tile difference value between reference image block 3 and image block A is 30, the tile difference value between reference image block 4 and image block A is 11, the tile difference value between reference image block 5 and image block A is 18, the tile difference value between reference image block 6 and image block A is 22, the tile difference value between reference image block 7 and image block A is 16, and the tile difference value between reference image block 8 and image block A is 10.

[0135] Furthermore, since the tile difference value 10 between reference image block 8 and image block A is the smallest tile difference value, the image block similar to image block A is reference image block 8.

[0136] In some embodiments, the three-dimensional image reconstruction method provided by the present invention further includes:

[0137] Light field encoding is performed on the 3D image to obtain the encoded information;

[0138] The encoded information is loaded into the light field display so that the light field display can perform light field display based on the encoded information.

[0139] In this invention, a three-dimensional image is light field encoded to obtain encoded information, which is then loaded onto a light field display so that the light field display can perform light field display based on the encoded information, resulting in a better light field display effect and thus improving the user's viewing experience.

[0140] The three-dimensional image reconstruction apparatus provided by the present invention is described below. The three-dimensional image reconstruction apparatus described below can be referred to in correspondence with the three-dimensional image reconstruction method described above.

[0141] Figure 8 This is a schematic diagram of the structure of the three-dimensional image reconstruction device of the present invention. Figure 8 As shown, the three-dimensional image reconstruction device includes:

[0142] The image acquisition module 810 is used to acquire multiple color image pairs of the object to be scanned; the color image pairs include the left color image captured by the left eye camera of the binocular camera and the right color image captured by the right eye camera;

[0143] The stereo matching module 820 is used to determine the disparity image pairs and initial depth image pairs corresponding to each of the multiple color image pairs;

[0144] The depth denoising module 830 is used to determine the target depth image pair based on multiple disparity image pairs and multiple initial depth image pairs;

[0145] The 3D reconstruction module 840 is used to perform 3D reconstruction based on the target depth image pair and the corresponding color image pair to obtain a 3D image of the object to be scanned.

[0146] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0147] According to the present invention, a three-dimensional image reconstruction apparatus includes a stereo matching module 820 comprising:

[0148] The parameter acquisition and stereo correction unit is specifically used to perform stereo correction on the color image pair based on the intrinsic and extrinsic parameters of the binocular camera to obtain the target color image pair;

[0149] The stereo matching unit is specifically used to process the target color image pair using a preset matching algorithm to obtain the disparity image pair and the initial depth image pair corresponding to the color image pair.

[0150] According to the present invention, a three-dimensional image reconstruction apparatus, including a depth denoising module 830, is specifically used for:

[0151] For each disparity image pair, the disparity constant value of the disparity image pair is determined based on the disparity value of each pixel in the disparity image pair;

[0152] An intermediate depth image pair is determined from multiple initial depth image pairs, and the disparity constant value of the disparity image pair corresponding to the intermediate depth image pair is less than or equal to a preset threshold.

[0153] The target depth image pair is determined based on the intermediate depth image pair.

[0154] According to the present invention, a three-dimensional image reconstruction apparatus, including a depth denoising module 830, is specifically used for:

[0155] Perform the following operations on each intermediate depth image in the intermediate depth image pair:

[0156] The intermediate depth image is divided into multiple non-overlapping image blocks; for each image block, an image block similar to the image block is determined in the preset region corresponding to the image block. The preset region is the region that includes the image block or the adjacent region of the image block.

[0157] Based on the position of each image patch in the intermediate depth image, image patches similar to the image patches are stitched together to obtain the target depth image corresponding to the intermediate depth image.

[0158] The target depth image pair includes the target depth image corresponding to the intermediate depth image of the intermediate depth image pair.

[0159] According to the present invention, a three-dimensional image reconstruction apparatus, including a depth denoising module 830, is specifically used for:

[0160] For each reference image block in the preset area, determine the depth difference between two pixels whose positions match in the image block and the reference image block;

[0161] The sum of the depth differences between two pixels that are matched in position is determined as the patch difference value between the image patch and the reference image patch;

[0162] The reference image block corresponding to the smallest tile difference value in the preset area is identified as an image block similar to the image block.

[0163] According to the present invention, a three-dimensional image reconstruction apparatus further includes:

[0164] The light field encoding module is used to encode the 3D image using light field to obtain encoded information; the encoded information is then loaded onto the light field display so that the light field display can perform light field display based on the encoded information.

[0165] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9 As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a three-dimensional image reconstruction method, which includes: acquiring multiple color image pairs of the object to be scanned; the color image pairs include a left color image captured by the left eye camera of a binocular camera and a right color image captured by the right eye camera; determining the disparity image pairs and initial depth image pairs corresponding to each of the multiple color image pairs, and determining a target depth image pair based on the multiple disparity image pairs and the multiple initial depth image pairs; and performing three-dimensional reconstruction based on the target depth image pair and the color image pairs corresponding to the target depth image pair to obtain a three-dimensional image of the object to be scanned.

[0166] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the three-dimensional image reconstruction method provided by the above methods. The method includes: acquiring multiple color image pairs of an object to be scanned; the color image pairs include a left color image captured by the left eye camera of a binocular camera and a right color image captured by the right eye camera; determining the disparity image pairs and initial depth image pairs corresponding to each of the multiple color image pairs, and determining a target depth image pair based on the multiple disparity image pairs and the multiple initial depth image pairs; and performing three-dimensional reconstruction based on the target depth image pair and the color image pairs corresponding to the target depth image pair to obtain a three-dimensional image of the object to be scanned.

[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the three-dimensional image reconstruction method provided by the above methods. The method includes: acquiring multiple color image pairs of an object to be scanned; the color image pairs include a left color image captured by the left eye camera of a binocular camera and a right color image captured by the right eye camera; determining a disparity image pair and an initial depth image pair corresponding to each of the multiple color image pairs, and determining a target depth image pair based on the multiple disparity image pairs and the multiple initial depth image pairs; and performing three-dimensional reconstruction based on the target depth image pair and the color image pair corresponding to the target depth image pair to obtain a three-dimensional image of the object to be scanned.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional image reconstruction method, characterized in that, include: Acquire multiple color image pairs of the object to be scanned; the color image pairs include a left color image captured by the left eye camera of the binocular camera and a right color image captured by the right eye camera; The process involves determining the disparity image pairs and initial depth image pairs corresponding to each of the plurality of color image pairs, and determining a target depth image pair based on the plurality of disparity image pairs and initial depth image pairs. The determination of the target depth image pair based on the plurality of disparity image pairs and initial depth image pairs includes: for each disparity image pair, determining a disparity constant value for the disparity image pair based on the disparity values ​​of each pixel in the disparity image pair; determining an intermediate depth image pair from the plurality of initial depth image pairs, wherein the disparity constant value of the disparity image pair corresponding to the intermediate depth image pair is less than or equal to a preset threshold; and determining the target depth image pair based on the intermediate depth image pair. The method for determining the target depth image pair includes: performing the following operations on each intermediate depth image in the intermediate depth image pair: dividing the intermediate depth image into multiple non-overlapping image blocks; for each image block, determining an image block similar to the image block in a preset region corresponding to the image block, wherein the preset region is a region including the image block or an adjacent region of the image block; and stitching the image blocks similar to the image block together based on the position of each image block in the intermediate depth image to obtain a target depth image corresponding to the intermediate depth image; the target depth image pair includes the target depth image corresponding to the intermediate depth image of the intermediate depth image pair. Three-dimensional reconstruction is performed based on the target depth image pair and the corresponding color image pair to obtain a three-dimensional image of the object to be scanned.

2. The three-dimensional image reconstruction method according to claim 1, characterized in that, The step of determining the disparity image pairs and initial depth image pairs corresponding to each of the plurality of color image pairs includes: Perform the following operations on each color image pair: Based on the intrinsic and extrinsic parameters of the binocular camera, stereo correction is performed on the color image pair to obtain a target color image pair. The target color image pair is then processed by a preset matching algorithm to obtain the disparity image pair and the initial depth image pair corresponding to the color image pair.

3. The three-dimensional image reconstruction method according to claim 1, characterized in that, Determining an image block similar to the image block within a preset region corresponding to the image block includes: For each reference image block in the preset region, determine the depth difference between two pixels whose positions match in the image block and the reference image block; The sum of the depth differences between the two pixels that are matched at the location is determined as the patch difference value between the image patch and the reference image patch; The reference image block corresponding to the smallest tile difference value in the preset area is determined as an image block similar to the image block.

4. The three-dimensional image reconstruction method according to claim 1 or 2, characterized in that, The method further includes: The three-dimensional image is optically encoded to obtain encoded information; The encoded information is loaded into the light field display so that the light field display performs light field display based on the encoded information.

5. A three-dimensional image reconstruction device, characterized in that, include: The image acquisition module is used to acquire multiple color image pairs of the object to be scanned; the color image pairs include a left color image captured by the left eye camera of the binocular camera and a right color image captured by the right eye camera; A stereo matching module is used to determine the disparity image pairs and initial depth image pairs corresponding to each of the plurality of color image pairs; A depth denoising module is used to determine a target depth image pair based on multiple disparity image pairs and multiple initial depth image pairs. Specifically, the depth denoising module is used to determine the disparity constant value of each disparity image pair based on the disparity value of each pixel in the disparity image pair. An intermediate depth image pair is determined from the multiple initial depth image pairs, wherein the disparity constant value of the disparity image pair corresponding to the intermediate depth image pair is less than or equal to a preset threshold. Based on the intermediate depth image pair, the target depth image pair is determined; the determination of the target depth image pair based on the intermediate depth image pair includes: performing the following operation on each intermediate depth image in the intermediate depth image pair: dividing the intermediate depth image into multiple non-overlapping image blocks; For each image block, an image block similar to the image block is determined in a preset region corresponding to the image block. The preset region is a region including the image block or an adjacent region of the image block. Based on the position of each image block in the intermediate depth image, the image blocks similar to the image block are stitched together to obtain a target depth image corresponding to the intermediate depth image. The target depth image pair includes the target depth image corresponding to the intermediate depth image of the intermediate depth image pair. The 3D reconstruction module is used to perform 3D reconstruction based on the target depth image pair and the corresponding color image pair to obtain a 3D image of the object to be scanned.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the three-dimensional image reconstruction method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional image reconstruction method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional image reconstruction method as described in any one of claims 1 to 4.

Images