3D Image Generation Method, Apparatus, and Computer Device

Abstract

A 3D image generation method, comprising: separating a target 2D image to obtain an initial color image and an initial depth image; obtaining a first color image and a target depth image including a plurality of reference pixel points by associating and processing the initial color image and the initial depth image; determining a target pixel point that is a pixel point existing as a hole in the first color image based on the depth value of the reference pixel point; determining a hole filling value of the target pixel point based on the reference pixel point and performing filling to generate a target color image; and interleaving the initial color image and the target color image to generate a target 3D image. The present invention further provides a 3D image generation device and a computer device.

Description

Technical Field

[0001] The present application relates to the field of 3D display technology, and specifically to a 3D image generation method, apparatus, and computer device.

Background Art

[0002] An RGBD image includes two images, one is a normal RGB 3-channel color image, and the other is a depth image. RGB contains graphic color information, and the depth image contains depth information. The 3-channel RGB color image is rendered from different angles to generate a new color image, and further, the generated color image is rendered at different angles to obtain a 3D model of the model texture.

[0003] In the prior art of 3D image generation methods, when a color image is generated by rendering from a certain angle, there are some hollow places in the newly generated color image formed by rendering. Therefore, it is necessary to provide a 3D image generation method that can fill the hollow.

Summary of the Invention

[0004] The present application provides a 3D image generation method, apparatus, and computer device capable of filling holes, which solve the problem in the prior art that there are holes in the color image after a new color image is generated by rendering, and fill the holes in the color image generated by rendering, and further generate a 3D image using the color image without holes.

[0005] The 3D image generation method according to the first aspect of the present invention is as follows: separating a target 2D image to obtain an initial color image and an initial depth image; associating and processing the initial color image and the initial depth image to respectively obtain a first color image and a target depth image including a plurality of reference pixel points in association; Determining a target pixel point that is a pixel point existing as a hole in the first color image based on the depth value of the reference pixel point; Determining a hole filling value for the target pixel point based on the reference pixel point, performing filling, and generating a target color image; Interleaving the initial color image and the target color image to generate a target 3D image.

[0006] A 3D image generation device according to a second aspect of the present invention includes: A separation unit that separates a target 2D image to obtain an initial color image and an initial depth image; A processing unit that associates and processes the initial color image and the initial depth image to respectively obtain and associate a first color image and a target depth image including a plurality of reference pixel points; A first determination unit that determines a target pixel point that is a pixel point existing as a hole in the first color image based on the depth value of the reference pixel point; A second determination unit that determines a hole filling value for the target pixel point based on the reference pixel point, performs filling, and generates a target color image; A generation unit that interleaves the initial color image and the target color image to generate a target 3D image.

[0007] A computer device according to a third aspect of the present invention includes at least one processor, a memory, and a transceiver connected to each other. Here, the memory is used to store program code. The processor calls the program code in the memory and executes the 3D image generation steps described in the first aspect above.

[0008] Compared with the prior art, the 3D image generation method and the 3D image generation apparatus provided by the present application separate a target 2D image to obtain an initial color image and an initial depth image, and associate and process the initial color image and the initial depth image, thereby respectively obtaining a first color image and a target depth image in association with each other. A target pixel point, which is a pixel point existing as a hole in the first color image, is determined by a reference pixel point in the target depth image. Then, based on the reference pixel point, a hole filling value of the target pixel point is determined to complete the filling of the hole in the first color image, and a target color image without a hole is obtained. Further, the initial color image and the initial depth image are interleaved to generate a target 3D image without a hole.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0010] The following clearly and completely describes the technical aspects in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the present application.

[0011] The present invention provides a 3D image generation method. The 3D image generation method provided by the present application is described below from the perspective of a 3D image generation device. The aforementioned 3D image generation device may be a terminal device such as a mobile phone or a tablet, or may be other devices such as a server.

[0012] FIG. 1 is a flowchart of a 3D image generation method provided by an embodiment of the present invention. As shown in FIG. 1, the 3D image generation method includes the following steps.

[0013] In step S01, a target 2D image is separated to obtain an initial color image and an initial depth image.

[0014] In this embodiment, the 3D image generation device separates the target 2D image of the 3D image to be generated to obtain the initial color image and the initial depth image. The target 2D image is an rgbd image and includes two images. One is a normal rgb 3-channel color image containing graphics color information, and the other is a depth image containing depth information. The initial color image is an rgb 3-channel color image and includes a plurality of pixel points. Each pixel point is represented by a coordinate value (x, y) and has a pixel value. The pixel value is rgb color information. The initial depth image is a depth image and includes a plurality of pixel points. The pixel points of the initial depth image have coordinates corresponding to the target depth image. Each pixel point of the initial depth image has a depth value representing depth information.

[0015] In step S02, by associating and processing the initial color image and the initial depth image, a first color image and a target depth image are obtained by corresponding association respectively. The target depth image includes a plurality of reference pixel points.

[0016] In this embodiment, the 3D image generation device processes the initial color image and the initial depth image in association with each other to obtain the first color image and the target depth image. The target depth image includes a plurality of reference pixel points, and the reference pixel point is any pixel point of the target depth image. Hereinafter, two related processing methods will be described respectively.

[0017] Next, a first related processing method will be described with reference to FIG. 2. FIG. 2 is a flowchart of a method for determining a first color image and a target depth image provided according to an embodiment of the present invention. The flowchart includes the following steps.

[0018] In step S021, an initial point cloud corresponding to the first depth image is determined.

[0019] In this embodiment, after obtaining the initial depth image, the aforementioned initial depth image is remapped to the first depth image by the following formula. JPEG2025521360000002.jpg25170

[0020] Here, I d is the initial depth image, I d1 is the first depth image, I d (z) is the depth value of any pixel point in the initial depth image, and Dmax is the maximum depth value among all pixel points of the initial depth image. MaxDepth is an empirical value and can take a value of 100. Of course, it can also take other values such as 80, 90, 110, etc., and is not specifically limited as long as it does not exceed the maximum floating-point number. The coordinates of each pixel point of the remapped first depth image correspond one-to-one to the coordinates of each pixel point in the initial depth image. I d1 (z) is the depth value of the pixel point whose coordinates are the same as the coordinates of I d (z) in the first depth image.

[0021] After obtaining the first depth image, the aforementioned first depth image is converted into the aforementioned initial point cloud by the following formula. JPEG2025521360000003.jpg19170

[0022] Here, 0 ≤ x < w, 0 ≤ y < h, P0 is the initial point cloud, P0(x, y, z) is the coordinate of any one point in the initial point cloud, w is the width of the first depth image, h is the height of the first depth image, and I d1 (x, y) is the depth value of the pixel point with coordinates (x, y) in the first depth image.

[0023] In step S022, by presetting the relative displacement, the coordinates of each point in the initial point cloud are adjusted to obtain a target point cloud.

[0024] In this embodiment, in accordance with FIG. 3, the preset relative displacement is described. FIG. 3 is a schematic diagram of the relative displacement of the camera during the rendering of the initial color image provided by the embodiment of the present application. When rendering the initial color image, the position of the camera 301 is C0. It is defined that the space where the camera 301 is located is a three-dimensional space defined by the x-axis, y-axis, and z-axis, and the coordinates of C0 are set as (0, 0, z). By changing the position of the camera 301, different rendering images are obtained, and different rendering images corresponding to when the camera 301 is in different positions are interleaved and displayed to generate a 3D display image. In order to make the sizes of different rendering images consistent, the camera 301 does not change in the z-axis. The changed position of the camera 301 is C1. The coordinates of C1 are (nx, ny, z). The change in the position of the camera 301 is the aforementioned preset relative displacement. The aforementioned preset relative displacement of the camera 301 is determined by the following formula. JPEG2025521360000004.jpg23170

[0025] Here, D is the preset relative displacement of the camera 301.

[0026] Furthermore, a preset relative displacement is added to the initial point cloud to obtain the aforementioned target point cloud. Specifically, the target point cloud is determined by the following formula. JPEG2025521360000005.jpg23170

[0027] Here, P0(x, y, z) is the coordinate of any one point among the initial point cloud, P1 is the aforementioned target point cloud, and P1(x, y, z) is the coordinate of the point obtained by adding D to P0(x, y, z) among the aforementioned target point cloud. Since the position of the camera does not change on the z-axis and the D value is (nx, ny, 0), the z values of the coordinates of the aforementioned target point cloud and the aforementioned initial point cloud are the same.

[0028] In step S023, the coordinates of each point in the target point cloud are processed to obtain a reference image.

[0029] In this embodiment, the aforementioned 3D image generation device processes the coordinates of each point in the target point cloud by the following formula to determine the aforementioned reference image. The aforementioned reference image is a depth image that matches the size of the initial depth image. The initial substitution value of the depth value of the aforementioned reference image is A. JPEG2025521360000006.jpg19170

[0030] Here, Z is the aforementioned reference image. Z(x, y) is the depth value of the pixel point whose coordinate in the aforementioned reference image is (x, y). Z(IP.x + 1, IP.y + 1) is the depth value of the pixel point whose coordinate in the aforementioned reference image is (IP.x + 1, IP.y + 1). FltErr = A - w / (2*z0). min is the value obtained by assigning the smaller value between Z(IP.x + 1, IP.y + 1) and FltErr to Z(x, y).

[0031] JPEG2025521360000007.jpg32170A is the initial depth value of the aforementioned reference depth map, w is the width of the first depth image, (x0, y0, z0) is the coordinate of any one point among the aforementioned target point clouds, IP.x is the x value of the point IP coordinate, and IP.y is the y value of the point IP coordinate.

[0032] Here, the reference image is a depth image that matches the size of the first depth image. The width of the reference image matches the width w of the first depth image. The height of the reference image matches the height h of the first depth image. The initial depth value of the reference image is A. The value of A may be 100000.0, or 90000.0, or 110000.0. As long as the value of A is greater than the depth value of the first depth image and less than the maximum floating-point number, it is not specifically limited.

[0033] Note that after determining the reference image, in order to optimize the finally generated target 3D image, the depth value of each pixel point in the reference image may be optimized by the following steps.

[0034] Next, with reference to FIG. 4, the optimization of the pixel points of the reference image will be described in detail. FIG. 4 is a schematic diagram of the first reference pixel point and four pairs of diagonal pixel points of the reference image according to the embodiment of the present application.

[0035] Determine the first reference pixel point. The first reference pixel point is the pixel point in the reference image whose depth value is to be optimized. When the depth value of the first reference pixel point is greater than the depth values of its four pairs of diagonal pixel points, the average value of the depth values of the four pairs of diagonal pixel points is determined as the depth value of the aforementioned first reference pixel point.

[0036] It is understood that the first reference pixel point may be any pixel point of the aforementioned reference image. Assuming the coordinate values of the first reference pixel point are (x, y), the optimization process will be described. Each of the four pairs of diagonal pixel points has coordinates that are the pixels above, below, to the left, to the right of the first reference pixel point, and two pairs of diagonal pixels (as shown in Figure 4). The vertically diagonal image points are the image points with coordinates (x, y - 1) and (x, y + 1) respectively. The horizontally diagonal image points are the image points with coordinates (x - 1, y) and (x + 1, y) respectively. The two pairs of diagonally diagonal image points are the image points with coordinates (x - 1, y - 1), (x + 1, y + 1), (x + 1, y - 1), and (x - 1, y + 1) respectively.

[0037] Next, with reference to Figure 4, a method for optimizing the aforementioned first reference pixel point will be described. The optimization method of the aforementioned first reference pixel point includes the following steps.

[0038] In step A1, if the depth value of the first reference pixel point satisfies Z(x, y)>Z(x - 1, y) and Z(x, y)>Z(x + 1, y), the following formula is satisfied. JPEG2025521360000008.jpg17170

[0039] In step A2, if the depth value of the first reference pixel point satisfies Z(x, y)>Z(x, y - 1) and Z1(x, y)>Z(x, y + 1), the following formula is satisfied. JPEG2025521360000009.jpg19170

[0040] In step A3, if the depth value of the first reference pixel point satisfies Z(x, y)>Z(x - 1, y - 1) and Z(x, y)>Z(x + 1, y + 1), the following formula is satisfied. JPEG2025521360000010.jpg16170

[0041] In step A4, if the depth value of the first reference pixel point satisfies Z(x, y)>Z(x + 1, y - 1) and Z(x, y)>Z(x - 1, y + 1), the following formula is satisfied. JPEG2025521360000011.jpg24170

[0042] In step A5, the average value is obtained by the following formula 11. JPEG2025521360000012.jpg18170

[0043] Here, the initial substitution values of Zsum and Ztol are 0. If the depth value of the first reference pixel point with coordinate values (x, y) satisfies any of the conditions in steps A1 to A4, a new depth value is obtained and the original depth value of the aforementioned first reference pixel point is overwritten.

[0044] However, Z(x, y) is the depth value of the first reference pixel point in the aforementioned reference image, Z(x + 1, y) is the depth value of the pixel point with coordinates (x + 1, y) in the aforementioned reference image, Z(x - 1, y) is the depth value of the pixel point with coordinates (x - 1, y) in the aforementioned reference image, Z(x, y - 1) is the depth value of the pixel point with coordinates (x, y - 1) in the aforementioned reference image, Z(x, y + 1) is the depth value of the pixel point with coordinates (x, y + 1) in the aforementioned reference image, Z(x + 1, y + 1) is the depth value of the pixel point with coordinates (x + 1, y + 1) in the aforementioned reference image, Z(x - 1, y - 1) is the depth value of the pixel point with coordinates (x - 1, y - 1) in the aforementioned reference image, Z(x - 1, y + 1) is the depth value of the pixel point with coordinates (x - 1, y + 1) in the aforementioned reference image, and Z(x + 1, y - 1) is the depth value of the pixel point with coordinates (x + 1, y - 1) in the aforementioned reference image.

[0045] Note that in steps A1, A2, A3, and A4, it is determined whether the depth value of the first reference pixel point is greater than the depth values of its four pairs of diagonal pixel points, and in step A5, its average value is obtained. However, there is no constraint on the priority execution order among steps A1, A2, A3, and A4. Step A1 may be executed first, step A2 may be executed first, step A3 may be executed first, or step A4 may be executed first, and it is not specifically limited.

[0046] In step S024, based on the depth values of the pixel points in the reference image, the pixel points in the initial color image and the pixel points in the initial depth image are processed to obtain the first color image and the target depth image.

[0047] In this embodiment, the aforementioned first color image is obtained by the following formula. JPEG2025521360000013.jpg20170

[0048] Here, I c is the initial color image, I c1 is the first color image, IP.x is the x value of the point IP coordinates, and IP.y is the y value of the point IP coordinates. I c (x, y) is the pixel value of the pixel point with coordinates (x, y) in the initial color image, Z(x, y) is the depth value of the pixel point with coordinates (x, y) in the reference image, and I c1 (IP.x, IP.y) is the pixel value of the pixel point with coordinates (IP.x, IP.y) in the first color image.

[0049] When (Z(x, y) + 1) > FltErr is satisfied, if the value of ((Z(x, y) + 1) > FltErr) is set to 1, the pixel value of the pixel point with coordinates (x, y) in the aforementioned initial depth image is assigned to the pixel point with coordinates (IP.x, IP.y) in the aforementioned first color image.

[0050] If ((Z(x,y)+1)>FltErr) is not satisfied and the value of ((Z(x,y)+1)>FltErr) is set to 0, the pixel value of the pixel point with coordinates (IP.x, IP.y) in the aforementioned first color image is 0.

[0051] The aforementioned target depth image is obtained by the following formula. JPEG2025521360000014.jpg19170

[0052] Here, I d is the initial depth image, I d2 is the target depth image, I d (x,y) is the depth value of the pixel point with coordinates (x,y) in the initial depth image, and I d2 (IP.x, IP.y) is the depth value of the pixel point with coordinates (IP.x, IP.y) in the target depth image. When the value of ((Z(x,y)+1)>FltErr) is 1, the depth value of the pixel point with coordinates (x,y) in the initial depth image is assigned to the pixel point with coordinates (IP.x, IP.y) in the target depth image. When the value of ((Z(x,y)+1)>FltErr) is 0, the depth value of the pixel point with coordinates (IP.x, IP.y) in the target depth image is 0.

[0053] In this method, the initial color image and the initial depth image are associated and processed by the initial point cloud, the target point cloud, and the reference image. Therefore, the pixel value of the pixel point in the obtained first color image and the depth value of the pixel point in the target depth image are associated.

[0054] Next, a second type of association processing method will be described. The second type of association processing method includes the following steps.

[0055] In step C1, the preset depth of field of the aforementioned target 3D image is determined. The preset depth of field of the target 3D image on the x-axis is nx0 to nx1, and the preset depth of field on the y-axis is ny0 to ny1.

[0056] In this embodiment, the preset depth of field is a predetermined parallax, which is a certain pixel unit. The x-axis is parallel to the width of the 3D image, and the y-axis direction is parallel to the height of the 3D image. For example, when the preset depth of field on the x-axis of the target 3D image is 10 to 100 pixel units, nx0 is 10 pixel units and nx1 is 100 pixel units. The preset depth of field may be 0 pixel units. For example, there is no parallax on the y-axis of the target 3D image. If the preset depth of field is 0 pixel units, ny0 is 0 pixel units and ny1 is 0 pixel units.

[0057] In step C2, the ratio between the preset depth of field and the depth range of the initial depth image is obtained by the following formula. JPEG2025521360000015.jpg25170

[0058] Here, DepthRateX is the ratio between the preset depth of field on the x-axis and the initial image depth range, DepthRateY is the ratio between the preset depth of field on the y-axis and the initial image depth range, Dmax is the maximum depth value of the initial depth image, and Dmin is the minimum depth value of the initial depth image.

[0059] In step C3, Pos_x and Pos_y are obtained by the following formula. JPEG2025521360000016.jpg24170

[0060] However, I d is the initial depth image, and I d (x, y) is the depth value of the pixel point with coordinates (x, y) in the initial depth image, x is the x value of the coordinates at the pixel point with coordinates (x, y), and y is the y value of the coordinates at the pixel point with coordinates (x, y).

[0061] In step C4, the first color image and the target depth image are obtained by the following formula. JPEG2025521360000017.jpg28170 Here, Ic is the initial color image, I c1 is the first color image, I c (x, y) is the pixel value of the image point with coordinates (x, y) in the initial color image, I c1 (Pos_x, Pos_y) is the pixel value of the pixel point with coordinates (Pos_x, Pos_y) in the first color image, I d is the initial depth image, I d2 is the target depth image, I d2 (Pos_x, Pos_y) is the depth value of the pixel point with coordinates (Pos_x, Pos_y) in the target depth image. The above formula means that the pixel value of the pixel point with coordinates (x, y) in the initial color image is assigned to the pixel point with coordinates (Pos_x, Pos_y) in the first color image, and the depth value of the pixel point with coordinates (x, y) in the initial depth image is assigned to the pixel point with coordinates (Pos_x, Pos_y) in the target depth image.

[0062] This method associates and processes the initial color image and the initial depth image based on Pos_x and Pos_y, and associates the offset amount of the first color image with respect to the initial color image with the depth value of the first depth image. The larger the depth value of the initial depth image, the larger the offset amount of the pixel point with coordinates (Pos_x, Pos_y) in the first depth image with respect to the pixel point with coordinates (x, y) in the initial depth image. The pixel value of the pixel point in the first color image obtained by this method and the depth value of the pixel point in the target depth image are associated. The target depth image includes a plurality of reference pixel points.

[0063] In step S03, target pixel points are determined based on the depth values of the aforementioned reference pixel points.

[0064] In this embodiment, after determining the first color image and the target depth image, based on the depth value of the reference pixel point, it is determined whether the pixel point with the same coordinates as the reference pixel point in the first color image is a hole, and the pixel point existing as a hole in the first color image is determined as the target pixel point.

[0065] Next, a method for determining the target pixel points in the first color image will be described.

[0066] I d2 If (x,y) ≤ 0 is satisfied, the pixel point with coordinates (x,y) in the first color image is determined as the target pixel point existing as a hole. Here, I d2 is the aforementioned target depth image, and I d2 (x,y) is the depth value of the aforementioned reference pixel point, and the aforementioned reference pixel point is the pixel point with coordinates (x,y) in the target depth image. That is, I d2 when (x,y) ≤ 0 is satisfied, the pixel value of the image point with coordinates (x,y) in the first color image is 0 and there is no coloring value, so the image point exists as a hole.

[0067] In step S04, the hole filling value of the target pixel point is determined based on the aforementioned reference image point.

[0068] In this embodiment, after determining the target pixel points existing as holes in the first color image, the 3D image generation device determines the hole filling value of the target pixel points based on the coordinate value and pixel value of the reference pixel point, fills the holes in the first color image based on the hole filling value of the target pixel points, and generates a target color image after filling.

[0069] Hereinafter, how to determine the hole filling value of the aforementioned target pixel points will be described. The method for determining the hole filling value of the aforementioned target pixel points includes the following steps.

[0070] In step B1, a preset path traversed by the aforementioned reference image point is set.

[0071] In this embodiment, the preset path may be provided according to the actual situation. For example, the number of preset paths may be 16, or 6, or 5, and is not specifically limited, and may be debugged according to the generation situation of the aforementioned target 3D image.

[0072] Hereinafter, an example in which there are 16 preset paths will be given to specifically describe the traversal method.

[0073] The aforementioned preset path is represented by Dirs, and 16 traversal search directions are preset, where Dirs = (-1, 1), (0, 1), (1, 1), (1, 0), (-1, 2), (1, 2), (2, 1), (2, -1), (-2, 3), (-1, 3), (1, 3), (2, 3), (3, 2), (3, 1), (3, -1), (3, -2). Refer to FIG. 5 together. FIG. 5 is a schematic diagram of the preset path provided by the embodiment of the present application. The preset path Dirs shown in FIG. 5 satisfies Dirs = (-2, 3).

[0074] In step B2, based on the coordinates of the aforementioned reference pixel point, traverse the aforementioned preset path to determine a first target pixel point and a second target pixel point that meet the preset conditions.

[0075] Next, in conjunction with FIG. 5, the traversal procedure of the aforementioned reference pixel point will be described. Starting from the aforementioned reference pixel point, with the coordinates of the aforementioned reference pixel point being (x, y), traverse in each preset direction according to the following procedure.

[0076] In step B21, according to the following formula, I d2 Perform negative-direction traversal until (FromX, FromY) > 0 or either FromX or FromY exceeds the boundary of the aforementioned target depth image. JPEG2025521360000018.jpg31170

[0077] Here, [i] in Dirs[i][0] and Dirs[i][1] indicates the preset path to be traversed. [0] indicates that Dirs[i][0] takes the value on the left side of the preset path coordinate value, and [1] indicates that Dirs[i][1] takes the value on the right side of the preset path coordinate value. For example, if the preset path is Dirs = (-2, 3), and traversing in the negative direction with (-2, 3) as the preset direction (as shown in Figure 5), JPEG2025521360000019.jpg10170

[0078] In step B22, according to the following formula, I d2 Perform forward traversal until (ToX, ToY) > 0 or either ToX or ToY exceeds the boundary of the target depth image. JPEG2025521360000020.jpg31170

[0079] Here, [i] in Dirs[i][0] and Dirs[i][1] indicates the preset path to be traversed. [0] indicates that Dirs[i][0] takes the value on the left side of the preset path coordinate value, and [1] indicates that Dirs[i][1] takes the value on the right side of the preset path coordinate value. For example, if the preset path is Dirs = (-2, 3), and traversing in the positive direction with (-2, 3) as the preset direction (as shown in Figure 5), JPEG2025521360000021.jpg9170

[0080] In step B23, determine whether FromX, FromY, ToX, and ToY exceed the boundary of the target depth image. If any one of them exceeds the boundary, the following formula is satisfied. JPEG2025521360000022.jpg18170

[0081] If none of FromX, FromY, ToX, and ToY exceeds the boundary of the aforementioned target depth image, the following formula is satisfied. JPEG2025521360000023.jpg25170 However, FLOAT_MAX is the maximum floating-point value.

[0082] In step B24, after traversing all 16 preset paths, a set of FromX, FromY, ToX, and ToY with the minimum value of FltDis is determined, and (FromX, FromY) is set as the coordinates of the first target pixel point in the target depth image, and (ToX, ToY) is set as the coordinates of the second target pixel point in the target depth image.

[0083] Note that in steps B21 and B22, FromX, FromY, ToX, and ToY for traversing the preset paths are determined, but there is no priority in the execution of steps B21 and B22. Step B21 may be executed first, or step B22 may be executed first, and specifically, it is not limited.

[0084] In step B3, based on the coordinates of the first target pixel point and the coordinates of the second target pixel point, the aforementioned hole filling value is determined.

[0085] In this embodiment, if the depth values of the first target pixel point and the second target pixel point are JPEG2025521360000024.jpg33170

[0086] Here, I d2 (FromX, FromY) is the depth value of the first target pixel point, and I d2 (ToX, ToY)| is the depth value of the second target pixel point.

[0087] The aforementioned hole filling value is obtained by the following formula. JPEG2025521360000025.jpg19170

[0088] Here, I c1 is the first color image, (x, y) are the coordinates of the target pixel point, and I c1 (x, y) is the hole filling value of the target pixel point, and I c1(FillX, FillY) is the pixel value of the pixel point whose coordinates in the first color image are (FillX, FillY). The formula means determining the pixel value of the pixel point whose coordinates in the first color image are (FillX, FillY) as the hole filling value.

[0089] Fill the hole of the first color image by filling the hole filling value into the target pixel point to obtain the target color image without holes.

[0090] In step S05, the initial color image and the target color image are interleaved to generate a target 3D image.

[0091] In this embodiment, the above-mentioned 3D image generation device generates the above-mentioned target 3D image by interleaving and displaying the above-mentioned initial color image and the target color image without holes.

[0092] Compared with the prior art, the 3D image generation method provided by the present application separates the target 2D image to obtain the initial color image and the initial depth image, and processes the initial color image and the initial depth image in association to obtain the first color image and the target depth image respectively. Next, the target pixel points existing as holes in the first color image are determined by the reference pixel points in the above-mentioned target depth image. The hole filling value of the target pixel point is determined based on the reference pixel point to complete the filling of the holes in the first color image, obtain the target color image without holes, and interleave the initial color image and the initial depth image to generate the target 3D image without holes.

[0093] The above is the description of the present application from the perspective of the 3D image generation method. Next, the present application will be described from the perspective of the 3D image generation device.

[0094] Referring to FIG. 6. FIG. 6 is a diagram showing the hardware configuration of the 3D display image provided by the embodiment of the present application. The above-mentioned 3D image generation device 600 includes the following modules.

[0095] The separation unit 601 separates the target 2D image to obtain an initial color image and an initial depth image.

[0096] The processing unit 602 processes the aforementioned initial color image and the aforementioned initial depth image in association with each other to obtain a first color image and a target depth image respectively. The target depth image includes a plurality of reference pixel points.

[0097] The first determination unit 603 determines a target pixel point based on the depth value of the aforementioned reference pixel point. The target pixel point is a pixel point that exists as a hole in the first color image.

[0098] The second determination unit 604 determines a hole filling value for the target pixel point based on the aforementioned reference pixel point, performs filling, and generates a target color image. The aforementioned target color image is obtained after the holes in the first image are filled.

[0099] The generation unit 605 interleaves the initial color image and the target color image to generate a target 3D image.

[0100] Optionally, the second determination unit 604 further performs the following operations. Set a preset path traversed by the reference pixel point. The preset path is arranged in the target depth image. Based on the coordinates of the reference pixel point, traverse the preset path to determine a first target pixel point and a second target pixel point that meet preset conditions. Determine the hole filling value based on the coordinates of the first target pixel point and the coordinates of the second target pixel point.

[0101] Optionally, if the depth value of the first target pixel point is smaller than the depth value of the second target pixel point, the aforementioned second determination unit 604 further determines the pixel value of the pixel point having the same coordinates as the first target pixel point in the first color image as the hole filling value, and if the depth value of the first target pixel point is greater than or equal to the depth value of the second target pixel point, determines the pixel value of the pixel point having the same coordinates as the second target pixel in the first color image as the hole filling value.

[0102] Optionally, the aforementioned first determination unit 603 further d2 If I(x,y)≤0 is satisfied, determines the pixel point with coordinates (x,y) in the first color image as the target pixel point existing as a hole. Here, I d2 is the target depth image, and I d2 (x,y) is the depth value of the reference pixel point, and the reference pixel point is the pixel point with coordinates (x,y) in the target depth image.

[0103] Optionally, the processing unit 602 further executes the following operations. Determines an initial point cloud corresponding to a first depth image which is an image obtained by remapping the initial depth image. Adjusts the coordinates of each point in the initial point cloud by presetting a relative displacement to obtain a target point cloud. Processes the coordinates of each point in the target point cloud to obtain a reference image which is a depth image adjusted to the size of the initial depth image. Based on the depth values of the pixel points in the reference image, processes the pixel points in the initial color image and the pixel points in the initial depth image to obtain the first color image and the target depth image.

[0104] Optionally, the processing unit 602 is further used to obtain the aforementioned reference image by the following formula. JPEG2025521360000026.jpg20170 Here, Z is the reference image, Z(x,y) is the depth value of the pixel point with coordinates (x,y) in the reference image, Z(IP.x + 1, IP.y + 1) is the depth value of the pixel point with coordinates (IP.x + 1, IP.y + 1) in the reference image, FltErr = A - w / (2*z0), and min is the smaller value between Z(IP.x + 1, IP.y + 1) and FltErr assigned to Z(x,y). JPEG2025521360000027.jpg36170 Here, A is the initial depth value of the reference image, w is the width of the first depth image, (x0, y0, z0) is the coordinate of any point in the target point cloud, IP.x is the x value of the point IP coordinate, and IP.y is the y value of the point IP coordinate.

[0105] Optionally, the aforementioned processing unit 602 may further Process the aforementioned initial color image according to the following formula to obtain the aforementioned first color image. JPEG2025521360000028.jpg20170 Here, I c is the aforementioned initial color image, I c1 is the aforementioned first color image, I c (x,y) is the pixel value of the pixel point with coordinates (x,y) in the aforementioned initial color image, Z(x,y) is the depth value of the pixel point with coordinates (x,y) in the aforementioned reference image, I c1 (IP.x, IP.y) is the pixel value of the pixel point with coordinates (IP.x, IP.y) in the aforementioned first color image.

[0106] Process the aforementioned initial depth image according to the following formula to obtain the aforementioned target depth image. JPEG2025521360000029.jpg20170

[0107] Here, I d is the aforementioned initial depth image, I d2 is the aforementioned target depth image, I d (x,y) is the depth value of the pixel point with coordinates (x,y) in the aforementioned initial depth image, Id2 (IP.x, IP.y) is the depth value of the pixel point with coordinates (IP.x, IP.y) in the target depth image.

[0108] Optionally, the processing unit 602 further determines the preset depth of field of the aforementioned target 3D image. For the target 3D image, the preset depth of field on the x-axis is nx0 to nx1, and the preset depth of field on the y-axis is ny0 to ny1. The ratio between the preset depth of field and the depth range of the initial depth image is determined by the following formula. JPEG2025521360000030.jpg26170Here, DepthRateX is the ratio between the preset depth of field on the x-axis and the depth range of the initial depth image, DepthRateY is the ratio between the preset depth of field on the y-axis and the depth range of the initial depth image, Dmax is the maximum depth value of the initial depth image, and Dmin is the minimum depth value of the initial depth image.

[0109] Pos_x and Pos_y are obtained by the following formula. JPEG2025521360000031.jpg30170

[0110] The first color image and the target depth image are obtained by the following formula. JPEG2025521360000032.jpg25170

[0111] However, I c is the aforementioned initial color image, I c1 is the aforementioned first color image, I c (x, y) is the pixel value of the pixel point with coordinates (x, y) in the aforementioned initial color image, I c1 (Pos_x, Pos_y) is the pixel value of the pixel point with coordinates (Pos_x, Pos_y) in the aforementioned first color image, I d is the aforementioned initial depth image, I d2 is the aforementioned target depth image, I d(x, y) is the depth value of the pixel point whose coordinates in the aforementioned initial depth image are (x, y), and I d2 (Pos_x, Pos_y) is the depth value of the pixel point whose coordinates in the aforementioned target depth image are (Pos_x, Pos_y).

[0112] Figure 7 is a schematic structural diagram of the server of the present invention. The server 700 according to this embodiment includes at least one processor 701, at least one network interface 704 or other user interface 703, a memory 705, and at least one communication bus 702. Optionally, the server 700 includes a user interface 703 including a display, a keyboard or a click device. The memory 705 may include high-speed RAM memory, or may include non-volatile memory, for example, at least one disk memory. The memory 705 stores execution instructions. When the server 700 operates, the processor 701 communicates with the memory 705. The processor 701 calls the instructions stored in the memory 705 to execute the above 3D image generation method. The operating system 706 includes various programs and realizes various basic operations and tasks according to the hardware.

[0113] The server provided by the embodiment of the present application can execute the 3D image generation method described in the above embodiment. Its implementation principle and technical effect are similar to the above description, and will not be described in detail here.

[0114] The embodiment of the present application further provides a computer-readable medium including computer execution instructions. The computer execution instructions can cause the server to realize the 3D image generation method described in the above embodiment. The implementation principle and technical effect of the computer-readable medium are similar to the above description, and will not be described in detail here.

[0115] A person skilled in the art should understand that all or some of the steps for implementing the above-mentioned various methods can be achieved by hardware related to program instructions. The aforementioned program may be stored in a computer-readable storage medium. When this program is executed, it executes steps including the steps of each of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program codes, such as ROM, RAM, magnetic disks, and optical disks.

[0116] The above-mentioned content is only a part of the embodiments of the present application, and does not limit the scope of the claims of the present application. Any equivalent structure or equivalent flow conversion performed using the content of the specification and drawings of the present application, or any application directly or indirectly in other related technical fields, is included in the scope of patent protection of the present application.

Claims

1. Separating a target 2D image to obtain an initial color image and an initial depth image; By associating and processing the initial color image and the initial depth image, obtaining a first color image and a target depth image including a plurality of reference pixel points in association with each other; Determining a target pixel point, which is a pixel point existing as a hole in the first color image, based on the depth value of the reference pixel point; Determining a hole filling value for the target pixel point based on the reference pixel point, performing filling, and generating a target color image; Interleaving the initial color image and the target color image to generate a target 3D image. A 3D image generation method comprising the steps.

2. The step of determining a hole filling value for the target pixel point based on the reference pixel point includes: Setting a preset path set in the target depth image traversed by the reference pixel point; Based on the coordinates of the reference pixel point, traversing the preset path to determine a first target pixel point and a second target pixel point that meet preset conditions; Determining the hole filling value based on the coordinates of the first target pixel point and the coordinates of the second target pixel point. The 3D image generation method according to claim 1.

3. The step of traversing the preset path based on the coordinates of the reference pixel point to determine a first target pixel point and a second target pixel point that meet preset conditions includes: If the depth value of the first target pixel point is smaller than the depth value of the second target pixel point, determining the pixel value of the pixel point having the same coordinates as the first target pixel point in the first color image as the hole filling value; If the depth value of the first target pixel point is greater than or equal to the depth value of the second target pixel point, determining the pixel value of the pixel point having the same coordinates as the second target pixel in the first color image as the hole filling value. The 3D image generation method according to claim 2.

4. The step of determining a target pixel point based on the depth value of the reference pixel point includes: I ｄ２ If (x, y) ≤ 0 is satisfied, a pixel point whose coordinates in the first color image are (x, y) is determined as the target pixel point existing as a hole, Here, I ｄ２ is the target depth image, and I ｄ２ (x, y) is the depth value of the reference pixel point, and the reference pixel point is the pixel point with coordinates (x, y) in the target depth image. The 3D image generation method according to claim 1.

5. The step of obtaining a first color image and a target depth image including a plurality of reference pixel points in association with each other by associating and processing the initial color image and the initial depth image includes: Determining an initial point cloud corresponding to a first depth image, which is an image obtained by remapping the initial depth image; Adjusting the coordinates of each point in the initial point cloud by presetting a relative displacement to obtain a target point cloud; Processing the coordinates of each point in the target point cloud to obtain a reference image, which is a depth image adjusted to the size of the initial depth image; Processing the pixel points in the initial color image and the pixel points in the initial depth image based on the depth values of the pixel points in the reference image to obtain the first color image and the target depth image, the 3D image generation method according to claim 1.

6. The step of processing the coordinates of each point in the target point cloud to obtain a reference image includes the following: Obtaining the reference image by the following formula: where Z is the reference image, Z(x, y) is the depth value of the pixel point with coordinates (x, y) in the reference image, Z(IP.x + 1, IP.y + 1) is the depth value of the pixel point with coordinates (IP.x + 1, IP.y + 1) in the reference image, FltErr = A - w / (2*z0), and min assigns the smaller value of both Z(IP.x + 1, IP.y + 1) and FltErr to Z(x, y); where A is the initial depth value of the reference image, w is the width of the first depth image, (x0, y0, z0) is the coordinates of any point in the target point cloud, IP.x is the x value of the coordinates of point IP, and IP.y is the y value of the coordinates of point IP, the 3D image generation method according to claim 5.

7. The step of processing the pixel points in the initial color image and the pixel points in the initial depth image based on the depth values of the pixel points in the reference image to obtain the first color image and the target depth image includes the following: Obtaining the first color image by processing the initial color image by the following formula; Here, I ｃ is the initial color image, and I ｃ１ is the first color image. I ｃ (x, y) is the pixel value of the pixel point with coordinates (x, y) in the initial color image, and Z(x, y) is the depth value of the pixel point with coordinates (x, y) in the reference image. I ｃ１ (IP.x, IP.y) is the pixel value of the pixel point with coordinates (IP.x, IP.y) in the first color image. Obtaining the target depth image by processing the initial depth image by the following formula; Here, I ｄ is the initial depth image, and I ｄ２ is the target depth image, and I ｄ (x, y) is the depth value of the pixel point with coordinates (x, y) in the initial depth image, and I ｄ２ (IP.x, IP.y) is the depth value of the pixel point with coordinates (IP.x, IP.y) in the target depth image. The 3D image generation method according to claim 6.

8. The step of respectively obtaining the first color image and the target depth image by associating and processing the initial color image and the initial depth image includes: Determining the preset depth of field of the target 3D image; Taking the preset depth of field of the target 3D image on the x-axis as nx0 to nx1 and the preset depth of field on the y-axis as ny0 to ny1; Obtaining the ratio of the preset depth of field to the depth range of the initial depth image by the following formula: Here, DepthRateX is the ratio of the preset depth of field on the x-axis to the depth range of the initial depth image, DepthRateY is the ratio of the preset depth of field on the y-axis to the depth range of the initial depth image, Dmax is the maximum depth value of the initial depth image, Dmin is the minimum depth value of the initial depth image, Pos_x and Pos_y are obtained by the following formula, The first color image and the target depth image are obtained by the following formula, Here, I ｃ is the initial color image, and I ｃ１ is the first color image, and I ｃ (x, y) is the image value of the image point with coordinates (x, y) in the initial color image, and I ｃ１ (Pos_x, Pos_y) is the pixel value of the pixel point with coordinates (Pos_x, Pos_y) in the first color image, and I ｄ is the initial depth image, and I ｄ２ is the target depth image, and I ｄ (x, y) is the depth value of the image point with coordinates (x, y) in the initial depth image, and I ｄ２ (Pos_x, Pos_y) is the depth value of the pixel point with coordinates (Pos_x, Pos_y) in the target depth image. The 3D image generation method according to any one of claims 1 to 4.

9. A 3D image generation device, A separation unit that separates a target 2D image to obtain an initial color image and an initial depth image, A processing unit that associates and processes the initial color image and the initial depth image to respectively obtain and associate a first color image and a target depth image including a plurality of reference pixel points, A first determination unit that determines a target pixel point, which is a pixel point existing as a hole in the first color image, based on the depth value of the reference pixel point, A second determination unit that determines a hole filling value of the target pixel point based on the reference pixel point and performs filling to generate a target color image, A 3D image generation device comprising a generation unit that interleaves the initial color image and the target color image to generate a target 3D image.

10. A computer device, Comprising at least one processor, a memory, and a transceiver that are connected to each other, Here, the memory is used to store program code, and the processor calls the program code in the memory to execute the 3D image generation method according to any one of claims 1 to 4.

Images