3D image generation method, device, and computer device

Abstract

A 3D image generation method, comprising: separating a target 2D image to obtain an initial color image and an initial depth image; obtaining a first color image and a target depth image including a plurality of reference pixel points by associating and processing the initial color image and the initial depth image; determining a target pixel point that is a pixel point existing as a hole in the first color image based on the depth value of the reference pixel point; determining a hole filling value of the target pixel point based on the reference pixel point and performing filling to generate a target color image; and interleaving the initial color image and the target color image to generate a target 3D image. The present invention further provides a 3D image generation device and a computer device.

Description

[Technical Field]

[0001] The present application relates to the field of 3D display technology, and in particular to a 3D image generation method, apparatus and computer device. [Background technology]

[0002] An RGBD image contains two images, one is a normal RGB three-channel color image, and the other is a depth image, where RGB contains graphic color information and depth image contains depth information. The three-channel RGB color image is rendered from different angles to generate a new color image, and the generated color image is then rendered at different angles to create a three-dimensional model of the model texture.

[0003] In the prior art of 3D image generation methods, when a color image is generated by rendering from a certain angle, there are some hollow areas in the new color image that is generated by rendering, so it is necessary to provide a 3D image generation method that can fill in the hollow areas. Summary of the Invention

[0004] To solve the problem in the prior art that cavities exist in a color image after a new color image is generated by rendering, the present application provides a 3D image generation method, device, and computer device capable of filling cavities, in which cavities in a color image generated by rendering are filled, and a 3D image is generated using the color image without cavities.

[0005] A 3D image generating method according to a first aspect of the present invention includes: Separating the target 2D image to obtain an initial color image and an initial depth image; acquiring a first color image and a target depth image including a plurality of reference pixel points in association with each other by processing the initial color image and the initial depth image in association with each other; determining a target pixel point, which is a pixel point that exists as a cavity in the first color image, based on the depth value of the reference pixel point; determining a void filling value of the target pixel point based on the reference pixel point, and performing filling to generate a target color image; and interleaving the initial color image and the target color image to generate the target 3D image.

[0006] A 3D image generating device according to a second aspect of the present invention comprises: a separation unit for separating the target 2D image to obtain an initial color image and an initial depth image; a processing unit that processes the initial color image and the initial depth image in association with each other to obtain a first color image and a target depth image including a plurality of reference pixel points in association with each other; a first determination unit for determining a target pixel point, which is a pixel point existing as a cavity in the first color image, based on the depth value of the reference pixel point; a second determination unit for determining a void filling value of the target pixel point according to the reference pixel point, and performing filling to generate a target color image; a generation unit for interleaving the initial color image and the target color image to generate a target 3D image.

[0007] A third aspect of the present invention provides a computer device comprising at least one processor, a memory, and a transceiver connected to each other, wherein the memory is used to store program code, and the processor invokes the program code in the memory to perform the 3D image generation step described in the first aspect.

[0008] Compared with the prior art, the 3D image generating method and device provided by this application separate a target 2D image to obtain an initial color image and an initial depth image, and then associate and process the initial color image and the initial depth image to obtain a first color image and a target depth image in correspondence with each other. A target pixel point, which is a pixel point that exists as a cavity in the first color image, is determined based on a reference pixel point in the target depth image. Then, a cavity filling value of the target pixel point is determined based on the reference pixel point to complete the cavity filling in the first color image and obtain a cavity-free target color image. The initial color image and the initial depth image are then interleaved to generate a cavity-free target 3D image. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a flowchart of a 3D image generation method provided by an embodiment of the present invention. [Figure 2] 4 is a flowchart of a method for determining a first color image and a target depth image provided by an embodiment of the present invention; [Figure 3] 1 is a schematic diagram illustrating the relative displacement of cameras when rendering an initial color image provided by an embodiment of the present invention; [Figure 4] 2 is a schematic diagram of a first reference pixel point and four pairs of diagonal pixel points of a reference image provided by an embodiment of the present invention; FIG. [Figure 5] FIG. 2 is a schematic diagram of a preset path provided by an embodiment of the present application. [Figure 6] FIG. 2 is a schematic diagram of a hardware configuration of a 3D display image provided by an embodiment of the present invention. [Figure 7] FIG. 2 is a configuration diagram of a server according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following clearly and completely describes the technical aspects of the embodiments of the present application, but it is clear that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments, and all other embodiments that a person skilled in the art can obtain based on the embodiments of the present application without performing any creative work fall within the scope of protection of the present application.

[0011] The present invention provides a 3D image generation method. The 3D image generation method provided by the present application will be described below from the perspective of a 3D image generation device. The 3D image generation device may be a terminal device such as a mobile phone or a tablet, or may be another device such as a server.

[0012] 1 is a flowchart of a 3D image generating method provided by an embodiment of the present invention. As shown in FIG. 1, the 3D image generating method includes the following steps:

[0013] In step S01, the target 2D image is separated to obtain an initial color image and an initial depth image.

[0014] In this embodiment, the 3D image generation device separates the target 2D image of the 3D image to be generated to obtain the initial color image and the initial depth image. The target 2D image is an RGBD image and includes two images. One is a normal RGB three-channel color image containing color information for graphics, and the other is a depth image containing depth information. The initial color image is an RGB three-channel color image and includes multiple pixel points. Each pixel point is represented by coordinate values ​​(x, y) and has a pixel value. The pixel value is RGB color information. The initial depth image is a depth image and includes multiple pixel points. The pixel points of the initial depth image have coordinates corresponding to the target depth image. Each pixel point of the initial depth image has a depth value representing depth information.

[0015] In step S02, the initial color image and the initial depth image are processed in association with each other to obtain a first color image and a target depth image in association with each other. The target depth image includes a plurality of reference pixel points.

[0016] In this embodiment, the 3D image generating device processes the initial color image and the initial depth image in association with each other to obtain the first color image and the target depth image. The target depth image includes a plurality of reference pixel points, and the reference pixel points are any pixel points of the target depth image. The two related processing methods are described below.

[0017] Next, a first related processing method will be described with reference to Fig. 2. Fig. 2 is a flowchart of a method for determining a first color image and a target depth image provided by an embodiment of the present invention. The flowchart includes the following steps:

[0018] In step S021, an initial point cloud corresponding to the first depth image is determined.

[0019] In this embodiment, after obtaining the initial depth image, the initial depth image is remapped to the first depth image according to the following formula: JPEG0007801505000001.jpg25170

[0020] Here, I d is the initial depth image, and I d1 is the first depth image, and I d (z) is the depth value of any pixel point in the initial depth image, and Dmax is the maximum depth value among all pixel points in the initial depth image. MaxDepth is an empirical value and can take a value of 100, and of course can take other values ​​such as 80, 90, 110, etc., and is not specifically limited as long as it does not exceed the maximum floating-point number. The coordinates of each pixel point in the remapped first depth image correspond one-to-one to the coordinates of each pixel point in the initial depth image. d1 (z) is the coordinate I in the first depth image d The depth value of the pixel point that is the same as the (z) coordinate.

[0021] After obtaining the first depth image, said first depth image is transformed into said initial point cloud by the following equation: JPEG0007801505000002.jpg19170

[0022] Here, 0 ≤ x < w, 0 ≤ y < h, P0 is the initial point cloud, P0(x, y, z) is the coordinates of any one point in the initial point cloud, w is the width of the first depth image, h is the height of the first depth image, and I d1 (x, y) is the depth value of the pixel point with coordinates (x, y) in the first depth image.

[0023] In step S022, by presetting the relative displacement, the coordinates of each point in the initial point cloud are adjusted to obtain a target point cloud.

[0024] In this embodiment, in accordance with FIG. 3, the preset relative displacement is described. FIG. 3 is a schematic diagram of the relative displacement of the camera during the rendering of the initial color image provided by the embodiment of the present application. When rendering the initial color image, the position of the camera 301 is C0. It is defined that the space where the camera 301 is located is a three-dimensional space defined by the x-axis, y-axis, and z-axis, and the coordinates of C0 are set as (0, 0, z). By changing the position of the camera 301, different rendering images are obtained, and the different rendering images corresponding to when the camera 301 is at different positions are interleaved and displayed to generate a 3D display image. In order to make the sizes of different rendering images consistent, the camera 301 does not change in the z-axis. The changed position of the camera 301 is C1. The coordinates of C1 are (nx, ny, z). The change in the position of the camera 301 is the aforementioned preset relative displacement. The aforementioned preset relative displacement of the camera 301 is determined by the following formula. JPEG0007801505000003.jpg23170

[0025] Here, D is the preset relative displacement of the camera 301.

[0026] Furthermore, a preset relative displacement is added to the initial point cloud to obtain the target point cloud. Specifically, the target point cloud is determined by the following formula: JPEG0007801505000004.jpg23170

[0027] Here, P0(x,y,z) is the coordinate of any one of the initial point clouds, P1 is the target point cloud, and P1(x,y,z) is the coordinate of a point in the target point cloud to which P0(x,y,z) has been added with D. Since the camera position does not change on the z axis and the D value is (nx,ny,0), the z values ​​of the coordinates of the target point cloud and the initial point cloud are the same.

[0028] In step S023, the coordinates of each point in the target point cloud are processed to obtain a reference image.

[0029] In this embodiment, the 3D image generating device processes the coordinates of each point in the target point cloud according to the following formula to determine the reference image: The reference image is a depth image that matches the size of the initial depth image. The initial substitution value of the depth value of the reference image is A. JPEG0007801505000005.jpg19170

[0030] Here, Z is the reference image mentioned above. Z(x,y) is the depth value of the pixel point with coordinates (x,y) in the reference image mentioned above. Z(IP.x+1,IP.y+1) is the depth value of the pixel point with coordinates (IP.x+1,IP.y+1) in the reference image mentioned above. FltErr=Aw / (2*z0). min is the smallest value assigned to Z(x,y) in both Z(IP.x+1,IP.y+1) and FltErr.

[0031] JPEG0007801505000006.jpg32170A is the initial depth value of the aforementioned reference depth map, w is the width of the first depth image, (x0, y0, z0) are the coordinates of any one point in the aforementioned target point cloud, IP.x is the x value of the coordinate of point IP, and IP.y is the y value of the coordinate of point IP.

[0032] Here, the reference image is a depth image that matches the size of the first depth image. The width of the reference image matches the width w of the first depth image. The height of the reference image matches the height h of the first depth image. The initial depth value of the reference image is A. The value of A may be 100000.0, 90000.0, or 110000.0, and is not specifically limited as long as the value of A is greater than the depth value of the first depth image and less than the maximum floating-point number.

[0033] After the reference image is determined, the depth value of each pixel point in the reference image may be optimized by the following steps in order to optimize the target 3D image that is finally generated.

[0034] Next, the optimization of pixel points of the reference image will be described in detail with reference to Fig. 4. Fig. 4 is a schematic diagram of a first reference pixel point and four pairs of diagonal pixel points of a reference image according to an embodiment of the present invention.

[0035] A first reference pixel point is determined, and the first reference pixel point is a pixel point in the reference image whose depth value should be optimized. If the depth value of the first reference pixel point is greater than the depth values ​​of the four pairs of diagonal pixel points, the average value of the depth values ​​of the four pairs of diagonal pixel points is determined as the depth value of the first reference pixel point.

[0036] Although it is understood that the first reference pixel point may be any pixel point in the reference image described above, the optimization process will be described assuming that the coordinate value of the first reference pixel point is (x, y). The four pairs of diagonal pixel points have coordinates above, below, left, right, and two pairs of diagonal pixel points of the first reference pixel point (shown in FIG. 4). The top and bottom diagonal image points are image points with coordinates (x, y-1) and (x, y+1), respectively. The left and right diagonal image points are image points with coordinates (x-1, y) and (x+1, y), respectively. The two pairs of diagonal image points are image points with coordinates (x-1, y-1), (x+1, y+1), (x+1, y-1), and (x-1, y+1), respectively.

[0037] Next, a method for optimizing the first reference pixel point will be described with reference to Figure 4. The method for optimizing the first reference pixel point includes the following steps.

[0038] In step A1, if the depth value of the first reference pixel point satisfies Z(x,y)>Z(x-1,y) and Z(x,y)>Z(x+1,y), the following formula is satisfied: JPEG0007801505000007.jpg17170

[0039] In step A2, if the depth value of the first reference pixel point satisfies Z(x,y)>Z(x,y-1) and Z1(x,y)>Z(x,y+1), the following formula is satisfied: JPEG0007801505000008.jpg19170

[0040] In step A3, if the depth value of the first reference pixel point satisfies Z(x, y)>Z(x-1, y-1) and Z(x, y)>Z(x+1, y+1), the following formula is satisfied: JPEG0007801505000009.jpg16170

[0041] In step A4, if the depth value of the first reference pixel point satisfies Z(x, y)>Z(x+1, y-1) and Z(x, y)>Z(x-1, y+1), the following formula is satisfied: JPEG0007801505000010.jpg24170

[0042] In step A5, the average value is calculated using the following equation 11. JPEG0007801505000011.jpg18170

[0043] Here, the initial substitution values ​​of Zsum and Ztol are 0, and if the depth value of the first reference pixel point with coordinate values ​​(x, y) satisfies any of the conditions in steps A1 to A4, a new depth value is calculated and overwrites the original depth value of the first reference pixel point mentioned above.

[0044] where Z(x,y) is the depth value of the first reference pixel point in the reference image, Z(x+1,y) is the depth value of the pixel point whose coordinates are (x+1,y) in the reference image, Z(x-1,y) is the depth value of the pixel point whose coordinates are (x-1,y) in the reference image, Z(x,y-1) is the depth value of the pixel point whose coordinates are (x,y-1) in the reference image, and Z(x,y+1) is the depth value of the pixel point whose coordinates are (x,y+1) in the reference image. where Z(x+1,y+1) is the depth value of the pixel point whose coordinates are (x+1,y+1) in the reference image, Z(x-1,y-1) is the depth value of the pixel point whose coordinates are (x-1,y-1) in the reference image, Z(x-1,y+1) is the depth value of the pixel point whose coordinates are (x-1,y+1) in the reference image, and Z(x+1,y-1) is the depth value of the pixel point whose coordinates are (x+1,y-1) in the reference image.

[0045] In steps A1, A2, A3, and A4, it is determined whether the depth value of the first reference pixel point is greater than the depth values ​​of its four pairs of diagonal pixel points, and the average value is calculated in step A5, but there is no restriction on the order of priority of execution among steps A1, A2, A3, and A4. Step A1 may be executed first, or step A2 may be executed first, or step A3 may be executed first, or step A4 may be executed first, and there is no specific restriction.

[0046] In step S024, the pixel points in the initial color image and the pixel points in the initial depth image are processed according to the depth values ​​of the pixel points in the reference image to obtain the first color image and the target depth image.

[0047] In this embodiment, the first color image is calculated using the following formula: JPEG0007801505000012.jpg20170

[0048] Here, I c is the initial color image, and I c1 is the first color image, IP.x is the x value of the coordinate of the point IP, and IP.y is the y value of the coordinate of the point IP. c (x, y) is the pixel value of the pixel point with coordinates (x, y) in the initial color image, Z(x, y) is the depth value of the pixel point with coordinates (x, y) in the reference image, and I c1 (IP.x, IP.y) is the pixel value of the pixel point with coordinates (IP.x, IP.y) in the first color image.

[0049] When (Z(x,y)+1)>FltErr is satisfied, if the value of ((Z(x,y)+1)>FltErr) is set to 1, the pixel value of the pixel point with coordinates (x,y) in the initial depth image is assigned to the pixel point with coordinates (IP.x,IP.y) in the first color image.

[0050] If (Z(x,y)+1)>FltErr is not satisfied, then if the value of ((Z(x,y)+1)>FltErr) is set to 0, the pixel value of the pixel point with coordinates (IP.x,IP.y) in the first color image mentioned above is 0.

[0051] The above-mentioned target depth image is calculated by the following formula. JPEG0007801505000013.jpg19170

[0052] Here, I d is the initial depth image, and I d2 is the target depth image, and I d (x, y) is the depth value of the pixel point with coordinates (x, y) in the initial depth image, and I d2 (IP.x,IP.y) is the depth value of the pixel point with coordinates (IP.x,IP.y) in the target depth image. When the value of ((Z(x,y)+1)>FltErr) is 1, the depth value of the pixel point with coordinates (x,y) in the initial depth image is assigned to the pixel point with coordinates (IP.x,IP.y) in the target depth image, and when the value of ((Z(x,y)+1)>FltErr) is 0, the depth value of the pixel point with coordinates (IP.x,IP.y) in the target depth image is 0.

[0053] In this method, the initial color image and the initial depth image are associated and processed using the initial point cloud, the target point cloud, and the reference image, so that the pixel values ​​of pixel points in the obtained first color image are associated with the depth values ​​of pixel points in the target depth image.

[0054] The second type of related processing method will be described below. The second type of related processing method includes the following steps.

[0055] In step C1, the preset depth of field of the target 3D image is determined. The preset depth of field of the target 3D image on the x-axis is nx0 to nx1, and the preset depth of field on the y-axis is ny0 to ny1.

[0056] In this embodiment, the preset depth of field has a predetermined parallax and is in a fixed pixel unit, with the x-axis parallel to the width of the 3D image and the y-axis parallel to the height of the 3D image. For example, if the preset depth of field on the x-axis of the target 3D image is in the range of 10 to 100 pixel units, nx0 is 10 pixel units and nx1 is 100 pixel units. The preset depth of field may be 0 pixel units. For example, the target 3D image has no parallax on the y-axis. If the preset depth of field is 0 pixel units, ny0 is 0 pixel units and ny1 is 0 pixel units.

[0057] In step C2, the ratio between the preset depth of field and the depth range of the initial depth image is calculated using the following formula: JPEG0007801505000014.jpg25170

[0058] Here, DepthRateX is the ratio of the preset depth of field to the initial image depth range on the x-axis, DepthRateY is the ratio of the preset depth of field to the initial image depth range on the y-axis, Dmax is the maximum depth value of the initial depth image, and Dmin is the minimum depth value of the initial depth image.

[0059] In step C3, Pos_x and Pos_y are calculated using the following formulas. JPEG0007801505000015.jpg24170

[0060] However, I d is the initial depth image, and I d (x, y) is the depth value of the pixel point with coordinates (x, y) in the initial depth image, x is the x value of the coordinate at the pixel point with coordinates (x, y), and y is the y value of the coordinate at the pixel point with coordinates (x, y).

[0061] In step C4, the first color image and the target depth image are calculated by the following equations: JPEG0007801505000016.jpg28170 where, Ic is the initial color image, and I c1 is the first color image, and I c (x, y) is the pixel value of the image point with coordinates (x, y) in the initial color image, and I c1 (Pos_x, Pos_y) is the pixel value of the pixel point whose coordinates are (Pos_x, Pos_y) in the first color image, and I d is the initial depth image, and I d2 is the target depth image, and I d2 (Pos_x, Pos_y) is the depth value of the pixel point with coordinates (Pos_x, Pos_y) in the target depth image. The above formula means that the pixel point with coordinates (Pos_x, Pos_y) in the first color image is assigned the pixel value of the pixel point with coordinates (x, y) in the initial color image, and that the pixel point with coordinates (Pos_x, Pos_y) in the target depth image is assigned the depth value of the pixel point with coordinates (x, y) in the initial depth image.

[0062] This method associates the initial color image and the initial depth image using Pos_x and Pos_y, and associates the offset amount of the first color image relative to the initial color image with the depth value of the first depth image. The larger the depth value of the initial depth image, the larger the offset amount of a pixel point in the first depth image whose coordinates are (Pos_x, Pos_y) relative to a pixel point in the initial depth image whose coordinates are (x, y). The pixel values ​​of pixel points in the first color image obtained by this method are associated with the depth values ​​of pixel points in the target depth image. The target depth image includes a plurality of reference pixel points.

[0063] In step S03, a target pixel point is determined based on the depth value of the reference pixel point.

[0064] In this embodiment, the 3D image generating device determines a first color image and a target depth image, and then determines whether a pixel point in the first color image that has the same coordinates as the reference pixel point is a cavity based on the depth value of the reference pixel point, and determines the pixel point that exists as a cavity in the first color image as the target pixel point.

[0065] Next, a method for determining a target pixel point in the first color image will be described.

[0066] I d2 If (x, y)≦0 is satisfied, the pixel point with coordinates (x, y) in the first color image is determined as the target pixel point existing as a cavity. d2 is the target depth image mentioned above, and I d2 (x, y) is the depth value of the reference pixel point, and the reference pixel point is the pixel point with coordinates (x, y) in the target depth image. d2 When (x, y)≦0 is satisfied, the pixel value of the image point whose coordinates are (x, y) in the first color image is 0, and there is no coloring value, so the image point exists as a void.

[0067] In step S04, a void filling value for the target pixel point is determined based on the reference image point.

[0068] In this embodiment, the 3D image generating device determines the target pixel point that exists as a cavity in the first color image, then determines a cavity filling value of the target pixel point based on the coordinate value and pixel value of the reference pixel point, fills the cavity in the first color image based on the cavity filling value of the target pixel point, and generates a filled target color image.

[0069] The following describes how to determine the void filling value of the target pixel point. The method for determining the void filling value of the target pixel point includes the following steps:

[0070] In step B1, a preset path along which the reference image point traverses is set.

[0071] In this embodiment, the preset paths may be set according to the actual situation, for example, the preset paths may be 16, 6, or 5, or may not be specifically limited, and may be debugged according to the generation situation of the above-mentioned target 3D image.

[0072] Below, the traverse method will be specifically described using an example in which there are 16 preset routes.

[0073] The preset route is represented by Dirs, which presets 16 traversal search directions, where Dirs = (-1,1), (0,1), (1,1), (1,0), (-1,2), (1,2), (2,1), (2,-1), (-2,3), (-1,3), (1,3), (2,3), (3,2), (3,1), (3,-1), and (3,-2). Please also refer to FIG. 5, which is a schematic diagram of a preset route provided by an embodiment of the present application. The preset route Dirs shown in FIG. 5 satisfies Dirs = (-2,3).

[0074] In step B2, the preset path is traversed based on the coordinates of the reference pixel point to determine the first and second target pixel points that meet the preset conditions.

[0075] Next, the procedure for traversing the reference pixel point described above will be explained with reference to Figure 5. Using the reference pixel point described above as the starting point and the coordinates of the reference pixel point described above as (x, y), traversal in each preset direction is performed according to the following procedure.

[0076] In step B21, I is calculated using the following formula: d2 Traversal in the negative direction is performed until (FromX, FromY)>0 or either FromX or FromY exceeds the boundary of the target depth image. JPEG0007801505000017.jpg31170

[0077] Here, [i] in Dirs[i][0] and Dirs[i][1] indicates the preset route to be traversed, [0] indicates that Dirs[i][0] takes the value on the left side of the preset route coordinate value, and [1] indicates that Dirs[i][1] takes the value on the right side of the preset route coordinate value. For example, if the preset route is Dirs=(-2,3) and (-2,3) is used as the preset direction and followed in the negative direction (as shown in Figure 5), JPEG0007801505000018.jpg10170

[0078] In step B22, I d2 A forward traversal is performed until either (ToX,ToY)>0 or either ToX or ToY exceeds the boundary of the target depth image. JPEG0007801505000019.jpg31170

[0079] Here, [i] in Dirs[i][0] and Dirs[i][1] indicates the preset route to be traversed, [0] indicates that Dirs[i][0] takes the value on the left side of the preset route coordinate value, and [1] indicates that Dirs[i][1] takes the value on the right side of the preset route coordinate value. For example, if the preset route is Dirs=(-2, 3) and (-2, 3) is used as the preset direction and followed forward (as shown in Figure 5), JPEG0007801505000020.jpg9170

[0080] In step B23, it is determined whether FromX, FromY, ToX, and ToY exceed the boundary of the target depth image. If any one of them exceeds the boundary, the following formula is satisfied. JPEG0007801505000021.jpg18170

[0081] If any of FromX, FromY, ToX, and ToY does not exceed the boundary of the target depth image, the following formula is satisfied. JPEG0007801505000022.jpg25170 where FLOAT_MAX is the maximum floating point value.

[0082] In step B24, after traversing all 16 preset paths, a set of FromX, FromY, ToX and ToY that minimizes the FltDis value is determined, and (FromX, FromY) is the coordinate of the first target pixel point in the target depth image, and (ToX, ToY) is the coordinate of the second target pixel point in the target depth image.

[0083] Steps B21 and B22 determine FromX, FromY, ToX, and ToY for traversing the preset route, but there is no priority in the execution of steps B21 and B22; either step B21 or step B22 may be executed first, and there are no specific limitations.

[0084] In step B3, the above-mentioned void filling value is determined based on the coordinates of the first target pixel point and the coordinates of the second target pixel point.

[0085] In this embodiment, if the depth values ​​of the first target pixel point and the second target pixel point are JPEG0007801505000023.jpg33170

[0086] Here, I d2 (FromX, FromY) is the depth value of the first target pixel point, and I d2 (ToX, ToY)| is the depth value of the second target pixel point.

[0087] The aforementioned cavity filling value is calculated by the following formula: JPEG0007801505000024.jpg19170

[0088] Here, I c1 is the first color image, (x, y) is the coordinate of the target pixel point, and I c1 (x, y) is the cavity filling value of the target pixel point, and I c1(FillX, FillY) is the pixel value of the pixel point whose coordinates are (FillX, FillY) in the first color image. The formula means that the pixel value of the pixel point whose coordinates are (FillX, FillY) in the first color image is determined as the void filling value.

[0089] The void filling value is filled into the target pixel point to fill the void in the first color image, and obtain the target color image without voids.

[0090] In step S05, the initial color image and the target color image are interleaved to generate a target 3D image.

[0091] In this embodiment, the 3D image generating device generates the target 3D image by interleaving and displaying the initial color image and the target color image without a cavity.

[0092] Compared with the prior art, the 3D image generation method provided in this application separates a target 2D image to obtain an initial color image and an initial depth image, and then associates and processes the initial color image and the initial depth image to obtain a first color image and a target depth image, respectively. Next, a target pixel point that exists as a cavity in the first color image is determined based on a reference pixel point in the target depth image. A cavity filling value for the target pixel point is determined based on the reference pixel point to complete the cavity filling in the first color image, and a cavity-free target color image is obtained. The initial color image and the initial depth image are interleaved to generate a cavity-free target 3D image.

[0093] The present application has been described above from the perspective of a 3D image generation method. Hereinafter, the present application will be described from the perspective of a 3D image generation device.

[0094] Please refer to Fig. 6. Fig. 6 is a diagram showing the hardware configuration of a 3D display image provided by an embodiment of the present application. The aforementioned 3D image generating device 600 includes the following modules:

[0095] The separation unit 601 separates the target 2D image to obtain an initial color image and an initial depth image.

[0096] The processing unit 602 associates and processes the initial color image and the initial depth image to obtain a first color image and a target depth image, respectively, which includes a plurality of reference pixel points.

[0097] The first determination unit 603 determines a target pixel point based on the depth value of the reference pixel point, where the target pixel point is a pixel point that exists as a cavity in the first color image.

[0098] The second determining unit 604 determines a void filling value of the target pixel point based on the reference pixel point, performs filling, and generates a target color image obtained after the void in the first image is filled.

[0099] The generation unit 605 interleaves the initial color image and the target color image to generate the target 3D image.

[0100] Optionally, the second determining unit 604 further performs the following operations: A preset path is set for the reference pixel point to traverse, and the preset path is positioned in the target depth image. Based on the coordinates of the reference pixel point, the preset path is traversed to determine a first target pixel point and a second target pixel point that meet a preset condition. The void filling value is determined based on the coordinates of the first target pixel point and the coordinates of the second target pixel point.

[0101] Optionally, the aforementioned second determination unit 604 further determines the pixel value of a pixel point in the first color image that has the same coordinates as the first target pixel point as the void filling value if the depth value of the first target pixel point is smaller than the depth value of the second target pixel point; and determines the pixel value of a pixel point in the first color image that has the same coordinates as the second target pixel point as the void filling value if the depth value of the first target pixel point is greater than or equal to the depth value of the second target pixel point.

[0102] Optionally, the aforementioned first determining unit 603 further comprises: d2 If (x, y)≦0 is satisfied, the pixel point with coordinates (x, y) in the first color image is determined to be the target pixel point that exists as a cavity. d2 is the target depth image, and I d2 (x, y) is the depth value of the reference pixel point, which is the pixel point with coordinates (x, y) in the target depth image.

[0103] Optionally, processing unit 602 further performs the following operations: An initial point cloud corresponding to a first depth image is determined, the first depth image being a remapped image. The coordinates of each point in the initial point cloud are adjusted by presetting a relative displacement to obtain a target point cloud. The coordinates of each point in the target point cloud are processed to obtain a reference image, which is a depth image scaled to the size of the initial depth image. According to the depth values ​​of pixel points in the reference image, the pixel points in the initial color image and the pixel points in the initial depth image are processed to obtain the first color image and the target depth image.

[0104] Optionally, the processing unit 602 is further used to determine the reference image as described above by the following formula: JPEG0007801505000025.jpg20170Here, Z is the reference image, Z(x,y) is the depth value of the pixel point in the reference image whose coordinates are (x,y), Z(IP.x+1,IP.y+1) is the depth value of the pixel point in the reference image whose coordinates are (IP.x+1,IP.y+1), FltErr = Aw / (2*z0), and min is the smaller value of both Z(IP.x+1,IP.y+1) and FltErr assigned to Z(x,y). JPEG0007801505000026.jpg36170Here, A is the initial depth value of the reference image, w is the width of the first depth image, (x0, y0, z0) is the coordinate of any one point in the target point cloud, IP.x is the x value of the coordinate of point IP, and IP.y is the y value of the coordinate of point IP.

[0105] Optionally, the processing unit 602 further comprises: The initial color image is processed to obtain the first color image according to the following formula: JPEG0007801505000027.jpg20170Here, I c is the initial color image, and I c1 is the first color image, and I c (x, y) is the pixel value of the pixel point with coordinates (x, y) in the initial color image, Z(x, y) is the depth value of the pixel point with coordinates (x, y) in the reference image, and I c1 (IP.x, IP.y) is the pixel value of the pixel point with coordinates (IP.x, IP.y) in the first color image.

[0106] The initial depth image is processed to obtain the target depth image according to the following equation: JPEG0007801505000028.jpg20170

[0107] Here, I d is the initial depth image, and I d2 is the target depth image, and I d (x, y) is the depth value of the pixel point with coordinates (x, y) in the initial depth image, and Id2 (IP.x, IP.y) is the depth value of the pixel point with coordinates (IP.x, IP.y) in the target depth image.

[0108] Optionally, the processing unit 602 further: Determine a preset depth of field of the target 3D image, where the preset depth of field of the target 3D image on the x-axis is nx0 to nx1; The preset depth of field on the y-axis is ny0 to ny1. The following formula determines the ratio between the preset depth of field and the depth range of the initial depth image: JPEG0007801505000029.jpg26170Here, DepthRateX is the ratio between the preset depth of field on the x-axis and the depth range of the initial depth image, DepthRateY is the ratio between the preset depth of field on the y-axis and the depth range of the initial depth image, Dmax is the maximum depth value of the initial depth image, and Dmin is the minimum depth value of the initial depth image.

[0109] Pos_x and Pos_y are calculated using the following formula: JPEG0007801505000030.jpg30170

[0110] The first color image and the target depth image are calculated using the following formulas: JPEG0007801505000031.jpg25170

[0111] However, I c is the initial color image mentioned above, and I c1 is the first color image mentioned above, and I c (x, y) is the pixel value of the pixel point with coordinates (x, y) in the initial color image, and I c1 (Pos_x, Pos_y) is the pixel value of the pixel point whose coordinates are (Pos_x, Pos_y) in the first color image, and I d is the initial depth image mentioned above, and I d2 is the target depth image mentioned above, and I d(x, y) is the depth value of the pixel point with coordinates (x, y) in the initial depth image, and I d2 (Pos_x, Pos_y) is the depth value of the pixel point whose coordinates are (Pos_x, Pos_y) in the target depth image.

[0112] FIG. 7 is a schematic diagram of the structure of a server according to the present invention. The server 700 according to this embodiment includes at least one processor 701, at least one network interface 704 or other user interface 703, memory 705, and at least one communication bus 702. Optionally, the server 700 includes the user interface 703, which may include a display, a keyboard, or a click device. The memory 705 may include a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The memory 705 stores execution instructions. When the server 700 is running, the processor 701 and the memory 705 communicate with each other. The processor 701 invokes the instructions stored in the memory 705 to execute the 3D image generation method. The operation system 706 includes various programs and performs various tasks according to the underlying operations and hardware.

[0113] The server provided in the embodiments of the present application can execute the 3D image generation method described in the above-mentioned embodiments, and its implementation principle and technical effect are similar to those described above, so they will not be further described here.

[0114] The embodiments of the present application further provide a computer-readable medium containing computer-executable instructions, which can cause a server to realize the 3D image generating method described in the above-mentioned embodiments. The implementation principles and technical effects of the computer-readable medium are similar to those described above, and will not be further described here.

[0115] Those skilled in the art should understand that all or part of the steps for implementing each of the above-mentioned methods can be achieved by hardware associated with program instructions. The above-mentioned program may be stored in a computer-readable storage medium. When the program is executed, it executes steps including the steps of each of the above-mentioned method embodiments. The above-mentioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disks, and optical disks.

[0116] The above contents are only some examples of the present application, and do not limit the scope of the claims of the present application. Any equivalent structure or equivalent flow conversion made by using the contents of the specification and drawings of the present application, or those used directly or indirectly in other related technical fields, are all included in the patent protection scope of the present application.

Claims

1. Separating the target 2D image to obtain an initial color image and an initial depth image; acquiring a first color image and a target depth image including a plurality of reference pixel points in association with each other by processing the initial color image and the initial depth image in association with each other; determining a target pixel point, which is a pixel point that exists as a cavity in the first color image, based on the depth value of the reference pixel point; determining a void filling value for the target pixel point based on the reference pixel point, and performing filling to generate a target color image; and interleaving the initial color image and the target color image to generate a target 3D image.

2. determining a void filling value for the target pixel point based on the reference pixel point, setting a preset path set in the target depth image that the reference pixel point traverses; traversing the preset path based on the coordinates of the reference pixel point to determine a first target pixel point and a second target pixel point that meet a preset condition; The 3D image generating method of claim 1 , further comprising: determining the void filling value based on the coordinates of the first target pixel point and the coordinates of the second target pixel point.

3. The step of determining a first target pixel point and a second target pixel point that meet a preset condition by traversing the preset path based on the coordinates of the reference pixel point includes: If the depth value of the first target pixel point is smaller than the depth value of the second target pixel point, determining the pixel value of a pixel point in the first color image that has the same coordinates as the first target pixel point as the void filling value; 3. The 3D image generating method according to claim 2, further comprising: if the depth value of the first target pixel point is equal to or greater than the depth value of the second target pixel point, determining the pixel value of a pixel point in the first color image that has the same coordinates as the second target pixel as the cavity filling value.

4. The step of determining a target pixel point based on a depth value of the reference pixel point includes: I ｄ２ If (x, y)≦0 is satisfied, a pixel point in the first color image whose coordinates are (x, y) is determined to be the target pixel point existing as a cavity; Here, I ｄ２ is the target depth image, and I ｄ２ The 3D image generating method according to claim 1 , wherein (x, y) is a depth value of the reference pixel point, and the reference pixel point is a pixel point in the target depth image whose coordinates are (x, y).

5. The step of processing the initial color image and the initial depth image in association with each other to obtain a first color image and a target depth image including a plurality of reference pixel points in association with each other includes: determining an initial point cloud corresponding to a first depth image, the initial depth image being a remapped image; adjusting the coordinates of each point in the initial point cloud by presetting a relative displacement to obtain a target point cloud; processing the coordinates of each point in the target point cloud to obtain a reference image, which is a depth image sized to the initial depth image; and processing pixel points in the initial color image and pixel points in the initial depth image based on depth values ​​of pixel points in the reference image to obtain the first color image and the target depth image.

6. The step of processing the coordinates of each point in the target point cloud to obtain the reference image includes: The reference image is calculated using the following formula: where Z is the reference image, Z(x, y) is the depth value of the pixel point with coordinates (x, y) in the reference image, Z(IP.x+1, IP.y+1) is the depth value of the pixel point with coordinates (IP.x+1, IP.y+1) in the reference image, FltErr=A-w / (2*z0), and min assigns the smaller value of Z(IP.x+1, IP.y+1) and FltErr to Z(x, y); 6. The 3D image generating method of claim 5, wherein A is the initial depth value of the reference image, w is the width of the first depth image, (x0, y0, z0) are the coordinates of any point in the target point cloud, IP.x is the x value of the coordinate of point IP, and IP.y is the y value of the coordinate of point IP.

7. The step of processing pixel points in the initial color image and pixel points in the initial depth image based on depth values ​​of pixel points in the reference image to obtain the first color image and the target depth image includes: obtaining the first color image by processing the initial color image according to the following equation: Here, I ｃ is the initial color image, and I ｃ１ is the first color image, and I ｃ (x, y) is the pixel value of the pixel point with coordinates (x, y) in the initial color image, Z(x, y) is the depth value of the pixel point with coordinates (x, y) in the reference image, and I ｃ１ (IP.x, IP.y) is the pixel value of the pixel point with coordinates (IP.x, IP.y) in the first color image, obtaining the target depth image by processing the initial depth image according to the following equation: Here, I ｄ is the initial depth image, and I ｄ２ is the target depth image, and I ｄ (x, y) is the depth value of the pixel point with coordinates (x, y) in the initial depth image, and I ｄ２ The 3D image generating method according to claim 6 , wherein (IP.x, IP.y) is the depth value of a pixel point with coordinates (IP.x, IP.y) in the target depth image.

8. The step of processing the initial color image and the initial depth image in association with each other to obtain the first color image and the target depth image in association with each other includes: determining a preset depth of field for the target 3D image; The preset depth of field of the target 3D image on the x-axis is defined as nx0 to nx1, and the preset depth of field on the y-axis is defined as ny0 to ny1; The ratio of the preset depth of field to the depth range of the initial depth image is calculated using the following formula: where DepthRateX is the ratio of the preset depth of field to the depth range of the initial depth image on the x-axis, DepthRateY is the ratio of the preset depth of field to the depth range of the initial depth image on the y-axis, Dmax is the maximum depth value of the initial depth image, and Dmin is the minimum depth value of the initial depth image; Pos_x and Pos_y are calculated using the following formula: The first color image and the target depth image are calculated using the following formula: Here, I ｃ is the initial color image, and I ｃ１ is the first color image, and I ｃ (x, y) is the image value of the image point with coordinates (x, y) in the initial color image, and I ｃ１ (Pos_x, Pos_y) is the pixel value of the pixel point whose coordinates are (Pos_x, Pos_y) in the first color image, and I ｄ is the initial depth image, and I ｄ２ is the target depth image, and I ｄ (x, y) is the depth value of the image point with coordinates (x, y) in the initial depth image, and I ｄ２ 5. The 3D image generating method according to claim 1, wherein (Pos_x, Pos_y) is the depth value of a pixel point whose coordinates in the target depth image are (Pos_x, Pos_y).

9. A 3D image generating device, comprising: a separation unit for separating the target 2D image to obtain an initial color image and an initial depth image; a processing unit that processes the initial color image and the initial depth image in association with each other to obtain a first color image and a target depth image including a plurality of reference pixel points in association with each other; a first determination unit for determining a target pixel point, which is a pixel point existing as a cavity in the first color image, based on the depth value of the reference pixel point; a second determination unit for determining a void filling value of the target pixel point based on the reference pixel point, and performing filling to generate a target color image; a generation unit for interleaving the initial color image and the target color image to generate a target 3D image.

10. 1. A computer device comprising: at least one processor, memory, and transceiver coupled to each other; A computer device, wherein the memory is used to store program code, and the processor calls the program code in the memory to execute the 3D image generation method according to any one of claims 1 to 4.

Images