Virtual viewpoint depth map processing method, device, apparatus and storage medium

Abstract

The present invention discloses a virtual viewpoint depth map processing method, equipment, device and storage medium. The present application forward maps a reference viewpoint depth map corresponding to a preset reference viewpoint to the position of a virtual viewpoint to obtain a virtual viewpoint depth map, uses a preset filtering method to smooth the virtual viewpoint depth map, retains the edge information of the depth map, reversely maps the smoothed virtual viewpoint depth map to the position of the preset reference viewpoint to synthesize a virtual viewpoint texture map based on different preset reference viewpoints, performs weighted fusion on the virtual viewpoint texture map, and performs hole filling processing and foreground edge filtering processing on the fused virtual viewpoint texture map, thereby improving the quality of the virtual viewpoint depth map.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a virtual viewpoint depth map processing method, equipment, device and storage medium. Background Art

[0002] Depth-based image rendering (DIBR) is an important technology in the field of virtual viewpoint synthesis. This technology uses the texture map and depth map of the reference viewpoint through three-dimensional coordinate transformation to obtain the view of any virtual viewpoint. In the process of synthesizing the virtual viewpoint texture map using DIBR technology, part of the background texture map is invisible in the reference viewpoint due to being blocked by the foreground object, but it is visible in the virtual viewpoint. In this case, there may be holes in the virtual viewpoint texture map. The depth discontinuity in the depth map is the cause of the holes. When filling the holes with depth map smoothing methods such as Gaussian filtering, mean filtering, and median filtering, the edge information of the depth map is not retained, resulting in ghosting in the edge area of ​​the synthesized texture map. Therefore, the quality of the virtual viewpoint depth map is poor.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a virtual viewpoint depth map processing method, equipment, device and storage medium, aiming to solve the technical problem that the existing depth map smoothing method cannot retain the edge information of the depth map, resulting in ghosting in the edge area of ​​the synthetic texture map and poor quality of the virtual viewpoint depth map.

[0005] To achieve the above object, the present invention provides a virtual viewpoint depth map processing method, the virtual viewpoint depth map processing method comprising:

[0006] Forward mapping the reference viewpoint depth map corresponding to the preset reference viewpoint to the position of the virtual viewpoint to obtain the virtual viewpoint depth map;

[0007] Performing bilateral filtering and smoothing processing on the virtual viewpoint depth map to obtain a smooth virtual viewpoint depth map;

[0008] According to the smoothed virtual viewpoint depth map, establishing a reverse mapping relationship between the virtual viewpoint and the reference viewpoint to synthesize a virtual viewpoint texture map based on different preset reference viewpoints;

[0009] The virtual viewpoint texture map is weightedly fused, and the fused virtual viewpoint texture map is subjected to hole filling processing and foreground edge filtering processing to obtain a target virtual viewpoint depth map.

[0010] Optionally, the step of forward mapping the reference viewpoint depth map corresponding to the preset reference viewpoint to the position of the virtual viewpoint to obtain the virtual viewpoint depth map includes:

[0011] Establishing a first spatial coordinate system according to internal parameters of a reference camera corresponding to the preset reference viewpoint, and mapping the reference viewpoint depth map to the first spatial coordinate system to obtain first spatial coordinates of the reference viewpoint depth map;

[0012] Establishing a second spatial coordinate system according to internal parameters of the virtual camera corresponding to the virtual viewpoint, and mapping the first spatial coordinates to the second spatial coordinate system to obtain second spatial coordinates of the reference viewpoint depth map;

[0013] Performing an inverse transformation on the second spatial coordinates to obtain a first mapping relationship between the second spatial coordinate system and the image coordinates of the virtual viewpoint;

[0014] According to the first mapping relationship, the reference viewpoint depth map is mapped to the position of the virtual viewpoint to obtain the virtual viewpoint depth map.

[0015] Optionally, the step of establishing a reverse mapping relationship between the virtual viewpoint and the reference viewpoint according to the smoothed virtual viewpoint depth map to synthesize a virtual viewpoint texture map based on different preset reference viewpoints includes:

[0016] Reverse mapping the smoothed virtual viewpoint depth map to obtain a second mapping relationship between the spatial coordinate system of the virtual viewpoint and the spatial coordinate system of the preset reference viewpoint;

[0017] According to the second mapping relationship, the reference viewpoint texture map corresponding to the preset reference viewpoint is mapped to the position of the virtual viewpoint to obtain the virtual viewpoint texture map.

[0018] Optionally, the step of performing weighted fusion on the virtual viewpoint texture map comprises:

[0019] Determining a first position of a reference camera corresponding to the preset reference viewpoint and a second position of a virtual camera corresponding to the virtual viewpoint;

[0020] determining a first weight according to a positional relationship between the first position and the second position;

[0021] Determining a depth value confidence of the virtual camera position, and determining a second weight according to the depth value confidence;

[0022] The virtual viewpoint texture maps based on different preset reference viewpoints are fused according to the first weight and the second weight.

[0023] Optionally, the step of determining the depth value confidence of the virtual camera position comprises:

[0024] Determine a first reference point from the virtual viewpoint depth map, and map the first reference point to a spatial coordinate system where the first position is located to obtain a second reference point;

[0025] According to the position depth value of the reference camera, the second reference point is mapped to the spatial coordinate system where the second position is located to obtain a third reference point;

[0026] The depth value confidence is determined using a first preset algorithm according to the coordinates of the first reference point and the coordinates of the third reference point.

[0027] Optionally, the step of performing hole filling processing and foreground edge filtering processing on the fused virtual viewpoint texture map includes:

[0028] Selecting a first central pixel point from the virtual viewpoint texture map and creating a first window of a preset size;

[0029] Using a second preset algorithm, filling calculation is performed on the values ​​of the pixels in the first window to perform hole filling processing;

[0030] Determining a foreground edge region of the virtual viewpoint texture image, and marking the foreground edge region;

[0031] Selecting a second central pixel point from the foreground edge area and creating a second window of a preset size;

[0032] A third preset algorithm is used to perform filtering calculation on the values ​​of the pixel points in the second window to perform filtering processing on the foreground edge area.

[0033] Optionally, the step of determining a foreground edge area of ​​the virtual viewpoint texture map and marking the foreground edge area comprises:

[0034] Selecting a reference pixel point and determining a gradient absolute value of a depth value of the reference pixel point;

[0035] If the absolute value of the gradient is greater than a preset threshold, determining that the reference pixel is an edge pixel;

[0036] A preset number of pixels are expanded around the edge pixels to determine the foreground edge area, and the foreground edge area is marked.

[0037] In addition, to achieve the above-mentioned purpose, the present invention further provides a virtual viewpoint depth map processing device, the virtual viewpoint depth map processing device comprising:

[0038] A forward mapping module, used to forward map a reference viewpoint depth map corresponding to a preset reference viewpoint to a position of a virtual viewpoint to obtain a virtual viewpoint depth map;

[0039] A post-processing module, used for performing bilateral filtering and smoothing processing on the virtual viewpoint depth map to obtain a smooth virtual viewpoint depth map;

[0040] A reverse mapping module, used to establish a reverse mapping relationship between the virtual viewpoint and the reference viewpoint according to the smoothed virtual viewpoint depth map, so as to synthesize a virtual viewpoint texture map based on different preset reference viewpoints;

[0041] The fusion processing module is used to perform weighted fusion on the virtual viewpoint texture map, and perform hole filling processing and foreground edge filtering processing on the fused virtual viewpoint texture map to obtain a target virtual viewpoint depth map.

[0042] A virtual viewpoint depth map processing method, device, apparatus and storage medium proposed in an embodiment of the present invention, compared with the prior art, in which the edge information of the depth map cannot be retained when the depth map is smoothed and the holes are filled using the existing algorithm, resulting in the phenomenon of ghosting in the edge area of ​​the synthetic texture map, and thus resulting in poor quality of the virtual viewpoint depth map, the present application forward maps the reference viewpoint depth map corresponding to the preset reference viewpoint to the position of the virtual viewpoint to obtain the virtual viewpoint depth map, and performs bilateral filtering smoothing on the virtual viewpoint depth map, which can effectively smooth the virtual viewpoint depth map and retain the edge According to the smoothed virtual viewpoint depth map, a reverse mapping relationship between the virtual viewpoint and the reference viewpoint is established to synthesize a virtual viewpoint texture map based on different preset reference viewpoints, and the virtual viewpoint texture map is weighted fused, and the fused virtual viewpoint texture map is hole-filled and foreground edge filtered, which not only effectively smoothes the virtual viewpoint depth map, but also retains the edge information of the depth map. After filling the holes in the virtual viewpoint texture map, the foreground edge of the texture map is filtered, which improves the quality of the virtual viewpoint texture map, thereby improving the quality of the virtual viewpoint depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the terminal\device structure of the hardware operating environment involved in the embodiment of the present invention;

[0044] Figure 2 It is a flowchart of a first embodiment of a method for processing a virtual viewpoint depth map according to the present invention;

[0045] Figure 3 for Figure 2 A detailed flow chart of step S40;

[0046] Figure 4Schematic diagram of a device for processing a virtual viewpoint depth map according to the present invention.

[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0049] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0050] The axis motion configuration terminal (also called terminal, device or terminal equipment) of the embodiment of the present invention may be a PC, or may be a mobile terminal equipment with a display function such as a smart phone, a tablet computer and a portable computer.

[0051] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0052] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0053] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0054] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a program of a virtual viewpoint depth map processing method.

[0055] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the program of the virtual viewpoint depth map processing method stored in the memory 1005. When the program of the virtual viewpoint depth map processing method is executed by the processor, the operations in the virtual viewpoint depth map processing method provided in the following embodiments are implemented.

[0056] Based on the above-mentioned device hardware structure, an embodiment of the virtual viewpoint depth map processing method of the present invention is proposed.

[0057] Reference Figure 2 A first embodiment of the present invention provides a method for processing a virtual viewpoint depth map, the method comprising:

[0058] Step S10, forward mapping the reference viewpoint depth map corresponding to the preset reference viewpoint to the position of the virtual viewpoint to obtain the virtual viewpoint depth map;

[0059] Step S20, performing bilateral filtering and smoothing processing on the virtual viewpoint depth map to obtain a smooth virtual viewpoint depth map;

[0060] Step S30, establishing a reverse mapping relationship between the virtual viewpoint and the reference viewpoint according to the smoothed virtual viewpoint depth map, so as to synthesize a virtual viewpoint texture map based on different preset reference viewpoints;

[0061] Step S40, weighted fusion is performed on the virtual viewpoint texture map, and hole filling processing and foreground edge filtering processing are performed on the fused virtual viewpoint texture map to obtain a target virtual viewpoint depth map.

[0062] Specifically, in step S10, a reference viewpoint depth map corresponding to a preset reference viewpoint is forward mapped to a position of a virtual viewpoint to obtain a virtual viewpoint depth map;

[0063] The preset reference viewpoint may be the actual shooting viewpoint of the real camera, the virtual viewpoint may be the shooting viewpoint of the virtual camera, the virtual camera may be converted by the shooting angle of the real camera, the image actually shot by the real camera at the preset reference viewpoint is the reference viewpoint image, the depth map of the reference viewpoint image is the reference viewpoint depth map, and the reference viewpoint image may be the depth map of the original image shot by the real camera. In this embodiment, the mapping between the image and the pixels in the image is based on the mapping of the coordinate system, a plane coordinate system is established with the reference viewpoint depth map and the virtual viewpoint depth map as reference, a three-dimensional space coordinate system is established with the virtual camera and the reference camera as reference systems, and the mapping between the image and the pixels can be regarded as a coordinate transformation based on the coordinate system. The reference viewpoint depth map is forward mapped to the position of the virtual viewpoint to obtain the virtual viewpoint depth map. The forward mapping refers to mapping the pixel points on the reference viewpoint depth map from the two-dimensional coordinate system to the three-dimensional coordinate system of the real camera, and then mapping them from the coordinate system of the real camera to the coordinate system of the virtual camera through translation and rotation, and finally performing an inverse transformation from the three-dimensional space coordinate system to the two-dimensional coordinate system, that is, mapping from the coordinate system of the virtual camera to the position of the virtual viewpoint.

[0064] The method of forward mapping the reference viewpoint depth map corresponding to the preset reference viewpoint to the position of the virtual viewpoint to obtain the virtual viewpoint depth map includes steps A1-A4:

[0065] Step A1, establishing a first spatial coordinate system according to internal parameters of a reference camera corresponding to the preset reference viewpoint, and mapping the reference viewpoint depth map to the first spatial coordinate system to obtain first spatial coordinates of the reference viewpoint depth map;

[0066] The internal parameters of the reference camera include focal length, optical center coordinates, etc. A first spatial coordinate system is established according to the internal parameters of the reference camera. The first spatial coordinate system is a three-dimensional spatial coordinate system, including the position of the reference camera. The first spatial coordinate system can be established with the position of the reference camera as the origin. The reference viewpoint depth map can establish a plane coordinate system, thereby obtaining the two-dimensional coordinates of the reference viewpoint depth map. The image coordinates of the reference viewpoint depth map are mapped into the first spatial coordinate system to obtain the first spatial coordinates of the reference viewpoint depth map. For example, the image coordinates (u, v) of the reference viewpoint depth map are transformed into the coordinates [X, Y, Z] in the reference camera coordinate system. T , the mapping transformation formula can be shown as follows (Formula 1):

[0067]

[0068] Where [u, v, 1] T is the homogeneous coordinate of the pixel point of the reference view depth map, Z is the depth value of the pixel point, and the depth value can be determined by the internal parameters and position of the real camera, [X, Y, Z] T is the coordinate of the real object corresponding to (u, v) in the first spatial coordinate system, f x 、f y 、c x 、c y is the internal parameter of the real camera, f x 、f y are the focal lengths in the x and y directions, c x 、c y are the coordinates of the optical center in the x and y directions respectively.

[0069] Step A2, establishing a second spatial coordinate system according to internal parameters of the virtual camera corresponding to the virtual viewpoint, and mapping the first spatial coordinates to the second spatial coordinate system to obtain second spatial coordinates of the reference viewpoint depth map;

[0070] The internal parameters of the virtual camera may be consistent with the internal parameters of the real camera. Referring to the first spatial coordinate system, a second spatial coordinate system is established according to the internal parameters of the virtual camera. The second spatial coordinate system includes the position of the virtual camera. The first spatial coordinate is mapped to the second spatial coordinate system to obtain the second spatial coordinate of the reference viewpoint depth map. The mapping transformation formula is as follows (Formula 2-4):

[0071]

[0072]

[0073] T=Rv ×(T c -T v ) (4)

[0074] Where R is a 3x3 rotation matrix, T is a translation vector, [X, Y, Z] T is the coordinate of the pixel point in the first spatial coordinate system, that is, the first spatial coordinate, [X1, Y1, Z1] T is the second spatial coordinate after the first spatial coordinate is mapped to the second spatial coordinate system, R c and T c is the rotation matrix and translation matrix of the preset reference viewpoint, R v and T v are the rotation matrix and translation matrix of the virtual viewpoint.

[0075] Step A3, performing an inverse transformation on the second space coordinates to obtain a first mapping relationship between the second space coordinate system and the image coordinates of the virtual viewpoint;

[0076] The second spatial coordinates are transformed, that is, the second spatial coordinates are mapped from a three-dimensional spatial coordinate system to a two-dimensional coordinate system, and the second spatial coordinates are mapped to the virtual viewpoint depth map to obtain a first mapping relationship between the second spatial coordinate system and the image coordinates of the virtual viewpoint, that is, a mapping relationship between the image of the virtual viewpoint and the virtual camera. The mapping transformation formula of the inverse transformation is as follows (Formula 5):

[0077]

[0078] Wherein, Z1 is the depth value of the pixel point, and the depth value can be determined by the internal parameters and position of the virtual camera, f x 、f y 、c x 、c y are the internal parameters of the virtual camera, which may be consistent with the internal parameters of the real camera, [u1,v1,1] T are the homogeneous coordinates of the pixel point (u1, v1) in the image coordinate system of the virtual viewpoint.

[0079] Step A4: Mapping the reference viewpoint depth map to the position of the virtual viewpoint according to the first mapping relationship to obtain the virtual viewpoint depth map.

[0080] According to the first mapping relationship, a forward mapping relationship of the reference viewpoint depth map to the virtual viewpoint can be obtained, that is, a mapping relationship from point (u, v) to point (u1, v1), so that the points on the reference viewpoint depth map can be mapped to the position of the virtual viewpoint, that is, according to the forward mapping relationship, the reference viewpoint depth map is mapped to the position of the virtual viewpoint to obtain the virtual viewpoint depth map.

[0081] Step S20, performing bilateral filtering and smoothing processing on the virtual viewpoint depth map to obtain a smooth virtual viewpoint depth map;

[0082] The virtual viewpoint depth map is subjected to bilateral filtering and smoothing processing using a preset algorithm. If part of the background texture map in the reference viewpoint is blocked by a foreground object, it is invisible in the reference viewpoint texture map, but visible in the virtual viewpoint texture map. In this case, there may be a hole area in the virtual viewpoint texture map. The depth discontinuity area in the depth map is the cause of the hole, so the depth map needs to be smoothed to reduce the hole area. A preferred method in the preset algorithm is a bilateral filtering algorithm, and the formula of the bilateral filtering algorithm is shown as (Formula 6):

[0083]

[0084] Wherein, d(i,j) is the depth value of the pixel point in the neighborhood of the pixel point (x,y). The size of (i,j) is determined by the filter radius. The filter radius can be determined by experiments and set by user. For example, if the filter radius is set to 7 pixels, then (i,j) is the pixel point within the range determined by the radius with (x,y) as the center. is the standard deviation of the bilateral filter, which can be customized. For example, when the filter radius is set to 7 pixels, The bilateral filtering method adopts a weighted average method, and uses the weighted average of the depth values ​​d(i,j) in the neighborhood range to determine the depth value d(x,y) of the center pixel point (x,y) of the range. The weighted average method is based on Gaussian distribution. The weight of the bilateral filtering method not only considers the Euclidean distance of the pixel points, that is, the influence of the position on the center pixel, but also considers the distance of the depth value. In the area where the depth value changes very little, the weight of the spatial domain plays a major role, which is equivalent to Gaussian smoothing. In the edge area of ​​the image, the depth value changes greatly, and the range domain weight of the pixel point becomes larger, thereby retaining the edge information and reducing the generation of holes.

[0085] Step S30, establishing a reverse mapping relationship between the virtual viewpoint and the reference viewpoint according to the smoothed virtual viewpoint depth map, so as to synthesize a virtual viewpoint texture map based on different preset reference viewpoints;

[0086] The reverse mapping means that, according to the depth of the virtual viewpoint, the pixel points on the virtual viewpoint depth map are mapped to the coordinate system of the virtual camera, and then mapped from the coordinate system of the virtual camera to the coordinate system of the real camera through translation and rotation, and mapped from the three-dimensional space coordinate system to the two-dimensional coordinate system through inverse transformation, that is, from the coordinate system of the real camera to the coordinate system of the reference viewpoint. The reverse mapping process is described in the following formula (Formula 7-9):

[0087] Mapping the pixel points on the virtual viewpoint depth map to the coordinate system of the virtual camera according to the depth of the virtual viewpoint:

[0088]

[0089] Where, Formula 7 is the same as Formula 5, [u1,v1,1] T is the homogeneous coordinate of the pixel point (u1, v1), Z1 is the depth value of the pixel point, [X1, Y1, Z1] T is the coordinate of the real object corresponding to the pixel point (u1, v1) in the virtual camera coordinate system, f x 、f y 、c x 、c y are the internal parameters of the virtual camera, which are the focal length and optical center coordinates in the x and y directions respectively.

[0090] Mapping from the coordinate system of the virtual camera to the coordinate system of the real camera:

[0091]

[0092] Among them, R' is a 3x3 rotation matrix, T' is a translation vector, [X1, Y1, Z1] T is the coordinate of the pixel point in the virtual camera coordinate system, [X', Y', Z'] T is the coordinate of the pixel point in the real camera coordinate system.

[0093] Mapping from the coordinate system of the real camera to the coordinate system of the reference viewpoint:

[0094]

[0095] Wherein, Z' is the depth value of the real camera, and (u', v') is the coordinate of the pixel point in the reference viewpoint coordinate system.

[0096] The method of establishing a reverse mapping relationship between the virtual viewpoint and the reference viewpoint according to the smoothed virtual viewpoint depth map to synthesize a virtual viewpoint texture map based on different preset reference viewpoints includes steps B1-B2:

[0097] Step B1, reverse mapping the smoothed virtual viewpoint depth map to obtain a second mapping relationship between the spatial coordinate system of the virtual viewpoint and the spatial coordinate system of the preset reference viewpoint;

[0098] By reverse mapping the smoothed virtual viewpoint depth map, a second mapping relationship between the spatial coordinate system of the virtual viewpoint and the spatial coordinate system of the preset reference viewpoint can be obtained. The reference camera, namely the real camera, can be reverse mapped to obtain a second mapping relationship between the spatial coordinate systems of the reference viewpoint and the virtual viewpoint, that is, spatial coordinate systems are respectively established according to the virtual viewpoint and the preset reference viewpoint, and the mapping relationship between the spatial coordinate system of the preset reference viewpoint and the spatial coordinate system of the virtual viewpoint is found through reverse mapping.

[0099] Step B2: Mapping the reference viewpoint texture map corresponding to the preset reference viewpoint to the position of the virtual viewpoint according to the second mapping relationship to obtain the virtual viewpoint texture map.

[0100] According to the second mapping relationship, a reverse mapping relationship from a point (u1, v1) on the virtual viewpoint image to a point (u', v') on the reference viewpoint can be obtained. According to the reverse mapping relationship, the reference viewpoint texture map is mapped to the position of the virtual viewpoint to obtain the virtual viewpoint texture map. There can be multiple reference viewpoints and reference cameras. Since each reference viewpoint corresponds to a reference viewpoint texture map, the reference viewpoint texture map corresponding to each reference viewpoint can be reverse mapped to obtain a virtual viewpoint texture map, thereby obtaining a virtual viewpoint texture map based on different preset reference viewpoints.

[0101] Step S40, weighted fusion is performed on the virtual viewpoint texture map, and hole filling processing and foreground edge filtering processing are performed on the fused virtual viewpoint texture map to obtain a target virtual viewpoint depth map.

[0102] The virtual viewpoint texture map is weightedly fused. The texture map quality obtained from different reference viewpoints and the focus of the object shooting are different, and the parts blocked by the foreground object are also different. Therefore, it is necessary to assign different weights to the texture maps obtained based on different preset reference viewpoints before fusion. The target virtual viewpoint depth map obtained retains the edge information of the depth map. After the weighted fusion of the virtual viewpoint texture map, the fused texture map is subjected to hole filling processing and foreground edge filtering processing, the holes in the depth map are filled, and the foreground edge is filtered to smooth the foreground edge and ensure the integrity of the edge information.

[0103] In this embodiment, a reference viewpoint depth map corresponding to a preset reference viewpoint is forward mapped to the position of a virtual viewpoint to obtain a virtual viewpoint depth map, and the virtual viewpoint depth map is smoothed using a preset filtering method to obtain a smooth virtual viewpoint depth map while retaining edge information of the depth map. Based on the smoothed virtual viewpoint depth map, a reverse mapping relationship between the virtual viewpoint and the reference viewpoint is established, and a virtual viewpoint texture map based on different preset reference viewpoints is synthesized. The virtual viewpoint texture map is weightedly fused, and the fused virtual viewpoint texture map is hole-filled and foreground edge-filtered, thereby reducing ghosting in the edge area of ​​the synthesized texture map and improving the quality of the virtual viewpoint texture map.

[0104] Further, see Figure 3 The second embodiment of the present invention provides a virtual viewpoint depth map processing method, based on the above Figure 2 The embodiment shown is a refinement of step S40 in the first embodiment.

[0105] Specifically, in step S40, weighted fusion is performed on the virtual viewpoint texture map, including steps S41-S44:

[0106] Step S41, determining a first position of a reference camera corresponding to the preset reference viewpoint, and a second position of a virtual camera corresponding to the virtual viewpoint;

[0107] Similarly, in this embodiment, the mapping between the image and the pixels in the image is also based on the mapping of the coordinate system. A plane coordinate system is established with reference to the reference viewpoint depth map and the virtual viewpoint depth map, and a three-dimensional space coordinate system is established with the virtual camera and the reference camera as the reference system. The mapping between the image and the pixels can be regarded as a coordinate transformation based on the coordinate system. The weight of the virtual viewpoint texture map includes two parts, wherein the first part of the weight is determined by the position of the virtual camera and the reference camera. Therefore, it is necessary to determine the position of the reference camera, i.e., the first position, and the position of the virtual camera, i.e., the second position.

[0108] Step S42, determining a first weight according to a positional relationship between the first position and the second position;

[0109] According to the positional relationship between the first position and the second position, the distance between the reference camera and the virtual camera is determined, and the first weight is determined by the distance between the reference camera and the virtual camera. For example, when the distance between the reference camera and the virtual camera is d i When the first weight can be i The first weight is determined inversely proportional to the value of the input signal.

[0110] Step S43, determining a depth value confidence of the virtual camera position, and determining a second weight according to the depth value confidence;

[0111] The weight of the virtual viewpoint texture map also includes a second weight, and the second weight is determined by the depth value confidence of the virtual camera position. The depth value confidence of the virtual camera position is first determined, and the second weight is determined according to the depth value confidence.

[0112] The step of determining the depth value confidence of the virtual camera position comprises steps C1-C3:

[0113] Step C1, determining a first reference point from the virtual viewpoint depth map, and mapping the first reference point to a spatial coordinate system where the first position is located, to obtain a second reference point;

[0114] Determine a pixel point from the virtual viewpoint depth map as the first reference point, the virtual viewpoint depth map can be used as a plane coordinate system, the first reference point can be (x, y), and map the first reference point to the spatial coordinate system where the first position is located. The spatial coordinate system where the first position is located is the first spatial coordinate system. According to the depth value of the virtual camera, the first reference point (x, y) is mapped to the position of the reference camera to obtain the second reference point, and the second reference point can be (u, v).

[0115] Step C2, mapping the second reference point to the spatial coordinate system where the second position is located according to the position depth value of the reference camera to obtain a third reference point;

[0116] According to the position depth value of the reference camera, the second reference point is mapped to the spatial coordinate system where the second position is located, the second position is the position of the reference camera, and the spatial coordinate system where the second position is located is the second spatial coordinate system. According to the position depth value of the reference camera, the second reference point (u, v) is mapped to the second spatial coordinate system to obtain a third reference point, and the third reference point can be (x_1, y_1).

[0117] Step C3: Determine the depth value confidence by using a first preset algorithm according to the coordinates of the first reference point and the coordinates of the third reference point.

[0118] The depth value confidence is obtained by using the first reference point and the third reference point using a first preset algorithm. The first preset algorithm may be the following formula (Formula 10):

[0119]

[0120] Wherein, dist is the depth value confidence. After determining the depth value confidence, the second weight can be determined. The second weight can be determined by the following formula (Formula 11):

[0121] conf_depth(x,y)=e -dist / 5 (11)

[0122] Step S44: According to the first weight and the second weight, the virtual viewpoint texture maps based on different preset reference viewpoints are merged.

[0123] After determining the weights of the two parts, the virtual viewpoint texture maps based on different preset reference viewpoints are weighted and then fused. The specific weighted fusion process can be the following formula (Formula 12):

[0124]

[0125] Among them, f i (x, y) is the value of the pixel point in the virtual viewpoint texture map, and f(x, y) is the weighted fusion with f in the virtual viewpoint texture map. i (x, y) corresponds to the pixel value, conf i (x, y) is the corresponding weight, which includes two parts, so the value of i is i=1,2, and the weight can be calculated by the following formula (Formula 13):

[0126] conf i (x,y) = conf_cam i (x,y)*conf_depth i (x,y) (13)

[0127] Among them, conf_cam i (x, y) is the first weight, which is determined by the distance between the reference camera and the virtual camera, conf_depth i (x, y) is a second weight, which is determined by the depth value confidence, where i can be the number of the virtual cameras, and the calculation method of the first weight can be the following formula (Formula 14):

[0128]

[0129] Among them, d i is the distance between the ith virtual camera and the reference camera.

[0130] In step S40, the hole filling process and foreground edge filtering process are performed on the fused virtual viewpoint texture map, including steps S45-S49:

[0131] Step S45, selecting a first central pixel point from the virtual viewpoint texture map, and creating a first window of a preset size;

[0132] The fused virtual viewpoint texture map is subjected to hole filling processing. A preferred hole filling processing method may be a joint bilateral filtering method. A pixel point is selected as the first central pixel point, and the first central pixel point may also be referred to as the current pixel point. A window of a preset size is established with the current pixel point as the center, such as creating a first window of size N*N. The first window contains N*N pixels, where N may be 30 or 40. Except for the first central pixel point, the other pixels in the first window are neighborhood pixels of the first central pixel point.

[0133] Step S46, using a second preset algorithm to perform filling calculation on the values ​​of the pixels in the first window to perform hole filling processing;

[0134] The second preset algorithm is used to calculate the pixels in the first window to fill the holes. The discontinuities in the depth map can be found by calculation, and then the pixels are filled by calculation. The filling method can be to use the average value of the pixels in the neighborhood of the hole, or the weighted average value of the neighborhood pixels as the depth value of the hole pixel to fill. The second preset algorithm can be the following formula (Formula 15-18):

[0135]

[0136] disp(i,j)=fB / depth(i,j) (16)

[0137] drange=max disp-min disp+1 (17)

[0138] max disp=fB / depth, min disp=fB / depth (18)

[0139] Among them, img(x, y) is the pixel point in the first window, disp(i, j) is the disparity value of the pixel point (i, j), depth(i, j) is the depth value of the pixel point (i, j), the value of fB is determined according to the internal parameters and position of the reference camera, drange is the maximum and minimum disparity value range, which is determined by the maximum and minimum disparity values, and the maximum and minimum disparity values ​​are determined by the corresponding fB values ​​of the reference camera. The maximum and minimum depth values ​​are determined according to the shooting scene, wherein dtr is a parameter used to narrow the disparity range, and the value can be a custom setting, for example, setting the value of dtr to 0.01666667.

[0140] Step S47, determining a foreground edge region of the virtual viewpoint texture image, and marking the foreground edge region;

[0141] After the holes of the virtual viewpoint texture map are filled, the edge area of ​​the virtual viewpoint texture map is determined and marked. The edge area is marked for filtering and smoothing the edge area to prevent ghosting or defects in the edge area.

[0142] The step of determining the foreground edge region of the virtual viewpoint texture image and marking the foreground edge region comprises steps D1-D3:

[0143] Step D1, selecting a reference pixel point and determining the gradient absolute value of the depth value of the reference pixel point;

[0144] A reference pixel is selected at the edge position of the virtual viewpoint depth map, and the gradient value of the reference pixel is determined. The horizontal gradient value and the vertical gradient value of the reference pixel can be determined from the horizontal and vertical directions, and the absolute values ​​of the horizontal gradient value and the vertical gradient value are calculated. For example, if the virtual viewpoint depth map is continuous, the absolute value of the gradient can be obtained by differentiating the edge area. If there is discontinuity in the virtual viewpoint texture map, the absolute value of the gradient can be obtained by forward difference quotient, backward difference quotient or intermediate difference quotient.

[0145] Step D2: if the absolute value of the gradient is greater than a preset threshold, determining that the reference pixel is an edge pixel;

[0146] If the absolute value of the gradient is greater than a preset threshold, the reference pixel is determined to be an edge pixel. The preset threshold is determined by the maximum and minimum disparity value range drange and the dtr parameter, and can be calculated by the following formula (Formula 19):

[0147] dthresh=dtr*drange (19)

[0148] Wherein, dthresh is the preset threshold.

[0149] Step D3: Expand a preset number of pixels around the edge pixel point to determine the foreground edge area, and mark the foreground edge area.

[0150] After determining that the reference pixel is an edge pixel, a preset number of pixels are expanded around the reference pixel, for example, 4 pixels are expanded in the horizontal direction and 4 pixels are expanded in the vertical direction. The expanded area is the foreground edge area, and the foreground edge area is marked.

[0151] Step S48, selecting a second central pixel point from the foreground edge area and creating a second window of a preset size;

[0152] A second central pixel point is selected from the foreground edge area, and a second window of a preset size is created with the second central pixel point as the center. For example, with the second central pixel point as the center, 4 pixels are extended around to create a 5*5 window as the second window. The second window is created to filter the foreground edge area.

[0153] Step S49: using a third preset algorithm, filtering calculation is performed on the values ​​of the pixels in the second window to perform filtering processing on the foreground edge area.

[0154] The pixel points in the second window are calculated by using a third preset algorithm to filter the foreground edge area. The value of the pixel point can be replaced by the average value or weighted average value of the pixel points in the neighborhood window of a certain pixel point to achieve the purpose of filtering. Taking the weighted average value of the pixel points in the neighborhood window as an example to replace the value of the pixel point, the formula of the third preset algorithm can be (Formula 20):

[0155]

[0156] Among them, (x, y) is the selected pixel that should be replaced, f(x, y) is the pixel value of the replaced pixel, and f(i, j) is the pixel value of the pixel in the neighborhood of point (x, y).

[0157] In this embodiment, a first weight is determined by determining a positional relationship between the virtual camera and the reference camera, a second weight is determined by the depth value confidence of the virtual camera, the virtual viewpoint texture map is weightedly fused according to the first weight and the second weight, and a central pixel point is selected from the fused virtual viewpoint texture map to create a first window and a second window of a preset size, thereby determining the neighborhood pixels of the central pixel point, and calculating the values ​​of the neighborhood pixels using a preset algorithm to perform hole filling and foreground edge filtering on the fused virtual viewpoint texture map, thereby reducing holes and ghosting phenomena in the virtual viewpoint texture map and improving the subjective and objective quality of the virtual viewpoint texture map.

[0158] Reference Figure 4 A first embodiment of the present invention provides a virtual viewpoint depth map processing device, the virtual viewpoint depth map processing device comprising:

[0159] A forward mapping module 10 is used to forward map a reference viewpoint depth map corresponding to a preset reference viewpoint to a position of a virtual viewpoint to obtain a virtual viewpoint depth map;

[0160] A post-processing module 20, configured to perform bilateral filtering and smoothing processing on the virtual viewpoint depth map to obtain a smooth virtual viewpoint depth map;

[0161] A reverse mapping module 30, configured to establish a reverse mapping relationship between the virtual viewpoint and the reference viewpoint according to the smoothed virtual viewpoint depth map, so as to synthesize a virtual viewpoint texture map based on different preset reference viewpoints;

[0162] The fusion processing module 40 is used to perform weighted fusion on the virtual viewpoint texture map, and perform hole filling processing and foreground edge filtering processing on the fused virtual viewpoint texture map to obtain a target virtual viewpoint depth map.

[0163] Optionally, the forward mapping module 10 includes:

[0164] A first mapping unit, configured to establish a first spatial coordinate system according to internal parameters of a reference camera corresponding to the preset reference viewpoint, and map the reference viewpoint depth map to the first spatial coordinate system to obtain first spatial coordinates of the reference viewpoint depth map;

[0165] A second mapping unit, configured to establish a second spatial coordinate system according to internal parameters of the virtual camera corresponding to the virtual viewpoint, and map the first spatial coordinates to the second spatial coordinate system to obtain second spatial coordinates of the reference viewpoint depth map;

[0166] an inverse transformation unit, configured to perform an inverse transformation on the second spatial coordinate system to obtain a first mapping relationship between the second spatial coordinate system and the image coordinates of the virtual viewpoint;

[0167] The third mapping unit is used to map the reference viewpoint depth map to the position of the virtual viewpoint according to the first mapping relationship to obtain the virtual viewpoint depth map.

[0168] Optionally, the reverse mapping module 30 includes:

[0169] A reverse mapping unit, configured to reversely map the smoothed virtual viewpoint depth map to obtain a second mapping relationship between the spatial coordinate system of the virtual viewpoint and the spatial coordinate system of the preset reference viewpoint;

[0170] The fourth mapping unit is used to map the reference viewpoint texture map corresponding to the preset reference viewpoint to the position of the virtual viewpoint according to the second mapping relationship to obtain the virtual viewpoint texture map.

[0171] Optionally, the fusion processing module 40 includes:

[0172] Determining a first position of a reference camera corresponding to the preset reference viewpoint and a second position of a virtual camera corresponding to the virtual viewpoint;

[0173] A first determining unit, configured to determine a first weight according to a positional relationship between the first position and the second position;

[0174] a second determining unit, configured to determine a depth value confidence of the virtual camera position, and determine a second weight according to the depth value confidence;

[0175] A fusion unit is used to fuse the virtual viewpoint texture maps based on different preset reference viewpoints according to the first weight and the second weight.

[0176] Optionally, the second determining unit includes:

[0177] A first mapping subunit is used to determine a first reference point from the virtual viewpoint depth map, and map the first reference point to a spatial coordinate system where the first position is located to obtain a second reference point;

[0178] A second mapping subunit is used to map the second reference point to a spatial coordinate system where the second position is located according to the position depth value of the reference camera to obtain a third reference point;

[0179] The calculation subunit is used to determine the depth value confidence by using a first preset algorithm according to the coordinates of the first reference point and the coordinates of the third reference point.

[0180] Optionally, the fusion processing module 40 further includes:

[0181] A creating unit, used for selecting a first central pixel point from the virtual viewpoint texture map and creating a first window of a preset size;

[0182] A calculation unit, used to perform a filling calculation on the values ​​of the pixels in the first window by using a second preset algorithm to perform a hole filling process;

[0183] A marking unit, used for determining a foreground edge region of the virtual viewpoint texture image and marking the foreground edge region;

[0184] A second creating unit, used for selecting a second central pixel point from the foreground edge area and creating a second window of a preset size;

[0185] The filtering unit is used to perform filtering calculation on the values ​​of the pixel points in the second window by using a third preset algorithm, so as to perform filtering processing on the foreground edge area.

[0186] Optionally, the marking unit comprises:

[0187] A first determination subunit is used to select a reference pixel point and determine a gradient absolute value of a depth value of the reference pixel point;

[0188] A second determining subunit, configured to determine that the reference pixel is an edge pixel if the absolute value of the gradient is greater than a preset threshold;

[0189] The marking subunit is used to expand a preset number of pixel points around the edge pixel point to determine the foreground edge area and mark the foreground edge area.

[0190] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0191] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0193] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for processing a virtual viewpoint depth map, characterized in that: The virtual viewpoint depth map processing method comprises the following steps: Forward mapping the reference viewpoint depth map corresponding to the preset reference viewpoint to the position of the virtual viewpoint to obtain the virtual viewpoint depth map; Performing bilateral filtering and smoothing processing on the virtual viewpoint depth map to obtain a smooth virtual viewpoint depth map; According to the smoothed virtual viewpoint depth map, establishing a reverse mapping relationship between the virtual viewpoint and the reference viewpoint to synthesize a virtual viewpoint texture map based on different preset reference viewpoints; Performing weighted fusion on the virtual viewpoint texture map, and performing hole filling processing and foreground edge filtering processing on the fused virtual viewpoint texture map to obtain a target virtual viewpoint depth map; The step of weighted fusion of the virtual viewpoint texture map comprises: Determining a first position of a reference camera corresponding to the preset reference viewpoint and a second position of a virtual camera corresponding to the virtual viewpoint; determining a first weight according to a positional relationship between the first position and the second position; Determining a depth value confidence of the virtual camera position, and determining a second weight according to the depth value confidence; According to the first weight and the second weight, the virtual viewpoint texture maps based on different preset reference viewpoints are merged; The step of determining the depth value confidence of the virtual camera position comprises: Determine a first reference point from the virtual viewpoint depth map, and map the first reference point to a spatial coordinate system where the first position is located to obtain a second reference point; According to the position depth value of the reference camera, the second reference point is mapped to the spatial coordinate system where the second position is located to obtain a third reference point; The depth value confidence is determined according to the coordinates of the first reference point and the coordinates of the third reference point.

2. The virtual viewpoint depth map processing method according to claim 1, characterized in that: The step of forward mapping the reference viewpoint depth map corresponding to the preset reference viewpoint to the position of the virtual viewpoint to obtain the virtual viewpoint depth map comprises: Establishing a first spatial coordinate system according to internal parameters of a reference camera corresponding to the preset reference viewpoint, and mapping the reference viewpoint depth map to the first spatial coordinate system to obtain first spatial coordinates of the reference viewpoint depth map; Establishing a second spatial coordinate system according to internal parameters of the virtual camera corresponding to the virtual viewpoint, and mapping the first spatial coordinates to the second spatial coordinate system to obtain second spatial coordinates of the reference viewpoint depth map; Performing an inverse transformation on the second spatial coordinates to obtain a first mapping relationship between the second spatial coordinate system and the image coordinates of the virtual viewpoint; According to the first mapping relationship, the reference viewpoint depth map is mapped to the position of the virtual viewpoint to obtain the virtual viewpoint depth map.

3. The virtual viewpoint depth map processing method according to claim 1, characterized in that: The step of establishing a reverse mapping relationship between the virtual viewpoint and the reference viewpoint according to the smoothed virtual viewpoint depth map to synthesize a virtual viewpoint texture map based on different preset reference viewpoints comprises: Reverse mapping the smoothed virtual viewpoint depth map to obtain a second mapping relationship between the spatial coordinate system of the virtual viewpoint and the spatial coordinate system of the preset reference viewpoint; According to the second mapping relationship, the reference viewpoint texture map corresponding to the preset reference viewpoint is mapped to the position of the virtual viewpoint to obtain the virtual viewpoint texture map.

4. The method for processing a virtual viewpoint depth map according to claim 1, wherein: The step of performing hole filling processing and foreground edge filtering processing on the fused virtual viewpoint texture map includes: Selecting a first central pixel point from the virtual viewpoint texture map and creating a first window of a preset size; Performing filling calculation on the values ​​of the pixel points in the first window to perform hole filling processing; Determining a foreground edge region of the virtual viewpoint texture image, and marking the foreground edge region; Selecting a second central pixel point from the foreground edge area and creating a second window of a preset size; A filtering calculation is performed on the values ​​of the pixel points in the second window to perform filtering processing on the foreground edge area.

5. The method for processing a virtual viewpoint depth map according to claim 4, wherein: The step of determining the foreground edge area of ​​the virtual viewpoint texture map and marking the foreground edge area comprises: Select a reference pixel point and determine the gradient absolute value of the depth value of the reference pixel point; If the absolute value of the gradient is greater than a preset threshold, determining that the reference pixel is an edge pixel; A preset number of pixels are expanded around the edge pixels to determine the foreground edge area, and the foreground edge area is marked.

6. A virtual viewpoint depth map processing device, characterized in that: The device comprises: A forward mapping module, used to forward map a reference viewpoint depth map corresponding to a preset reference viewpoint to a position of a virtual viewpoint to obtain a virtual viewpoint depth map; A post-processing module, used for performing bilateral filtering and smoothing processing on the virtual viewpoint depth map to obtain a smooth virtual viewpoint depth map; A reverse mapping module, used to establish a reverse mapping relationship between the virtual viewpoint and the reference viewpoint according to the smoothed virtual viewpoint depth map, so as to synthesize a virtual viewpoint texture map based on different preset reference viewpoints; A fusion processing module, used for performing weighted fusion on the virtual viewpoint texture map, and performing hole filling processing and foreground edge filtering processing on the fused virtual viewpoint texture map to obtain a target virtual viewpoint depth map; The fusion processing module is used to implement: Determining a first position of a reference camera corresponding to the preset reference viewpoint and a second position of a virtual camera corresponding to the virtual viewpoint; determining a first weight according to a positional relationship between the first position and the second position; Determining a depth value confidence of the virtual camera position, and determining a second weight according to the depth value confidence; According to the first weight and the second weight, the virtual viewpoint texture maps based on different preset reference viewpoints are merged; The fusion processing module is used to implement: Determine a first reference point from the virtual viewpoint depth map, and map the first reference point to a spatial coordinate system where the first position is located to obtain a second reference point; According to the position depth value of the reference camera, the second reference point is mapped to the spatial coordinate system where the second position is located to obtain a third reference point; The depth value confidence is determined based on the coordinates of the first reference point and the coordinates of the third reference point.

7. A virtual viewpoint depth map processing device, characterized in that: The virtual viewpoint depth map processing device includes: a memory, a processor, and a program for implementing the virtual viewpoint depth map processing method stored in the memory, and the processor is used to execute the program for implementing the virtual viewpoint depth map processing method to implement the steps of the virtual viewpoint depth map processing method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores a program for implementing a virtual viewpoint depth map processing method, and the program for implementing a virtual viewpoint depth map processing method is executed by a processor to implement the steps of the virtual viewpoint depth map processing method as described in any one of claims 1 to 5.

Images