Depth map processing method and apparatus, and storage medium

Abstract

The embodiment of the present disclosure provides a depth map processing method, device and storage medium. The depth map sequence of continuous frames and the color map sequence one-to-one registered with the depth map sequence are obtained, the depth map sequence includes a target depth map of a current time and each frame depth map of a previous time, the displacement change of each pixel in the target depth map is estimated by using the color map sequence, in the case that the displacement change of at least one target pixel is displayed as static, the target pixel is positioned in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, and the depth information of the target pixel in each frame depth map of the previous time is used to replace the depth information of the target pixel in the target depth map by using time filtering. The embodiment of the present disclosure can avoid using time filtering on the moving object in the depth map, thereby improving the accuracy of hole filling in the depth map.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to depth map processing methods, apparatus and storage media. Background Technology

[0002] In 3D computer graphics and computer vision, a depth map is an image or image channel that contains information about the distance of the surface of a scene object from the viewpoint, used to simulate 3D shapes or reconstruct them.

[0003] However, during depth map acquisition, areas may be missed by the camera or algorithmic issues may arise, resulting in holes at the edges of the depth map, such as... Figure 1 As shown, black area 10 is a hole compared to its adjacent area 11. The hole contains both foreground and background, but the depth values ​​of the foreground and background differ greatly.

[0004] Therefore, the relevant technology effectively fills the voids through time filtering.

[0005] In related technologies, temporal filtering refers to refining the current time depth map based on depth information from previous time depth maps. The term "current time depth map" refers to the depth image currently being processed by a depth image refinement device to obtain a refined depth image. The term "previous time depth map" refers to a depth image previously processed by a depth image refinement device to obtain a refined depth image.

[0006] Temporal filtering includes replacing the depth information of the current time's depth image with the depth information of a previous time's depth image, or replacing the depth information of the current time's depth image with a weighted sum of the depth information of the current time's depth image and the depth information of the previous time's depth image. However, these are just examples, and other types of temporal filtering can be used.

[0007] In summary, temporal filtering generally works like this: for a given pixel, obtain the depth values ​​of that pixel from the depth maps of the previous n frames, and replace the depth value of the current pixel with the depth information of these n pixels.

[0008] Since holes are unlikely to appear in all previous time depth maps at the same time, temporal filtering can fill the current time depth map to solve the problem of holes in the depth map. Summary of the Invention

[0009] In view of the shortcomings of the above-mentioned related technologies, the purpose of this disclosure is to provide a depth map processing method, apparatus and storage medium to solve the technical problem that the related technologies cannot accurately fill the holes in the depth map.

[0010] The first aspect of this disclosure provides a depth map processing method, which includes:

[0011] Acquire a sequence of depth maps for consecutive frames and a sequence of color maps that are registered one-to-one with the depth map sequence. The depth map sequence includes the target depth map at the current time and the depth maps of each frame at previous times.

[0012] The displacement change of each pixel in the target depth map is estimated using a sequence of color images;

[0013] When the displacement change of at least one target pixel appears to be static, the target pixel is located in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence.

[0014] Temporal filtering is employed to replace the depth information in the target depth map with the depth information of the target pixel in the depth map of each frame in previous time.

[0015] In some embodiments, estimating the displacement change of each pixel in the target depth map using a color image sequence includes:

[0016] The optical flow method is used to estimate the movement vector of each pixel in the target depth map using a sequence of color images, and the displacement change is obtained based on the movement vector.

[0017] In some embodiments, the depth processing method further includes:

[0018] For pixels whose displacement change is estimated to be a movement, their depth values ​​are retained in the target depth map.

[0019] In some embodiments, estimating the displacement change of each pixel in the target depth map using a color image sequence includes:

[0020] In the color image sequence, a target color image registered with the target depth image is taken. For each pixel in the target color image, its displacement change between each two adjacent color images at previous times is detected.

[0021] In some embodiments, when the displacement change of at least one target pixel appears to be static, before locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, the depth map processing method further includes:

[0022] If the displacement of any pixel between each pair of adjacent color images is estimated to be stationary, then it is identified as a stationary target pixel.

[0023] In some embodiments, when the displacement change of at least one target pixel appears to be static, before locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, the depth map processing method further includes:

[0024] For any pixel, if its displacement estimate between at least one pair of adjacent color images is non-stationary, it is identified as a moving pixel and its depth value is preserved in the target depth map.

[0025] In some embodiments, when the displacement change of at least one target pixel appears to be static, before locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, the depth map processing method further includes:

[0026] Determine whether the displacement change of each pixel is not greater than a threshold;

[0027] Pixels whose displacement changes are no greater than a threshold are selected as target pixels, and the displacement changes of the target pixels are static.

[0028] A second aspect of this disclosure provides a depth map processing apparatus, comprising:

[0029] The acquisition module acquires a sequence of depth maps for consecutive frames and a sequence of color maps that are registered one-to-one with the depth map sequence. The depth map sequence includes the target depth map at the current time and the depth maps of each frame at previous times.

[0030] The displacement estimation module uses a sequence of color images to estimate the displacement change of each pixel in the target depth map;

[0031] The pixel localization module locates the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence when the displacement change of at least one target pixel appears to be static.

[0032] The temporal filtering module employs temporal filtering to replace the depth information of a target pixel in the target depth map with its depth information from depth maps in previous frames.

[0033] The fifth aspect of this disclosure provides a computer apparatus, including: a communicator, a memory, and a processor; the communicator is used to communicate with an external source; the memory stores program instructions; and the processor is used to execute the program instructions to perform a depth map processing method as described in any of the first aspects.

[0034] The fourth aspect of this disclosure provides a computer-readable storage medium storing program instructions that are executed to perform a depth map processing method as described in any of the first aspects.

[0035] As described above, this disclosure provides a depth map processing method, apparatus, and storage medium. It acquires a sequence of depth maps for consecutive frames and a sequence of color maps registered one-to-one with the depth map sequence. The depth map sequence includes a target depth map at the current time and depth maps from previous frames. The displacement change of each pixel in the target depth map is estimated using the color map sequence. When the displacement change of at least one target pixel appears static, the target pixel is located in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence. Temporal filtering is applied, replacing the depth information in the target depth map with the depth information from the depth maps of previous frames. In this disclosure, pixel displacement detection is performed on the target depth map at the current time beforehand to identify stationary pixels, which are then subjected to temporal filtering. Therefore, this disclosure avoids applying temporal filtering to moving objects in the depth map, thereby improving the accuracy of hole filling in the depth map. Attached Figure Description

[0036] Figure 1 A depth map that demonstrates the relevant technologies.

[0037] Figure 2 One of the flowcharts illustrating a depth map processing method according to an embodiment of the present disclosure.

[0038] Figure 3 A second flowchart illustrating the depth map processing method according to an embodiment of this disclosure.

[0039] Figure 4 This demonstrates a depth map obtained using temporal filtering techniques.

[0040] Figure 5 A depth map obtained using temporal filtering processing according to embodiments of the present disclosure is shown.

[0041] Figure 6 A schematic diagram of a depth map processing apparatus according to an embodiment of the present disclosure is shown.

[0042] Figure 7 A schematic diagram of the structure of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0043] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the information disclosed herein. This disclosure can also be implemented or applied to systems through other different specific embodiments, and various details in this disclosure can also be modified or changed according to different viewpoints and application systems without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be combined with each other.

[0044] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, so that those skilled in the art to which this disclosure pertains can readily implement it. This disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

[0045] In this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic represented in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics represented may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples represented in this disclosure, as well as the features of those different embodiments or examples.

[0046] Furthermore, the terms "first" and "second" are used for illustrative purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the representation of this disclosure, "a plurality of" means two or more, unless otherwise expressly and specifically defined.

[0047] For the purpose of clarity, devices unrelated to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0048] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0049] Although the terms first, second, etc., are used in some instances herein to refer to various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, first interface and second interface, etc., are used. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of features, steps, operations, elements, modules, items, kinds, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, modules, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0050] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this disclosure. The singular form used herein includes the plural form unless the statement explicitly indicates otherwise. The word "comprising" as used in this specification means to specify a particular characteristic, region, integer, step, action, element, and / or component, and does not exclude the presence or addition of other characteristics, regions, integers, steps, actions, elements, and / or components.

[0051] Terms such as "below" and "above" indicating relative space are used to more easily explain the relationship of one device relative to another in the accompanying drawings. These terms refer not only to their meaning as shown in the drawings but also to other meanings or functions of the device in use. For example, if the device in the drawings is rotated, a device previously described as "below" another device may now be described as "above" another device. Therefore, the exemplary term "below" encompasses both above and below. The device may be rotated 90° or other angles, and the terms representing relative space are interpreted accordingly.

[0052] Although not explicitly defined, all terms, including technical and scientific terms used herein, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries shall be further interpreted as having a meaning consistent with the relevant technical literature and the message of the present disclosure, and shall not be over-interpreted as having an ideal or overly formulaic meaning unless otherwise defined.

[0053] In practice, the inventors of this case discovered that time filtering can usually only be applied to static scenes. If there are moving objects in the scene, the filtered value of the moving objects will be incorrect after time filtering. Therefore, time filtering is not suitable for scenes with moving objects.

[0054] Therefore, this embodiment proposes to perform displacement change detection on pixels in the current depth map, and only apply time filtering to the pixel when the pixel is detected to be stationary. This can avoid applying time filtering to moving objects in the depth map, thereby improving the accuracy of hole filling.

[0055] Figure 2 This is a flowchart of a depth map processing method provided in an embodiment of this disclosure. The execution subject of this method is a depth map processing system, such as... Figure 2 As shown, this method includes the following steps:

[0056] Step 210: Obtain the depth map sequence of consecutive frames and the color map sequence that is registered one-to-one with the depth map sequence. The depth map sequence includes the target depth map at the current time and the depth maps of each frame at previous times.

[0057] Step 220: Estimate the displacement change of each pixel in the target depth map using the color image sequence;

[0058] Step 230: When the displacement change of at least one target pixel appears to be static, locate the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence;

[0059] Step 240: Using temporal filtering, for the target pixel, replace its depth information in the target depth map with its depth information in the depth maps of previous frames.

[0060] In this embodiment, pixel displacement detection is performed on the target depth map at the current time beforehand to identify stationary pixels in the target depth map, and then temporal filtering is applied to them. Therefore, this embodiment can avoid applying temporal filtering to moving objects in the depth map, thereby improving the accuracy of hole filling in the depth map.

[0061] In this embodiment of the disclosure, the one-to-one registration of the depth map sequence and the color map sequence means that each pair of registered depth maps and color maps has the same shooting time, image size and shooting position, which can improve the accuracy and feasibility of the solution.

[0062] A depth map sequence consists of multiple consecutively captured depth maps, while a color map sequence consists of multiple consecutively captured color maps. The depth maps from previous time periods are used to perform temporal filtering on the target depth map at the current time.

[0063] In this embodiment of the disclosure, since the depth map sequence and the color map sequence are registered one-to-one, for each pixel in the target depth map, the corresponding pixel can be found in the color map at the current time, thereby enabling the estimation of the displacement change of each pixel pair in the target depth map using the color map sequence.

[0064] In one alternative embodiment, estimating the displacement change of each pixel in the target depth map using a color image sequence includes:

[0065] The optical flow method is used to estimate the movement vector of each pixel in the target depth map using a sequence of color images, and the displacement change is obtained based on the movement vector.

[0066] Optical flow is the instantaneous velocity of pixels moving on the imaging plane of a spatially moving object. The optical flow method utilizes the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames.

[0067] In short: optical flow is the instantaneous rate, which is equivalent to the displacement of the target point when the time interval is very small (such as between two consecutive frames in a video).

[0068] Specifically, the basic principle of optical flow is:

[0069] 1. Basic Assumptions

[0070] (1) Brightness remains constant. That is, the brightness of the same pixel does not change when it moves between different frames. This is an assumption of the basic optical flow method (which must be satisfied by all variants of the optical flow method) and is used to derive the basic equation of the optical flow method;

[0071] (2) The time is continuous or the motion is "small motion". That is, the change in time will not cause a drastic change in the target position, and the displacement between adjacent frames should be relatively small. This is also an indispensable assumption of the optical flow method.

[0072] 2. Basic constraint equations

[0073] Consider the light intensity of a pixel I(xy,t) in the first frame (where t represents its time dimension). It moves a distance of (dx,dy) to the next frame, taking time dt. Because it is the same pixel, according to the first assumption mentioned above, the light intensity of this pixel remains unchanged before and after the movement, that is:

[0074] I(x,y,t)=I(x+dx,y+dy,t+dt) (1)

[0075] Expanding the right side of equation (1) using Taylor series, we get:

[0076]

[0077] Where ε represents a second-order infinitesimal term, which can be ignored. Substituting (2) into (1) and dividing by dt, we get:

[0078]

[0079] Let μ and ν be the velocity vectors of the optical flow along the X-axis and Y-axis, respectively, then we get:

[0080]

[0081] make These represent the partial derivatives of the gray level of a pixel in the image along the X, Y, and T directions, respectively.

[0082] In summary, equation (3) can be written as:

[0083] I x μ+I u ν+I t =0 (5)

[0084] Among them, I x I y I t Both can be obtained from image data, and (μ, ν) is the optical flow vector to be determined.

[0085] There is only one constraint equation, but two unknowns. In this case, the exact values ​​of μ and ν cannot be obtained. Therefore, additional constraints need to be introduced. Introducing constraints from different perspectives leads to different methods for calculating optical flow fields. Based on the differences in theoretical foundation and mathematical methods, these methods can be divided into four types: gradient-based (differential) methods, matching-based methods, energy-based (frequency) methods, phase-based methods, and neurodynamic methods, without further elaboration here.

[0086] In this embodiment, optical flow detection is performed on pixels in every two adjacent color images in the color image sequence to obtain a movement vector for each pixel. This movement vector is a two-dimensional vector, which allows the displacement change to be obtained. In this case, for each pixel, its displacement change across multiple color images can be obtained.

[0087] Specifically, estimating the displacement change of each pixel in the target depth map using a color image sequence can include:

[0088] In the color image sequence, a target color image registered with the target depth image is taken. For each pixel in the target color image, its displacement change between each two adjacent color images at previous times is detected.

[0089] In this embodiment, for each pixel, a displacement change can be obtained for every two adjacent color images, and multiple displacement changes can be obtained for the corresponding color image sequence.

[0090] In one embodiment, for any pixel that is estimated to be stationary between each pair of adjacent color images, it is identified as a stationary target pixel.

[0091] In another embodiment, for any pixel whose displacement estimate between at least one pair of adjacent color images is non-stationary, it is identified as a moving pixel and its depth value is retained in the target depth map.

[0092] Therefore, for pixels whose displacement change is estimated to be moving, their depth values ​​are retained in the target depth map without temporal filtering.

[0093] In this embodiment of the disclosure, when the displacement change of at least one target pixel appears to be static, before locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, the depth map processing method further includes:

[0094] Determine whether the displacement change of each pixel is not greater than a threshold;

[0095] Pixels whose displacement change is no greater than a threshold are identified as target pixels, and the displacement change of the target pixels is stationary; otherwise, they are identified as moving pixels.

[0096] under, Figure 3 A specific depth map processing method is proposed for the flowchart. Specifically, this method includes the following steps:

[0097] Step 310: Obtain the current frame depth map Dm and its previous m-1 frame depth maps D1...Dm, and its registered color map sequence R1...Rm, where the previous m1 frame depth maps are used as temporal filters;

[0098] Step 320: Perform dense optical flow detection on each and all previous color images Ri and R(i-1), detecting the displacement of each pixel to obtain the movement vector of each pixel. The movement vector is a two-dimensional vector, and the displacement change is obtained. Set a threshold Th. If the value is greater than Th, the pixel is considered a moving pixel; otherwise, it is a stationary target pixel.

[0099] Step 330: For each pixel in the current frame depth map Dm, check whether all m-1 corresponding pixels in the color image sequence R2…Rm are stationary points. If they are all stationary points, use the median (maximum, minimum, or average) of the m-1 depths of the corresponding pixels in D2…Dm as the depth value of the current point to replace the depth value of the corresponding pixel in the current frame depth map Dm; if they are not all stationary points, retain the depth value in Dm without filtering.

[0100] Figure 4 This shows the depth map obtained by temporal filtering using relevant techniques. Figure 5 This displays a depth map obtained using temporal filtering processing according to embodiments of the present disclosure, and... Figure 4 compared to, Figure 5 Having fewer void areas indicates that the embodiments of this disclosure can fill depth map voids more accurately.

[0101] like Figure 6 The diagram illustrates a module schematic of a depth map processing apparatus according to an embodiment of this disclosure. It should be noted that the principle of the depth map processing apparatus can be referenced from the depth map processing methods in previous embodiments; therefore, the same technical content will not be repeated here.

[0102] The depth map processing device 600 may include:

[0103] The acquisition module 610 acquires a sequence of depth maps of consecutive frames and a sequence of color maps that are registered one-to-one with the depth map sequence. The depth map sequence includes the target depth map at the current time and the depth maps of each frame at previous times.

[0104] The displacement estimation module 620 uses a color image sequence to estimate the displacement change of each pixel in the target depth map;

[0105] The pixel localization module 630 locates the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence when the displacement change of at least one target pixel appears to be static.

[0106] The temporal filtering module 640 employs temporal filtering to replace the depth information in the target depth map with the depth information in the depth maps of previous frames.

[0107] In some embodiments, the displacement estimation module 620 is specifically used for:

[0108] The optical flow method is used to estimate the movement vector of each pixel in the target depth map using a sequence of color images, and the displacement change is obtained based on the movement vector.

[0109] In some embodiments, the time filtering module 640 is further configured to:

[0110] For pixels whose displacement change is estimated to be a movement, their depth values ​​are retained in the target depth map.

[0111] In some embodiments, the displacement estimation module 620 is specifically used for:

[0112] In the color image sequence, a target color image registered with the target depth image is taken. For each pixel in the target color image, its displacement change between each two adjacent color images at previous times is detected.

[0113] In some embodiments, the pixel positioning module 630 is further configured to:

[0114] If the displacement change of at least one target pixel appears to be stationary, based on the registration relationship between the depth map sequence and the color map sequence, before locating the target pixel in the depth map sequence, for any pixel, if its displacement between each pair of adjacent color maps is estimated to be stationary, it is determined to be a stationary target pixel.

[0115] In some embodiments, the pixel positioning module 630 is further configured to:

[0116] If the displacement change of at least one target pixel appears to be stationary, based on the registration relationship between the depth map sequence and the color map sequence, before locating the target pixel in the depth map sequence, for any pixel whose displacement estimate between at least one pair of adjacent color maps is non-stationary, it is identified as a moving pixel and its depth value is retained in the target depth map.

[0117] In some embodiments, the displacement estimation module 620 is further configured to:

[0118] If the displacement change of at least one target pixel appears to be stationary, based on the registration relationship between the depth map sequence and the color map sequence, before locating the target pixel in the depth map sequence, it is determined whether the displacement change of each pixel is not greater than a threshold.

[0119] Pixels whose displacement changes are no greater than a threshold are selected as target pixels, and the displacement changes of the target pixels are static.

[0120] It should be noted that, in Figure 6The various functional modules in the embodiments can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a program instruction product. A program instruction product includes one or more program instructions. When the program instructions are loaded and executed on a computer, all or part of the flow or function according to this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0121] and, Figure 6 The apparatus disclosed in the embodiments can be implemented through other modular division methods. The apparatus embodiments shown above are merely illustrative. For example, the module division is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules or modules may be combined or dynamically integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces. The indirect coupling or communication connection between devices or modules may be electrical or other forms.

[0122] in addition, Figure 6 The functional modules and sub-modules in the embodiments can be dynamically integrated within a single processing unit, or each module can exist physically independently, or two or more modules can be dynamically integrated within a single unit. These dynamic units can be implemented in hardware or as software functional modules. If these dynamic units are implemented as software functional modules and executed as independent products for sale or use, they can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0123] This disclosure also provides a computer-readable storage medium storing program instructions, which are executed before being run. Figure 2 The method steps in the embodiments.

[0124] The method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or implemented as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium after being downloaded via a network. Thus, the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA).

[0125] It should be specifically noted that the flowchart representations of the embodiments described above in this disclosure can be understood as representing modules, segments, or portions of code comprising one or more executable instructions for implementing a specific logical function or process. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved.

[0126] For example, Figure 2 The order of the steps in the illustrated embodiments may vary in specific scenarios and is not limited to the above representation.

[0127] like Figure 7 The diagram shown illustrates the structure of a computer device according to an embodiment of the present disclosure.

[0128] In some embodiments, the computer device is used to load program instructions for implementing a depth map processing method. The computer device may specifically be implemented as, for example, a server, desktop computer, laptop computer, mobile terminal, etc., and may be used by an implementer who stores and / or runs these program instructions for commercial purposes such as development or testing.

[0129] Figure 7 The computer device 700 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0130] like Figure 7 As shown, the computer device 700 is presented in the form of a general-purpose computing device. The components of the computer device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

[0131] The storage unit stores program code, which can be executed by the processing unit 710, enabling the computer device to implement the embodiments of the present disclosure (e.g., Figure 2 The method steps are described in the example.

[0132] In some embodiments, storage unit 720 may include volatile storage units, such as random access memory (RAM) 721 and / or cache storage unit 722, and may further include read-only memory (ROM) 723.

[0133] In some embodiments, the storage unit 720 may also include a program / utility 724 having a set (at least one) of program modules 725, such program modules 725 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0134] In some embodiments, bus 730 may include a data bus, an address bus, and a control bus.

[0135] In some embodiments, the computer device 700 may also communicate with one or more external devices 70 (e.g., keyboard, pointing device, Bluetooth device, etc.) via an input / output (I / O) interface 750. Optionally, the computer device 700 also includes a display unit 740 connected to the input / output (I / O) interface 750 for display purposes. Furthermore, the computer device 700 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the computer device 700 via a bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the computer device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0136] In summary, the embodiments of this disclosure provide a depth map processing method, apparatus, and storage medium. By acquiring a sequence of depth maps of consecutive frames and a sequence of color maps that are registered one-to-one with the depth map sequence, the depth map sequence includes a target depth map at the current time and depth maps of each frame at previous times. The displacement change of each pixel in the target depth map is estimated using the color map sequence. When the displacement change of at least one target pixel appears to be static, the target pixel is located in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence. Temporal filtering is applied, and for the target pixel, its depth information in the target depth map is replaced by its depth information in the depth maps of each frame at previous times.

[0137] In this embodiment, pixel displacement detection is performed on the target depth map at the current time beforehand to identify stationary pixels in the target depth map, and then temporal filtering is applied to them. Therefore, this embodiment can avoid applying temporal filtering to moving objects in the depth map, thereby improving the accuracy of hole filling in the depth map.

[0138] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. A depth map processing method, characterized in that, include: Obtain a sequence of depth maps for consecutive frames and a sequence of color maps that are registered one-to-one with the depth map sequence. The depth map sequence includes the target depth map at the current time and the depth maps of each frame at previous times. Estimating the displacement change of each pixel in the target depth map using the color image sequence includes: using optical flow to estimate the movement vector of each pixel in the target depth map using the color image sequence, and obtaining the displacement change based on the movement vector; When the displacement change of at least one target pixel appears to be static, the target pixel is located in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, wherein the static state is defined as the displacement change not exceeding a threshold. Temporal filtering is employed to replace the depth information in the target depth map with the depth information in the depth maps of previous frames, thereby filling the holes in the target depth map.

2. The depth map processing method according to claim 1, characterized in that, The depth processing method further includes: For pixels whose displacement change is estimated to be a movement, their depth values ​​are retained in the target depth map.

3. The depth map processing method according to claim 1, characterized in that, The step of estimating the displacement change of each pixel in the target depth map using the color image sequence includes: In the color image sequence, a target color image registered with the target depth image is taken, and for each pixel in the target color image, its displacement change between each two adjacent color images at a previous time is detected.

4. The depth map processing method according to claim 3, characterized in that, When the displacement change of at least one target pixel appears static, before locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, the depth map processing method further includes: If the displacement of any pixel between each pair of adjacent color images is estimated to be stationary, then it is identified as a stationary target pixel.

5. The depth map processing method according to claim 4, characterized in that, When the displacement change of at least one target pixel appears static, before locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, the depth map processing method further includes: For any pixel, if its displacement estimate between at least one pair of adjacent color images is non-stationary, it is identified as a moving pixel and its depth value is retained in the target depth image.

6. The depth map processing method according to claim 1, characterized in that, When the displacement change of at least one target pixel appears static, before locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, the depth map processing method further includes: Determine whether the displacement change of each pixel is not greater than a threshold; The target pixel is defined as a pixel whose displacement change is no greater than a threshold, and the displacement change of the target pixel is stationary.

7. A depth map processing apparatus, characterized in that, include: The acquisition module acquires a sequence of depth maps of consecutive frames and a sequence of color maps that are registered one-to-one with the depth map sequence. The depth map sequence includes the target depth map at the current time and the depth maps of each frame at previous times. The displacement estimation module estimates the displacement change of each pixel in the target depth map using the color image sequence, including: using optical flow to estimate the movement vector of each pixel in the target depth map using the color image sequence, and obtaining the displacement change based on the movement vector; A pixel positioning module, when the displacement change of at least one target pixel appears to be static, locates the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence, wherein the static state is defined as the displacement change not exceeding a threshold. The temporal filtering module employs temporal filtering to replace the depth information of the target pixel in the target depth map with its depth information in the depth maps of previous frames, thereby filling the holes in the target depth map.

8. A computer device, characterized in that, include: Communicators, memory, and processors; The communicator is used for communication with the outside world; The memory stores program instructions; The processor is used to run the program instructions to perform the depth map processing method as described in claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The system stores program instructions that are executed to perform the depth map processing method as described in claims 1 to 6.

Images