Depth map reconstruction method, system, device, storage medium and processor

Abstract

The present invention discloses a method, system, device, storage medium and processor for reconstructing a depth map. The method comprises: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixel points of each depth map; determining the pixel area to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction with the edge pixel point as a reference; in the process of virtual viewpoint mapping of the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions. The present invention solves the technical problem that, during the virtual viewpoint reconstruction process of the depth map, the original hole area generates a mapping texture of the foreground.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a depth map reconstruction method, system, device, storage medium and processor. Background Art

[0002] Currently, depth image based rendering (DIBR) algorithms are designed based on the assumption of a correct depth map and do not optimize for depth map compression losses. Any pixel in the original image can be mapped to the target's virtual viewpoint position to fill holes, but this assumes that the original depth map is sufficiently accurate.

[0003] However, after the depth map is compressed, transmitted, and processed, there will be a certain amount of image loss. When such a depth map is mapped to a virtual viewpoint, it will cause the foreground to fly out, resulting in the generation of foreground mapping texture in the original empty area.

[0004] With respect to the technical problem that, during the virtual viewpoint reconstruction process of the depth map, the original hole area generates a mapped texture of the foreground, no effective solution has been proposed so far. Summary of the Invention

[0005] Embodiments of the present invention provide a depth map reconstruction method, system, device, storage medium, and processor to at least solve the technical problem that, during the virtual viewpoint reconstruction process of the depth map, the original hole area generates a foreground mapping texture.

[0006] According to one aspect of an embodiment of the present invention, a method for reconstructing a depth map is provided. The method may include: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixels of each depth map; determining a pixel region to be shielded in the depth map based on the edge pixels of each depth map, wherein the pixel region to be shielded includes at least one pixel in a foreground direction relative to the edge pixel; and performing image mapping on the pixel region to be shielded in the depth map of a target viewpoint during virtual viewpoint mapping of the depth map, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0007] According to another aspect of an embodiment of the present invention, another method for reconstructing a depth map is provided. The method may include: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixels of each depth map; and in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0008] According to another aspect of an embodiment of the present invention, another method for reconstructing a depth map is provided. The method may include: displaying depth maps of multiple original viewpoints; displaying edge pixel points extracted after edge detection on each depth map; and displaying an image result after image mapping is performed on a pixel area to be shielded in the depth map of a target viewpoint during virtual viewpoint mapping of the depth map, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions, and the pixel area to be shielded includes at least one pixel point in a foreground direction based on the edge pixel point.

[0009] According to another aspect of an embodiment of the present invention, another method for reconstructing a depth map is provided. The method may include: determining a target viewpoint from a plurality of original viewpoints, wherein the target viewpoint is a viewpoint that satisfies a mapping condition among the plurality of original viewpoints; obtaining a depth map of the target viewpoint, performing edge detection on the depth map of the target viewpoint, and obtaining edge pixel points of the depth map of the target viewpoint; determining a pixel area to be shielded based on the edge pixel points of the depth map of the target viewpoint, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction with the edge pixel point as a reference; in the process of performing virtual viewpoint mapping on the depth maps of the plurality of original viewpoints, selecting to perform image mapping on the pixel area to be shielded in the depth map of the target viewpoint.

[0010] According to another aspect of an embodiment of the present invention, another method for reconstructing a depth map is also provided. The method may include: during a live broadcast, obtaining depth maps of multiple original viewpoints on a live broadcast screen; performing edge detection on the depth map of each original viewpoint to obtain edge pixel points of each depth map; determining the pixel area to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction with the edge pixel point as a reference; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints.

[0011] According to another aspect of an embodiment of the present invention, another method for reconstructing a depth map is provided. The method may include: obtaining depth maps of multiple original viewpoints, selecting an image of a predetermined area in the depth map of each original viewpoint, and obtaining a pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction with an edge pixel point as a reference; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0012] According to another aspect of an embodiment of the present invention, a depth map reconstruction system is also provided. The method may include: a client for displaying depth maps of multiple original viewpoints; a cloud server, communicating with the client, for obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint to generate edge pixels, and then determining a pixel area to be shielded in the depth map, wherein the pixels to be shielded include at least one pixel in the foreground direction with the edge pixels as a reference; wherein, in the process of performing virtual viewpoint mapping on the depth map, the cloud server performs image mapping on the pixel area to be shielded in the depth map of the target viewpoint, and returns a reconstructed image generated based on the mapping result to the client, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0013] According to another aspect of an embodiment of the present invention, a depth map reconstruction device is also provided. The method may include: an acquisition unit for acquiring depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and acquiring edge pixel points of each depth map; a determination unit for determining a pixel area to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction with the edge pixel point as a reference; and a prohibition unit for performing image mapping on the pixel area to be shielded in the depth map of a target viewpoint during virtual viewpoint mapping of the depth map, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0014] According to another aspect of an embodiment of the present invention, a processor is provided, wherein the processor is configured to run a program, wherein when the program is run, the depth map reconstruction method of the embodiment of the present invention is executed.

[0015] According to another aspect of an embodiment of the present invention, a depth map reconstruction system is also provided. The depth map reconstruction system includes: a processor; a memory connected to the processor, and configured to provide the processor with instructions for processing the following processing steps: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixel points of each depth map; determining a pixel area to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction based on the edge pixel point; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0016] In an embodiment of the present invention, depth maps of multiple original viewpoints are obtained, edge detection is performed on the depth map of each original viewpoint, and edge pixels of each depth map are obtained; based on the edge pixels of each depth map, a pixel area to be shielded in the depth map is determined, wherein the pixel area to be shielded includes at least one pixel in a foreground direction with the edge pixel as a reference; and in the process of virtual viewpoint mapping of the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints. That is, after obtaining the edge pixels of the depth map, the present application performs image mapping on the pixel area to be shielded in the depth map during the process of virtual viewpoint mapping of the depth map, thereby ensuring that the virtual viewpoint reconstruction based on the depth map has robustness against compression loss, improving the interpolation quality of the foreground edge of the depth map, and solving the technical problem that the original hole area generates the foreground mapping texture during the virtual viewpoint reconstruction of the depth map, thereby achieving the technical effect that the original hole area does not generate the foreground mapping texture during the virtual viewpoint reconstruction of the depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1A is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a depth map reconstruction method according to an embodiment of the present invention;

[0019] Figure 1B is a schematic structural diagram of depth map reconstruction in a specific application scenario according to an embodiment of the present invention;

[0020] Figure 2 is a flowchart of a depth map reconstruction method according to an embodiment of the present invention;

[0021] Figure 3 is a flowchart of another depth map reconstruction method according to an embodiment of the present invention;

[0022] Figure 4 is a flowchart of a depth map reconstruction method according to an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of a depth map reconstruction system according to an embodiment of the present invention;

[0024] Figure 6 is a schematic diagram of a depth map loss according to an embodiment of the present invention;

[0025] Figure 7 is a schematic diagram of the effect of depth map compression loss on DIBR quality according to an embodiment of the present invention;

[0026] Figure 8 is a schematic diagram of edge detection and edge mapping shielding of a depth map according to an embodiment of the present invention;

[0027] Figure 9 is a schematic diagram of an interpolation result according to an embodiment of the present invention;

[0028] Figure 10 is a schematic diagram of a depth map reconstruction device according to an embodiment of the present invention;

[0029] Figure 11 is a schematic diagram of another depth map reconstruction device according to an embodiment of the present invention;

[0030] Figure 12 is a schematic diagram of another depth map reconstruction device according to an embodiment of the present invention; and

[0031] Figure 13 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0035] Depth Image Based Rendering (DIBR) uses depth information to project the reference image into 3D Euclidean space, and then projects the 3D space points onto the imaging plane of the virtual camera.

[0036] A depth map is an image or image channel containing information about the distance to the surface of scene objects from the viewpoint.

[0037] Free viewpoint video: To provide a highly flexible viewing experience, users can interactively adjust the viewing angle and watch from any desired viewpoint.

[0038] The 6DoF parameter refers to the six degrees of freedom, specifically the translation parameters along three directions and the rotation parameters around three axes.

[0039] Example 1

[0040] According to an embodiment of the present invention, an embodiment of a method for reconstructing a depth map is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1A FIG is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a depth map reconstruction method according to an embodiment of the present invention. Figure 1A As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1A The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1A More or fewer components than shown, or with Figure 1A Different configurations shown.

[0042] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0043] The memory 104 can be used to store software programs and modules for application software, such as the program instructions / data storage device corresponding to the depth map reconstruction method in the embodiment of the present invention. The processor 102 executes the software programs and modules stored in the memory 104 to perform various functional applications and data processing, thereby implementing the depth map reconstruction method of the aforementioned application. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0044] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0045] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0046] It should be noted that, in some optional embodiments, the above Figure 1A The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1AThis is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the aforementioned computer device (or mobile device).

[0047] Figure 1B The present invention is a schematic structural diagram of a depth map reconstruction system for a specific application scenario according to an embodiment of the present invention, which illustrates the layout of a depth map reconstruction system 20. The depth map reconstruction system 20 may include an acquisition array 21 composed of multiple acquisition devices, a data processing device 22, a cloud-based server cluster 23 (which may include: server 231, server 232, server 233, server 234), a playback control device 24, a playback terminal 25, and an interactive terminal 26. After acquiring edge pixels of the depth map, the depth map reconstruction system 20 performs image mapping on the pixel areas to be shielded in the depth map during virtual viewpoint mapping of the depth map. This ensures that the virtual viewpoint reconstruction based on the depth map is robust against compression loss and improves the interpolation quality of the foreground edges of the depth map.

[0048] Specifically, refer to Figure 1B The acquisition array 21 may include a plurality of cameras, which may be placed in different positions of the on-site acquisition area in a fan shape according to a preset multi-angle free viewing angle range.

[0049] The data processing device 22 can send instructions to each camera in the acquisition array 21 through a wireless local area network. Based on the instructions sent by the data processing device 22, each acquisition device in the acquisition array 21 transmits the image captured by the camera to the data processing device 22.

[0050] The interactive terminal 26 of this embodiment triggers an instruction to reconstruct the depth map based on an interactive operation. When the data processing device 22 detects an interactive operation occurring on the operating interface of the interactive terminal 26, it can respond to the instruction, obtain depth maps of multiple original viewpoints, perform edge detection on the depth map of each original viewpoint, obtain edge pixel points of each depth map, and determine the pixel area to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction with the edge pixel point as a reference. In the process of virtual viewpoint mapping of the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions, and then the image result after the virtual viewpoint position mapping of the depth map is uploaded to the server cluster 23 in the cloud. The server cluster 23 can send the image result to the interactive terminal 26 for display.

[0051] As another optional implementation, after the data processing device 22 detects an interactive operation occurring on the operating interface of the interactive terminal 26, the depth maps of multiple original viewpoints can be uploaded to the server cluster 23 in the cloud. The server cluster 23 performs edge detection on the depth map of each original viewpoint, obtains the edge pixel points of each depth map, and determines the pixel area to be shielded in the depth map based on the edge pixel points of each depth map. In the process of mapping the depth map to a virtual viewpoint, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint. The server cluster 23 then sends the mapped image result to the interactive terminal 26 for display.

[0052] Then, the playback control device 24 can receive the image results sent by the server cluster 23, and the playback terminal 25 receives the image results from the playback control device 24 and plays them in real time. The playback control device 24 can be a manual playback control device or a virtual playback control device. In a specific implementation, a director control device, such as a director console, can serve as a playback control device in the embodiments of the present invention.

[0053] This embodiment utilizes the aforementioned depth map reconstruction system. On the one hand, a user can trigger a depth map reconstruction command via the interactive terminal 26; on the other hand, a user can directly view the resulting image after the depth map is mapped to a virtual viewpoint position via the playback terminal 25. It is understood that the aforementioned depth map reconstruction system 20 may include only the playback terminal 25 or only the interaction terminal 26, or the same terminal device may serve as both the playback terminal 25 and the interaction terminal 26.

[0054] Those skilled in the art will understand that, after compression, transmission, and other processing, the depth map will have a certain image loss. When such a depth map is mapped to a virtual viewpoint, it will cause the foreground to fly out, resulting in the generation of foreground mapping texture in the original hole area. As a result, in the process of reconstructing the virtual viewpoint of the depth map, the technical problem of foreground mapping texture being generated in the original hole area arises.

[0055] In view of this, the embodiment of this specification provides a solution. Figure 1A or Figure 1B Under the operating environment shown, this application provides Figure 2 It should be noted that the depth map reconstruction method of this embodiment can be Figure 1A The mobile terminal of the embodiment shown executes or Figure 1B The depth map reconstruction system is performed as shown.

[0056] Figure 2 FIG. 1 is a flowchart of a depth map reconstruction method according to an embodiment of the present invention. Figure 2As shown, the method may include the following steps:

[0057] Step S202 : obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixels of each depth map.

[0058] In the technical solution provided in the above step S202 of the present invention, there can be multiple original viewpoints, each original viewpoint has a corresponding depth map, and depth maps of multiple original viewpoints are obtained. The depth map is an image or image channel containing information related to the distance of the surface of the scene object of the viewpoint, and can be a depth map used for compression, transmission and other processing.

[0059] This embodiment performs edge detection on the depth map of each original viewpoint. Edge detection is a concept in image processing and computer vision, and can be used to detect pixel points at the edge of the depth map of the original viewpoint, that is, the edge pixel points of the depth map, thereby obtaining the edge pixel points of each depth map.

[0060] Optionally, in this embodiment, edge detection is performed on the depth map of each original viewpoint. The edge detection can be performed by simply comparing |Depth_left-Depth_right|>THR.

[0061] Step S204 : determining a pixel region to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel region to be shielded includes at least one pixel point in a direction toward the foreground with the edge pixel point as a reference.

[0062] In the technical solution provided in step S204 of the present invention, the edge of the depth map suffers a certain loss due to compression and transmission, and the original large value of the foreground is changed to a smaller value. When such a depth map is mapped to the virtual viewpoint, it will cause the foreground to fly out, resulting in the generation of a foreground mapping texture in the original hole area. However, this embodiment obtains depth maps of multiple original viewpoints, performs edge detection on the depth map of each original viewpoint, and obtains edge pixel points of each depth map. Then, based on the edge pixel points of each depth map, a pixel area to be shielded in the depth map can be determined. The pixel area to be shielded includes a pixel set. The multiple pixel sets can be pre-marked at least one pixel point facing the foreground direction based on the edge pixel point as a reference, for example, the pixel points other than the pixel points whose edge of the depth map is lost after compression. When performing the mapping operation, the other pixel points in the depth map other than the pixel area to be shielded can be reconstructed using the depth maps of all original viewpoints.

[0063] Step S206 : in the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint.

[0064] In the technical solution provided in the above-mentioned step S206 of the present invention, after determining the pixel area to be shielded in the depth map based on the edge pixel points of each depth map, in the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among multiple original viewpoints that meets the mapping conditions.

[0065] In this embodiment, the process of mapping the depth map to a virtual viewpoint refers to the process of mapping the depth map to the virtual viewpoint position of the target, that is, a virtual viewpoint reconstruction (DIBR) process based on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint. For example, the pixel area to be shielded includes pixels in the depth map that are relatively close to the virtual viewpoint, and image mapping is performed on them, while image mapping is prohibited for pixels in the depth map that are relatively far from the virtual viewpoint. The target viewpoint is a viewpoint among multiple original viewpoints that meets the mapping conditions, that is, a viewpoint used to realize virtual viewpoint mapping of the depth map, thereby ensuring that the DIBR algorithm is robust against compression loss, and significantly improving the difference quality of the foreground edge of the depth map.

[0066] Through the above steps S202 to S206 of the present application, depth maps of multiple original viewpoints are obtained, edge detection is performed on the depth map of each original viewpoint, and edge pixels of each depth map are obtained; based on the edge pixels of each depth map, a pixel area to be shielded in the depth map is determined, wherein the pixel area to be shielded includes at least one pixel in a foreground direction with the edge pixel as a reference; and in the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints. In other words, after obtaining the edge pixels of the depth map, this embodiment performs image mapping on the pixel area to be shielded in the depth map during the process of performing virtual viewpoint mapping on the depth map, thereby ensuring that the virtual viewpoint reconstruction based on the depth map has robustness against compression loss, improving the interpolation quality of the foreground edge of the depth map, and solving the technical problem that the original hole area generates foreground mapping texture during the virtual viewpoint reconstruction of the depth map, thereby achieving the technical effect that the original hole area does not generate foreground mapping texture during the virtual viewpoint reconstruction of the depth map.

[0067] The above method of this embodiment is further introduced below.

[0068] As an optional implementation manner, the target viewpoint is: a viewpoint among the multiple original viewpoints whose distance from the virtual viewpoint is within a predetermined range.

[0069] In this embodiment, the target viewpoint is a viewpoint among multiple original viewpoints that meets the mapping conditions, the distance between each original viewpoint and the virtual viewpoint is obtained, and it is determined whether the distance between each original viewpoint and the virtual viewpoint is within a predetermined range. If there is a viewpoint among the multiple original viewpoints whose distance with the virtual viewpoint is within the predetermined range, it is determined that the viewpoint meets the mapping conditions and is determined as the target viewpoint.

[0070] As an optional implementation manner, the pixels within the pixel area to be shielded are marked, wherein the mapping operation is performed only on the marked pixels in the depth map of the target viewpoint.

[0071] For real-world image reconstruction, multiple images are retrieved, all captured by real-world cameras in a certain configuration. In this embodiment, images from multiple original cameras within a predetermined range from the current virtual viewpoint are read to complete the image reconstruction of that virtual viewpoint. During the virtual viewpoint mapping process from the depth map, the pixels of the images captured by original cameras close to the virtual viewpoint are used as the basis for image reconstruction, while the images captured by original cameras farther from the virtual viewpoint are masked and not mapped during the image reconstruction process.

[0072] The reason for shielding the mapping of pixels of the original camera image that is far from the virtual viewpoint is mainly due to the physical distance between the original camera and the virtual viewpoint. The greater the distance between the original camera and the virtual viewpoint, the greater the error in the reconstructed image after mapping. Therefore, to ensure the quality of the reconstructed image and avoid the generation of holes, this embodiment shields the original camera images that cause holes and large errors from participating in the image reconstruction process.

[0073] Therefore, in this embodiment, after determining the pixel region to be masked in the depth map based on the edge pixels of each depth map, not every pixel in the depth map can be mapped. This embodiment marks the pixels in the pixel region to be masked. The marked pixels may be pixels in the image captured by the original camera close to the virtual viewpoint, so that the mapping operation is only performed on the marked pixels in the depth map of the target viewpoint.

[0074] As an optional implementation, for original viewpoints other than the target viewpoint, no mapping operation is performed on the marked pixel points.

[0075] In this embodiment, for the original viewpoints other than the target viewpoint among the multiple original viewpoints, even if they have marked pixels, mapping operations may not be performed on them, that is, the mapping of the original viewpoints other than the target viewpoint among the multiple original viewpoints is shielded. For example, there are M target viewpoints closest to the virtual viewpoint position, and no mapping operations are performed on them.

[0076] As an optional implementation, when performing image mapping on the area in the depth map except the pixel area to be shielded, the depth maps of all original viewpoints are used for reconstruction.

[0077] In this embodiment, during the process of performing virtual viewpoint mapping on the depth map, image mapping may be performed on areas other than the pixel region to be shielded in the depth map, wherein the pixels in the areas other than the pixel region to be shielded in the depth map are unlabeled pixels. When performing image mapping on the unlabeled pixels in the areas other than the pixel region to be shielded in the depth map, the depth maps of all original viewpoints may be used for reconstruction.

[0078] As an optional implementation, edge pixels are used as reference points, a predetermined number of pixels are extracted in the foreground direction, and written into a pixel set to be shielded, wherein the pixel area to be shielded includes the pixel set.

[0079] In this embodiment, after edge detection is performed on the depth map of each original viewpoint and the edge pixel points of each depth map are obtained, a predetermined number of pixel points can be extracted in the foreground direction with the edge pixel points of the depth map as the reference point. For example, a predetermined number of pixel points in the same row of the edge pixel points along the foreground direction are marked, and the marked predetermined number of pixel points are determined as the above-mentioned predetermined number of pixel points extracted in the foreground direction, wherein the predetermined number is a parameter that can be set, for example, 1, 2, 3, 4, 5, etc.

[0080] As an optional implementation, the number of pixels to be extracted in the foreground direction is determined based on the loss degree of the depth map, wherein the loss degree of the depth map is proportional to the number of pixels to be extracted.

[0081] In this embodiment, when determining the number of pixels that need to be extracted in the foreground direction, the degree of loss of the depth map can be determined first, and the degree of loss of the edges of the depth map caused by compression, transmission, etc. can be determined. The degree of loss of the depth map in this embodiment is proportional to the number of extracted pixels, that is, the greater the degree of loss of the depth map, the more pixels are extracted, and the smaller the degree of loss of the depth map, the smaller the number of extracted pixels. Optionally, this embodiment determines the number of pixels to be extracted in the foreground direction as the predetermined number of pixels written into the set of pixels to be shielded.

[0082] As an optional implementation, in step S202, before performing edge detection on the depth map of the original viewpoint, the method further includes: determining the position of the virtual viewpoint to be mapped; marking the first set of original viewpoints whose distance from the virtual viewpoint position exceeds a predetermined threshold as original viewpoints on which edge detection needs to be performed; and marking the second set of original viewpoints whose distance from the virtual viewpoint position does not exceed the predetermined threshold as original viewpoints on which edge detection does not need to be performed.

[0083] In this embodiment, before performing edge detection on the depth map of the original viewpoint, the position of the virtual viewpoint to be mapped can be determined first, which can be a 6DoF virtual viewpoint position. Then, a first number of pixel points whose distance from the virtual viewpoint position exceeds a predetermined threshold are determined, and the first number of pixel points are combined into a first original viewpoint set, wherein the predetermined threshold is a critical value for measuring the distance from the virtual viewpoint position, and the first original viewpoint set can be marked as the original viewpoint on which edge detection needs to be performed.

[0084] Optionally, this embodiment obtains a second number of pixels whose distance from the virtual viewpoint position does not exceed a predetermined threshold. This second number of pixels can be combined into a second set of original viewpoints and marked as original viewpoints for which edge detection is not required. In other words, the second set of original viewpoints that are closer to the virtual viewpoint position can still be mapped without being affected by the marked original viewpoints for which edge detection is required. The second number is a configurable threshold, which can be represented by M.

[0085] As an optional implementation, in the process of performing virtual viewpoint mapping on the depth map, pixel points other than the pixel area to be shielded are mapped, and values ​​are taken from the image of the first original camera based on the newly mapped depth map to generate a first reconstructed image.

[0086] In this embodiment, in the process of mapping the depth map to the virtual viewpoint position, image mapping can be performed on the pixel area to be shielded in the depth map of the target viewpoint. When mapping the pixel points other than the pixel area to be shielded, the pixel points in the depth map other than the pixel area to be shielded can be mapped to the new 6DoF virtual viewpoint position according to the spatial geometric relationship, thereby obtaining a newly mapped depth map. The newly mapped depth map is post-processed, and values ​​are taken from the image of the first original camera based on the newly mapped depth map to generate a first reconstructed image, wherein the first original camera can be the image of the first camera among the images of N cameras. In this way, a reconstructed image of the new virtual viewpoint is formed. Since the reconstructed image is obtained from the first camera image and the depth map, it can be referred to as P1.

[0087] As an optional embodiment, after generating the first reconstructed image, the method also includes: performing image reconstruction of other virtual perspectives on the depth map to obtain multiple reconstructed images, wherein the multiple virtual perspectives are determined based on cameras arranged at different perspective positions; and fusing the first reconstructed image and the multiple reconstructed images to generate a reconstruction result of the depth map.

[0088] In this embodiment, after generating the above-mentioned first reconstructed image, the depth map can also be reconstructed from other virtual perspectives in addition to the virtual perspective corresponding to the above-mentioned virtual viewpoint, so that multiple reconstructed images can be obtained, wherein multiple virtual perspectives can be determined by cameras arranged at different perspective positions. Optionally, this embodiment processes the images of N cameras in the same manner as the image of the above-mentioned first original camera, thereby obtaining N reconstructed images P1, P2...PN.

[0089] After reconstructing the depth map from other virtual perspectives to obtain multiple reconstructed images, the first reconstructed image and the multiple reconstructed images can be fused, which can be done by weighted averaging the pixels of the first reconstructed image and the pixels of the multiple reconstructed images, as well as performing a hole filling algorithm. Due to the occlusion relationship of the depth map, in this embodiment, the image pixels mapped in one reconstructed image may not be mapped in other reconstructed images among the multiple reconstructed images. For example, the image pixels mapped in reconstructed image P1 may not be mapped in reconstructed image P2. Alternatively, the image pixels mapped in reconstructed images P1 and P2 are determined, and then weighted averaged to obtain the final image pixels, which are determined as the reconstruction result of the depth map.

[0090] Optionally, when fusing the first reconstructed image and multiple reconstructed images to generate a reconstructed result of a depth map, for a certain pixel position (x, y) in the depth map, all m pixels with mapping values ​​at the pixel position (x, y) can be obtained in the reconstructed images P1, P2...PN (if the images of all cameras have no value, the position (x, y) can be marked as a hole pixel, and the next step is performed after the above steps are completed for these pixels. For the m pixels obtained (m!=0), a weighted average is performed on these m pixels to obtain a final value, wherein the weighted weight can be a simple average, or weighted according to the inverse of the distance between the virtual viewpoint position and the position of each camera, wherein the closer the camera position is, the greater the corresponding weight.

[0091] Optionally, the next step is to interpolate pixel positions (x, y) that have no values ​​in all reconstructed images P1, P2...PN from surrounding pixels using a hole filling algorithm.

[0092] The embodiment of the present invention further provides another depth map reconstruction method.

[0093] Figure 3 FIG. 1 is a flow chart of another depth map reconstruction method according to an embodiment of the present invention. Figure 3 As shown, the method may include the following steps:

[0094] Step S302: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixels of each depth map.

[0095] In the technical solution provided in step S302 of the present invention, there may be multiple original viewpoints, each of which has a corresponding depth map, which may be a depth map for compression and transmission, to obtain depth maps of multiple original viewpoints.

[0096] During the compression and transmission process of the depth map of this embodiment, the edge of the depth map will suffer a certain loss. This embodiment can perform edge detection on the depth map of the original viewpoint, so as to obtain the lost pixel points at the edge of the depth map, which can be pixel points facing the foreground direction based on the edge pixel points. There can be multiple pixel points, which form the pixel area to be shielded.

[0097] Optionally, in this embodiment, edge detection is performed on the depth map of each original viewpoint. The edge detection can be performed by simply comparing |Depth_left-Depth_right|>THR.

[0098] Step S304 : in the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint.

[0099] In the technical solution provided in the above step S304 of the present invention, after edge detection is performed on the depth map of each original viewpoint and the lost pixels in each depth map are obtained, in the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among multiple original viewpoints that meets the mapping conditions.

[0100] In this embodiment, the edge of the depth map suffers a certain loss due to compression and transmission, and changes from the original large value of the foreground to a smaller value. Such a depth map will cause the foreground to fly out when mapped to the virtual viewpoint, resulting in the generation of foreground mapping texture in the original hole area. In the process of mapping the depth map to the virtual viewpoint, this embodiment performs image mapping on the pixel area to be shielded in the depth map of the target viewpoint. The process of mapping the depth map to the virtual viewpoint is also a virtual viewpoint reconstruction process based on the depth map. For example, the pixel area to be shielded includes the pixel points in the depth map that are relatively close to the virtual viewpoint, and image mapping is performed on them, while image mapping is prohibited for the pixel points in the depth map that are relatively far from the virtual viewpoint, thereby ensuring that the DIBR algorithm is robust against compression loss and significantly improving the difference quality of the foreground edge of the depth map.

[0101] As an optional implementation, in step S302, after performing edge detection on the depth map of each original viewpoint and obtaining the lost pixels in each depth map, the method further includes: determining the pixel area to be shielded in the depth map based on the edge pixels of each depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point.

[0102] This embodiment, after implementing edge detection on the depth map of each original viewpoint and obtaining the lost pixels in each depth map, can determine the pixel area to be shielded in the depth map based on the edge pixels of each depth map. The pixel area to be shielded includes at least one pixel point in the foreground direction with the edge pixel point as a reference, which can be at least one pre-marked pixel point in the foreground direction with the edge pixel point as a reference, for example, at least one pixel point of the depth map edge after compression except for the pixel value that causes loss, so as to ensure that the DIBR algorithm is robust against compression loss.

[0103] The embodiment of the present invention further provides another depth map reconstruction method.

[0104] Figure 4FIG. 1 is a flowchart of a depth map reconstruction method according to an embodiment of the present invention. Figure 4 As shown, the method may include the following steps:

[0105] Step S402: display depth maps of multiple original viewpoints.

[0106] In the technical solution provided in the above step S402 of the present invention, there can be multiple original viewpoints, each original viewpoint has a corresponding depth map, which can be a depth map for compression and transmission. Optionally, the depth maps of the multiple original viewpoints obtained are displayed on a graphical user interface.

[0107] Step S404: Display edge pixel points extracted after edge detection on each depth map.

[0108] In the technical solution provided in the above step S404 of the present invention, after displaying the depth maps of multiple original viewpoints, edge pixels extracted after edge detection is performed on each depth map may be displayed on a graphical user interface.

[0109] In this embodiment, edge detection is performed on the depth map of the original viewpoint to detect edge pixels in the depth map of the original viewpoint, and the edge pixels extracted after edge detection on each depth map are displayed on the graphical user interface.

[0110] Step S406 , displaying an image result after image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint during the process of performing virtual viewpoint mapping on the depth map.

[0111] In the technical solution provided in the above-mentioned step S406 of the present invention, after displaying the edge pixel points extracted after edge detection on each depth map, the image result after performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint can be displayed. The target viewpoint is a viewpoint that meets the mapping conditions among multiple original viewpoints, and the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point.

[0112] In this embodiment, the depth map suffers a certain loss in its edge due to compression and transmission, changing from the original larger value of the foreground to a smaller value. Such a depth map, when mapped to a virtual viewpoint, will cause the foreground to fly out, thereby resulting in the generation of foreground mapping texture in the original hole area. This embodiment can determine the pixel area to be shielded in the depth map based on the edge pixel points of each depth map. The pixel area to be shielded includes a set of pixel points. The multiple pixel point sets can be at least one pixel point pre-marked with the edge pixel point as the reference toward the foreground direction, for example, the pixel points other than the pixel points that are lost after the edge of the depth map is compressed. The other pixel points in the depth map except the pixel area to be shielded can be reconstructed using the depth map of all original viewpoints when performing the mapping operation. Therefore, this embodiment can perform image mapping on the pixel area to be shielded in the depth map of the target viewpoint during the process of virtual viewpoint mapping the depth map, thereby obtaining an image result after the depth map is mapped to the virtual viewpoint. The image result can be the mapping result of the depth map being mapped to the virtual viewpoint, which does not contain the pixel area to be shielded, and then the above image result is displayed on the graphical user interface.

[0113] As an optional implementation, an embodiment of the present invention also provides another depth map reconstruction method. The method may include: determining a target viewpoint from multiple original viewpoints, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints; obtaining a depth map of the target viewpoint, performing edge detection on the depth map of the target viewpoint, and obtaining edge pixel points of the depth map of the target viewpoint; determining a pixel area to be shielded based on the edge pixel points of the depth map of the target viewpoint, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point; in the process of performing virtual viewpoint mapping on the depth maps of the multiple original viewpoints, selecting the pixel area to be shielded in the depth map of the target viewpoint to perform image mapping.

[0114] In this embodiment, each original viewpoint has a corresponding depth map, and the depth maps of multiple original viewpoints are obtained. The target viewpoint is determined from the multiple original viewpoints. The target viewpoint is the viewpoint that meets the mapping conditions among the multiple original viewpoints, so that this embodiment can only perform edge detection on the depth map of the target viewpoint.

[0115] This embodiment performs edge detection on the depth map of the target viewpoint, which can be used to detect pixel points at the edge of the depth map of the target viewpoint, that is, edge pixel points of the depth map, thereby obtaining edge pixel points of the depth map of the target viewpoint.

[0116] Optionally, in this embodiment, edge detection is performed on the depth map of the target viewpoint. The edge detection can be performed by simply comparing |Depth_left-Depth_right|>THR.

[0117] In this embodiment, the depth map of the target viewpoint has undergone compression and transmission, and its edges have suffered certain losses, changing from the original large value of the foreground to a smaller value. When such a depth map of the target viewpoint is mapped to the virtual viewpoint, it will cause the foreground to fly out, resulting in the generation of a foreground mapping texture in the original hole area. However, this embodiment performs edge detection on the depth map of the target viewpoint, obtains the edge pixel points of the target viewpoint, and then determines the pixel area to be shielded in the depth map of the target viewpoint based on the edge pixel points of the depth map of the target viewpoint. The pixel area to be shielded includes a pixel set, which includes at least one pixel point facing the foreground direction with the edge pixel point as the reference.

[0118] In the process of performing virtual viewpoint mapping on the depth maps of multiple original viewpoints, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint. For example, the pixel area to be shielded includes the pixel points in the depth map of the target viewpoint that are relatively close to the virtual viewpoint, and image mapping is performed on them, while image mapping is prohibited for the pixel points in the depth map of the target viewpoint that are relatively far from the virtual viewpoint, thereby ensuring that the DIBR algorithm is robust against compression loss and significantly improving the difference quality of the foreground edge of the depth map.

[0119] As an optional implementation, an embodiment of the present invention also provides another method for reconstructing a depth map. The method may include: during a live broadcast, obtaining depth maps of multiple original viewpoints on the live broadcast screen; performing edge detection on the depth map of each original viewpoint to obtain edge pixel points of each depth map; based on the edge pixel points of each depth map, determining the pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction with the edge pixel point as a reference; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0120] The depth map reconstruction method of this embodiment can be applied in transaction scenarios, such as in a shopping live broadcast scenario. The live broadcast scenario of this embodiment has multiple original viewpoints, each of which has a corresponding depth map, and the depth maps of the multiple original viewpoints are obtained.

[0121] This embodiment performs edge detection on the depth map of each original viewpoint in the live scene, which can be used to detect pixel points at the edge in the depth map of the original viewpoint, thereby obtaining edge pixel points of each depth map.

[0122] Optionally, in this embodiment, edge detection is performed on the depth map of each original viewpoint in the live scene. The edge detection can be performed by simply comparing |Depth_left-Depth_right|>THR.

[0123] In this embodiment, the depth map in the live scene has suffered certain edge loss due to compression and transmission, and has changed from the original large foreground value to a smaller value. Such a depth map will cause the foreground to fly out when mapped to the virtual viewpoint in the live scene, resulting in the generation of foreground mapping texture in the original hole area. However, this embodiment obtains depth maps of multiple original viewpoints in the live scene, performs edge detection on the depth map of each original viewpoint, and obtains the edge pixel points of each depth map. Then, based on the edge pixel points of each depth map in the live scene, it can determine the pixel area to be shielded in the depth map. The pixel area to be shielded includes a set of pixel points. The set of multiple pixel points can be at least one pre-marked pixel point facing the foreground direction with the edge pixel point as the reference.

[0124] In this embodiment, in a live broadcast scene, the process of performing virtual viewpoint mapping refers to the process of mapping the depth map to the virtual viewpoint position of the target, that is, based on the virtual viewpoint reconstruction process of the depth map in the live broadcast scene, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint. For example, the pixel area to be shielded includes pixels in the depth map that are relatively close to the virtual viewpoint, and image mapping is performed on them, while image mapping is prohibited for pixels in the depth map that are relatively far from the virtual viewpoint. The target viewpoint is a viewpoint that meets the mapping conditions among multiple original viewpoints, that is, a viewpoint used to realize virtual viewpoint mapping of the depth map of the live broadcast scene, thereby ensuring that the DIBR algorithm is robust against compression loss, and significantly improving the difference quality of the foreground edge of the depth map.

[0125] As an optional implementation, an embodiment of the present invention also provides another method for reconstructing a depth map. The method may include: obtaining depth maps of multiple original viewpoints, selecting an image of a predetermined area in the depth map of each original viewpoint, and obtaining a pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction with an edge pixel point as a reference; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0126] In this embodiment, there may be multiple original viewpoints, each of which has a corresponding depth map. To obtain depth maps for multiple original viewpoints, a portion of the depth map of each original viewpoint may be selected for mapping. In this embodiment, an image of a predetermined region in the depth map of each original viewpoint may be selected. Edge detection may be performed on the depth map of each original viewpoint in the live broadcast scene to obtain an image of the predetermined region in the depth map of each original viewpoint, which is then determined as the pixel region to be masked in the depth map.

[0127] In this embodiment, the depth map suffers some edge loss due to compression and transmission, changing from a larger foreground value to a smaller value. When such a depth map is mapped to a virtual viewpoint, it causes the foreground to pop out, resulting in a foreground mapping texture in the original hole area. This embodiment obtains depth maps of multiple original viewpoints, selects an image of a predetermined area in the depth map of each original viewpoint, and obtains a pixel area to be shielded in the depth map. The pixel area to be shielded includes a set of pixel points. The set of multiple pixel points can be at least one pre-marked pixel point facing the foreground with an edge pixel point as a reference.

[0128] In this embodiment, the process of performing virtual viewpoint mapping refers to a virtual viewpoint reconstruction process based on the depth map in the live scene, and image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint. For example, the pixel area to be shielded includes pixels in the depth map that are relatively close to the virtual viewpoint, and image mapping is performed on them, while image mapping is prohibited for pixels in the depth map that are relatively far from the virtual viewpoint. The target viewpoint is a viewpoint among multiple original viewpoints that meets the mapping conditions, that is, a viewpoint used to realize virtual viewpoint mapping of the depth map of the live scene, thereby ensuring that the DIBR algorithm is robust against compression loss, and significantly improving the difference quality of the foreground edge of the depth map.

[0129] As an optional implementation manner, this embodiment may perform edge detection on the depth map of the original viewpoint to determine that edge pixels in the depth map constitute the predetermined area.

[0130] In this embodiment, edge detection is performed on the depth map of the original viewpoint. This edge detection can be used to detect edge pixels in the depth map of the original viewpoint, thereby obtaining edge pixels of each depth map. Based on the edge pixels of each depth map, this embodiment can form a predetermined area of ​​this embodiment using the edge pixels, thereby achieving the purpose of selecting a portion of the depth map of the original viewpoint for mapping.

[0131] This embodiment masks pixel values ​​lost after the depth map edge is compressed or transmitted during the DIBR mapping process, thereby ensuring that the DIBR algorithm is robust against compression loss. Therefore, the method of this embodiment is a DIBR method robust against depth map compression loss. For the depth map edge loss caused by compression or transmission, a secondary viewpoint depth map mapping masking strategy is adopted. Therefore, compared with other algorithms in the relevant technology, this embodiment can significantly improve the interpolation quality of the foreground edge of the depth map, solves the technical problem that the original hole area generates the foreground mapping texture during the virtual viewpoint reconstruction of the depth map, and achieves the technical effect that the original hole area does not generate the foreground mapping texture during the virtual viewpoint reconstruction of the depth map.

[0132] Example 2

[0133] The embodiment of the present invention further provides a schematic diagram of a depth map reconstruction system. It should be noted that the depth map reconstruction system of this embodiment can be used to execute the depth map reconstruction method of embodiment 1 of the present invention.

[0134] Figure 5 FIG is a schematic diagram of a depth map reconstruction system according to an embodiment of the present invention. Figure 5 As shown, the depth map reconstruction system 50 may include: a client 51 and a cloud server 52.

[0135] The client 51 is configured to display depth images of multiple original viewpoints.

[0136] In this embodiment, there can be multiple original viewpoints, each of which has a corresponding depth map, which can be a depth map for compression and transmission. Optionally, the client 51 of this embodiment includes a graphical interface, and the depth map can be displayed on the above-mentioned graphical user interface.

[0137] The cloud server 52 communicates with the client 51 and is used to obtain the depth map of the original viewpoint. After performing edge detection on the depth map of each original viewpoint to generate edge pixel points, the pixel area to be shielded in the depth map is determined, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point.

[0138] The depth map reconstruction system of this embodiment may further include a cloud server 52, which communicates with the client 51 to obtain the depth map of the original viewpoint sent by the client 51, and can be used to perform edge detection on the depth map of the original viewpoint, detect the pixel points at the edge of the depth map of the original viewpoint, and determine the pixel area to be shielded in the depth map based on the edge pixel points of each depth map. The pixel area to be shielded includes multiple pixel points that are not mapped, and can be pre-marked pixel points facing the foreground direction with the edge pixel points as the reference, while other pixel points in the depth map except the pixel area to be shielded can be mapped.

[0139] In the process of mapping the depth map to a virtual viewpoint, the cloud server 52 performs image mapping on the pixel area to be masked in the depth map of the target viewpoint, and returns the reconstructed image generated based on the mapping result to the client, where the target viewpoint is a viewpoint among multiple original viewpoints that meets the mapping conditions.

[0140] In this embodiment, the cloud server 52 can perform image mapping on the pixel area to be masked in the depth map of the target viewpoint during the process of mapping the depth map to a virtual viewpoint, thereby obtaining a mapping result of mapping the depth map to a virtual viewpoint, and generating a reconstructed image based on the mapping result. The cloud server 52 can then return the generated reconstructed image to the client 51.

[0141] In this embodiment, the depth map of the original viewpoint is displayed by the client 51; the depth map of the original viewpoint sent by the client 51 is obtained by the cloud server 52, and after edge detection is performed on the depth map of each original viewpoint to generate edge pixels, the pixel area to be shielded in the depth map is determined, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction with the edge pixel point as a reference; wherein, in the process of virtual viewpoint mapping of the depth map, the cloud server 52 performs image mapping on the pixel area to be shielded in the depth map of the target viewpoint, and returns the reconstructed image generated based on the mapping result to the client 51, wherein the target viewpoint is a viewpoint that meets the mapping conditions among multiple original viewpoints. That is to say, after obtaining the edge pixel points of the depth map, the client of this embodiment performs image mapping on the pixel area to be shielded in the depth map during the process of virtual viewpoint mapping of the depth map, thereby ensuring that the virtual viewpoint reconstruction based on the depth map is robust against compression loss, improving the interpolation quality of the foreground edge of the depth map, and solving the technical problem that the original hole area produces the foreground mapping texture during the virtual viewpoint reconstruction of the depth map, thereby achieving the technical effect that the original hole area does not produce the foreground mapping texture during the virtual viewpoint reconstruction of the depth map.

[0142] Example 3

[0143] The preferred implementation of the above method of this embodiment is further introduced below with examples.

[0144] In related technologies, the DIBR algorithm is designed based on the assumption of a correct depth map, and does not consider optimizing for depth map loss during compression and transmission. The DIBR algorithm process is as follows:

[0145] 1) Based on spatial geometry, the depth map is mapped to the new 6DoF virtual viewpoint. This newly mapped depth map is then post-processed and used to determine values ​​in the original camera image. This creates a reconstructed image of the new virtual viewpoint. Since this reconstructed image is derived from the first camera image and the depth map, it can be referred to as reconstructed image P1. The same operation is performed on all N camera images, resulting in reconstructed images P1, P2, ..., PN.

[0146] 2) For the reconstructed images P1, P2, ..., PN, these N reconstructed images are fused, mainly including weighted pixel averaging and hole filling algorithms. Due to occlusion relationships in the depth map, the image pixels mapped in the reconstructed image P1 may not be mapped in the reconstructed image P2. Alternatively, after the reconstructed images P1 and P2 are mapped, the mapped image pixels are weighted averaged to obtain the final image pixels. This can be achieved through the following methods:

[0147] In step a, for a certain pixel position (x, y) in the image, obtain all m pixels with mapped values ​​at the pixel position (x, y) in P1, P2...PN. (If the images of all cameras have no value, the pixel position (x, y) can be marked as a hole pixel. After completing step a for all image pixel positions, you can jump to step b.)

[0148] Among them, for the m pixels obtained (m!=0), these m pixels are weighted averaged to obtain the final value. The weighted weight can typically be a simple average, or weighted according to the inverse of the distance between the virtual viewpoint position and the position of each camera (the closer the camera is, the greater the weight).

[0149] In step b, for positions (x, y) that have no values ​​in all reconstructed images P1, P2, ..., PN, a hole filling algorithm may be used to interpolate values ​​from surrounding pixels.

[0150] In the above methods, any pixel of the original image can be mapped to the virtual viewpoint position of the target, thus filling the hole. However, the assumption here is based on the premise that the depth map of the original viewpoint is sufficiently accurate. Usually, there will be some compression loss at the edge of the depth map after compression or transmission, such as Figure 6 As shown, Figure 6 is a schematic diagram of a depth map loss according to an embodiment of the present invention. Figure 6 In the process of compression and transmission, the edges of the depth map will suffer certain losses. For example, the values ​​of boxes 1, 2, and 3 change from the original large foreground values ​​to smaller values. When such a depth map is mapped to the virtual viewpoint position, the foreground will appear to fly out, resulting in the generation of foreground mapping textures in the original empty area. Figure 7 As shown, Figure 7 FIG. 1 is a schematic diagram showing the effect of compression loss of a depth map on DIBR quality according to an embodiment of the present invention. Figure 7 In the example, the DIBR logic needs to be optimized for the images in boxes 4, 5, 6, and 7 so that the DIBR algorithm is robust to the compression loss of the depth map.

[0151] This embodiment further proposes the following algorithm to address the above problem.

[0152] This embodiment maps the depth map to a new 6DoF virtual viewpoint based on spatial geometry. The newly mapped depth map is then post-processed, and values ​​are taken from the original camera image based on the newly mapped depth map. This creates a reconstructed image of the new virtual viewpoint. Since this reconstructed image is derived from the first camera image and the depth map, it can be referred to as reconstructed image P1. The same operation is performed on all N camera images, resulting in reconstructed images P1, P2, ..., PN.

[0153] In this embodiment, the following additional logic may be added during the depth map mapping process:

[0154] Step a: perform edge detection on the depth map of each original viewpoint. Edge detection can be performed by simply comparing |Depth_left-Depth_right|>THR. For pixels detected at the edge, P pixels in the same row along the foreground direction of the edge pixel can be further marked (P is a configurable parameter = 1, 2, 3, 4, 5...) as shown in the following example: Figure 6 shown.

[0155] In step b, the M original viewpoints (M is a configurable threshold) closest to the virtual viewpoint can be mapped according to the original method without being affected by the marked pixels detected in step a.

[0156] In step c, for the remaining NM original viewpoints in the depth map, the pixels marked in step a are masked from mapping (ie, these pixels are not mapped), and the remaining unmarked pixels are still mapped according to the original method.

[0157] The remaining steps can be processed according to the original method.

[0158] Figure 8 FIG. 1 is a schematic diagram of edge detection and edge mapping shielding of a depth map according to an embodiment of the present invention. Figure 8 As shown, the N reconstructed images P1, P2...PN are fused, which mainly includes the weighted average of pixels and the hole filling algorithm. Due to the occlusion relationship of the depth map, the image pixels mapped in the reconstructed image P1 may not be mapped in the reconstructed image P2. Alternatively, after the reconstructed images P1 and P2 are mapped, the mapped image pixels are weighted averaged to obtain the final pixel. Optionally, this can be achieved by the following method:

[0159] In step a, for a certain pixel position (x, y) in the image, obtain all m pixels with mapped values ​​at the pixel position (x, y) in the reconstructed images P1, P2...PN. (If the images of all cameras have no value, the pixel position (x, y) can be marked as a hole pixel. After executing step a for all image pixel positions, jump to step b.)

[0160] Among them, for the m pixels obtained (m!=0), these m pixels are weighted averaged to obtain the final value. The weighted weight can typically be a simple average, or weighted according to the inverse of the distance between the virtual viewpoint position and the position of each camera (the closer the camera is, the greater the weight).

[0161] In step b, for pixel positions (x, y) that have no values ​​in all reconstructed images P1, P2, ..., PN, a hole filling algorithm may be used to interpolate values ​​from surrounding pixels that have already been obtained.

[0162] Figure 9 FIG. 1 is a schematic diagram of an interpolation result according to an embodiment of the present invention. Figure 9 As shown in FIG, by masking the pixel values ​​lost after the edge of the foreground depth map is compressed during the DIBR mapping process, the DIBR algorithm is guaranteed to be robust against compression loss.

[0163] This embodiment proposes a DIBR method that is robust to depth map compression loss. For the depth map edge loss caused by compression, a secondary viewpoint depth map mapping shielding strategy is adopted. Therefore, compared with other algorithms in the relevant technology, the interpolation quality of the foreground edge of this embodiment is significantly improved, and the technical problem of generating foreground mapping texture in the original hole area during the virtual viewpoint reconstruction of the depth map is solved. The technical effect of not generating foreground mapping texture in the original hole area during the virtual viewpoint reconstruction of the depth map is achieved.

[0164] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0165] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0166] Example 4

[0167] According to an embodiment of the present invention, a depth map reconstruction device for implementing the above-mentioned depth map reconstruction method is also provided. It should be noted that the depth map reconstruction device of this embodiment can be used to execute the depth map reconstruction method of the embodiment of the present invention.

[0168] Figure 10 FIG is a schematic diagram of a depth map reconstruction device according to an embodiment of the present invention. Figure 10 As shown, the depth map reconstruction device 100 may include: an acquisition unit 101 , a determination unit 102 and a prohibition unit 103 .

[0169] The acquisition unit 101 is configured to acquire depth maps of multiple original viewpoints, perform edge detection on the depth map of each original viewpoint, and acquire edge pixels of each depth map.

[0170] The determining unit 102 is configured to determine a pixel region to be shielded in the depth map based on edge pixels of each depth map, wherein the pixel region to be shielded includes at least one pixel in a direction toward the foreground with the edge pixel as a reference.

[0171] The prohibition unit 103 is configured to perform image mapping on a pixel region to be shielded in the depth map of a target viewpoint during virtual viewpoint mapping of the depth map, wherein the target viewpoint is a viewpoint that meets the mapping condition among the multiple original viewpoints.

[0172] It should be noted that the acquisition unit 101, determination unit 102, and prohibition unit 103 described above correspond to steps S202 to S206 in Example 1. The examples and application scenarios implemented by the three units and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above units, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0173] Figure 11 FIG. 1 is a schematic diagram of another apparatus for reconstructing a depth map according to an embodiment of the present invention. Figure 11 As shown, the depth map reconstruction device 110 may include: a first acquisition unit 111 and a mapping unit 112.

[0174] The first acquisition unit 111 is configured to acquire depth maps of multiple original viewpoints, perform edge detection on the depth map of each original viewpoint, and acquire edge pixels of each depth map.

[0175] The mapping unit 112 is configured to perform image mapping on a pixel region to be masked in the depth map of a target viewpoint during virtual viewpoint mapping of the depth map, wherein the target viewpoint is a viewpoint among multiple original viewpoints that meets the mapping condition.

[0176] It should be noted that the first acquisition unit 111 and the first prohibition unit 112 correspond to steps S302 and S304 in Example 1. The examples and application scenarios implemented by the two units and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above units, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0177] Figure 12 FIG. 1 is a schematic diagram of another apparatus for reconstructing a depth map according to an embodiment of the present invention. Figure 12 As shown, the depth map reconstruction device 120 may include: a first display unit 121 , a presentation unit 122 and a second display unit 122 .

[0178] The first display unit 121 is configured to display depth maps of multiple original viewpoints.

[0179] The display unit 122 is used to display edge pixels extracted after edge detection is performed on each depth map.

[0180] The second display unit 122 is used to display the image result after the depth map is mapped to a virtual viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among multiple original viewpoints, and the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point.

[0181] It should be noted that the first display unit 121, the display unit 122, and the second display unit 123 correspond to steps S402 and S406 in Example 1. The examples and application scenarios implemented by the three units and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above units, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0182] In the depth map reconstruction device of this embodiment, after obtaining the edge pixel points of the depth map, in the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map, thereby ensuring that the virtual viewpoint reconstruction based on the depth map is robust against compression loss, improving the interpolation quality of the foreground edge of the depth map, and solving the technical problem that the original hole area generates foreground mapping texture during the virtual viewpoint reconstruction of the depth map, thereby achieving the technical effect that the original hole area does not generate foreground mapping texture during the virtual viewpoint reconstruction of the depth map.

[0183] Example 5

[0184] Embodiments of the present invention may provide a depth map reconstruction system, which may include a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may be replaced by a terminal device such as a mobile terminal.

[0185] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0186] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the depth map reconstruction method of the application: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixel points of each depth map; based on the edge pixel points of each depth map, determining the pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point; in the process of virtual viewpoint mapping the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints.

[0187] Optionally, Figure 13 1 is a block diagram of a computer terminal according to an embodiment of the present invention. Figure 13 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 132 , a memory 134 , and a transmission device 136 .

[0188] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the depth map reconstruction method and apparatus in the embodiments of the present invention. The processor executes the software programs and modules stored in the memory to perform various functional applications and data processing, thereby implementing the depth map reconstruction method described above. The memory can include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal A via a network. Examples of the aforementioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0189] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain depth maps of multiple original viewpoints, perform edge detection on the depth map of each original viewpoint, and obtain edge pixel points of each depth map; based on the edge pixel points of each depth map, determine the pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point; in the process of virtual viewpoint mapping the depth map, perform image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints.

[0190] Optionally, the processor may also execute the following program code: taking edge pixels as reference points, extracting a predetermined number of pixels in the foreground direction, and writing them into a set of pixels to be shielded, wherein the pixel area to be shielded includes the set of pixels.

[0191] Optionally, the processor may further execute program code of the following steps: marking pixel points in the pixel area to be shielded, wherein the marked pixel point indicates that the virtual viewpoint mapping operation is not performed on the pixel point.

[0192] Optionally, the processor may further execute program code of the following steps: determining the number of pixels extracted in the foreground direction based on the degree of loss of the depth map, wherein the degree of loss of the depth map is proportional to the number of pixels extracted.

[0193] Optionally, the processor may also execute the program code of the following steps: determining the position of the virtual viewpoint to be mapped before performing edge detection on the depth map of the original viewpoint; marking the first set of original viewpoints whose distance from the virtual viewpoint position exceeds a predetermined threshold as original viewpoints on which edge detection needs to be performed; and marking the second set of original viewpoints whose distance from the virtual viewpoint position does not exceed the predetermined threshold as original viewpoints on which edge detection does not need to be performed.

[0194] Optionally, the processor may also execute the program code of the following steps: in the process of performing virtual viewpoint mapping on the depth map, mapping the pixel points except the pixel area to be shielded, and taking values ​​from the image of the first original camera based on the newly mapped depth map to generate a first reconstructed image.

[0195] Optionally, the processor may also execute the program code of the following steps: after generating the first reconstructed image, reconstructing the depth map from other virtual perspectives to obtain multiple reconstructed images, wherein the multiple virtual perspectives are determined based on cameras arranged at different perspective positions; and fusing the first reconstructed image and the multiple reconstructed images to generate a reconstruction result of the depth map.

[0196] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain depth maps of multiple original viewpoints, perform edge detection on the depth map of each original viewpoint, and obtain edge pixel points of each depth map; in the process of mapping the depth map to a virtual viewpoint, perform image mapping on the pixel area to be masked in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0197] Optionally, the processor may also execute the following program code: based on the edge pixel points of each depth map, determine the pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point.

[0198] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: display the depth maps of multiple original viewpoints; display the edge pixel points extracted after edge detection on each depth map; and display the image result after performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint in the process of virtual viewpoint mapping the depth map, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions, and the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point.

[0199] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the target viewpoint from multiple original viewpoints, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints; obtain the depth map of the target viewpoint, perform edge detection on the depth map of the target viewpoint, and obtain the edge pixel points of the depth map of the target viewpoint; determine the pixel area to be shielded based on the edge pixel points of the depth map of the target viewpoint, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point; in the process of performing virtual viewpoint mapping on the depth maps of multiple original viewpoints, select the pixel area to be shielded in the depth map of the target viewpoint to perform image mapping.

[0200] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: during the live broadcast process, obtain depth maps of multiple original viewpoints on the live broadcast screen; perform edge detection on the depth map of each original viewpoint to obtain edge pixel points of each depth map; based on the edge pixel points of each depth map, determine the pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point; in the process of virtual viewpoint mapping the depth map, perform image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints.

[0201] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain depth maps of multiple original viewpoints, select the image of a predetermined area in the depth map of each original viewpoint, and obtain the pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point; in the process of virtual viewpoint mapping the depth map, perform image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints.

[0202] Optionally, the processor may further execute program code of the following steps: performing edge detection on the depth map of the original viewpoint to determine that edge pixels in the depth map constitute a predetermined area.

[0203] According to an embodiment of the present invention, a method for reconstructing a depth map is provided. By acquiring depth maps of multiple original viewpoints, edge detection is performed on the depth map of each original viewpoint to obtain edge pixels of each depth map; based on the edge pixels of each depth map, a pixel region to be shielded in the depth map is determined, wherein the pixel region to be shielded includes at least one pixel in the foreground direction with the edge pixel as a reference; in the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel region to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint that meets the mapping conditions among the multiple original viewpoints. After acquiring the edge pixels of the depth map, the present application prohibits mapping the pixel region to be shielded in the depth map, thereby ensuring that the virtual viewpoint reconstruction based on the depth map has robustness against compression loss, improving the interpolation quality of the foreground edge of the depth map, and solving the technical problem that the original hole area generates foreground mapping texture during the virtual viewpoint reconstruction of the depth map, thereby achieving the technical effect that the original hole area does not generate foreground mapping texture during the virtual viewpoint reconstruction of the depth map.

[0204] It can be understood by those skilled in the art that Figure 13 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 13 It does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal A may also include Figure 13 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 13 Different configurations shown.

[0205] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0206] Example 6

[0207] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the depth map reconstruction method provided in the first embodiment.

[0208] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0209] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixel points of each depth map; determining a pixel area to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction based on the edge pixel point; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0210] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: taking the edge pixel point as the reference point, extracting a predetermined number of pixel points in the foreground direction, and writing the pixel point set to be shielded, wherein the pixel area to be shielded includes the pixel point set.

[0211] Optionally, the computer-readable storage medium is further configured to store program code for executing the following steps: marking pixel points in the pixel area to be shielded, wherein the marked pixel point indicates that the virtual viewpoint mapping operation is not performed on the pixel point.

[0212] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: determining the number of pixels extracted in the foreground direction based on the degree of loss of the depth map, wherein the degree of loss of the depth map is proportional to the number of pixels extracted.

[0213] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: before performing edge detection on the depth map of the original viewpoint, determining the position of the virtual viewpoint to be mapped; marking a first set of original viewpoints whose distance from the virtual viewpoint position exceeds a predetermined threshold as original viewpoints on which edge detection needs to be performed; and marking a second set of original viewpoints whose distance from the virtual viewpoint position does not exceed the predetermined threshold as original viewpoints on which edge detection does not need to be performed.

[0214] Optionally, the computer-readable storage medium is further configured to store program code for executing the following steps: in the process of performing virtual viewpoint mapping on the depth map, mapping pixel points other than the pixel area to be shielded, and taking values ​​from the image of the first original camera based on the newly mapped depth map to generate a first reconstructed image.

[0215] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: after generating the first reconstructed image, performing image reconstructing of other virtual perspectives on the depth map to obtain multiple reconstructed images, wherein the multiple virtual perspectives are determined based on cameras arranged at different perspective positions; and fusing the first reconstructed image and the multiple reconstructed images to generate a reconstruction result of the depth map.

[0216] As an optional example, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each original viewpoint, and obtaining edge pixel points of each depth map; in the process of mapping the depth map to a virtual viewpoint, performing image mapping on the pixel area to be masked in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0217] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: determining a pixel area to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction based on the edge pixel point.

[0218] As an optional example, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: displaying depth maps of multiple original viewpoints; displaying edge pixel points extracted after edge detection on each depth map; and displaying the image result after image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint in the process of virtual viewpoint mapping the depth map, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions, and the pixel area to be shielded includes at least one pixel point map in the foreground direction based on the edge pixel point.

[0219] As an optional example, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: determining a target viewpoint from multiple original viewpoints, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that satisfies a mapping condition; obtaining a depth map of the target viewpoint, performing edge detection on the depth map of the target viewpoint, and obtaining edge pixel points of the depth map of the target viewpoint; determining a pixel area to be shielded based on the edge pixel points of the depth map of the target viewpoint, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction based on the edge pixel point; in the process of performing virtual viewpoint mapping on the depth maps of multiple original viewpoints, selecting to perform image mapping on the pixel area to be shielded in the depth map of the target viewpoint.

[0220] As an optional example, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: during a live broadcast, obtaining depth maps of multiple original viewpoints on the live broadcast screen; performing edge detection on the depth map of each original viewpoint to obtain edge pixel points of each depth map; determining a pixel area to be shielded in the depth map based on the edge pixel points of each depth map, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction based on the edge pixel point; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0221] As an optional example, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining depth maps of multiple original viewpoints, selecting an image of a predetermined area in the depth map of each original viewpoint, and obtaining a pixel area to be shielded in the depth map, wherein the pixel area to be shielded includes at least one pixel point in the foreground direction based on the edge pixel point; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints that meets the mapping conditions.

[0222] Optionally, the computer-readable storage medium is further configured to store program code for executing the following steps: performing edge detection on the depth map of the original viewpoint to determine that edge pixels in the depth map constitute a predetermined area.

[0223] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0224] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0225] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0226] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0227] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0229] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A depth map reconstruction method, comprising: Acquire depth maps of multiple original viewpoints, perform edge detection on the depth map of each original viewpoint, and obtain edge pixels of each depth map; Determining a pixel region to be shielded in the depth map based on each edge pixel point of the depth map, wherein the pixel region to be shielded includes at least one pixel point in a foreground direction with the edge pixel point as a reference; In the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints whose distance from the virtual viewpoint is within a predetermined range.

2. The method according to claim 1, wherein Pixel points within the pixel area to be shielded are marked, wherein a mapping operation is performed only on the marked pixel points in the depth map of the target viewpoint.

3. The method according to claim 2, wherein: For original viewpoints other than the target viewpoint, no mapping operation is performed on the marked pixel points.

4. The method according to claim 1, wherein When image mapping is performed on the area in the depth map except the pixel area to be shielded, the depth maps of all original viewpoints are used for reconstruction.

5. The method according to claim 1, wherein Taking the edge pixel point as a reference point, a predetermined number of pixel points are extracted in a direction toward the foreground, and written into the pixel point set to be shielded, wherein the pixel area to be shielded includes the pixel point set.

6. The method according to claim 1, wherein The number of pixels to be extracted in a foreground direction is determined based on a loss degree of the depth map, wherein the loss degree of the depth map is proportional to the number of pixels to be extracted.

7. The method according to any one of claims 1 to 6, wherein: Before performing edge detection on the depth map of the original viewpoint, the method further includes: determining a position of the virtual viewpoint to be mapped; Marking a first set of original viewpoints whose distance from the virtual viewpoint exceeds a predetermined threshold as original viewpoints for which edge detection needs to be performed; A second set of original viewpoints whose distance from the virtual viewpoint position exceeds a predetermined threshold is marked as original viewpoints for which edge detection does not need to be performed.

8. The method according to claim 7, wherein: In the process of performing virtual viewpoint mapping on the depth map, pixels other than the pixel area to be shielded are mapped, and values ​​are taken from the image of the first original camera based on the newly mapped depth map to generate a first reconstructed image.

9. The method according to claim 8, wherein After generating the first reconstructed image, the method further comprises: Reconstructing images of other virtual perspectives on the depth map to obtain a plurality of reconstructed images, wherein the plurality of virtual perspectives are determined based on cameras arranged at different perspective positions; The first reconstructed image and the multiple reconstructed images are fused to generate a reconstructed result of the depth map.

10. A depth map reconstruction method, comprising: Acquire depth maps of multiple original viewpoints, perform edge detection on the depth map of each original viewpoint, and obtain edge pixels of each depth map; In the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints whose distance from the virtual viewpoint is within a predetermined range, and the pixel area to be shielded includes at least one pixel point in the foreground direction with the edge pixel point as a reference.

11. The method according to claim 10, wherein: After performing edge detection on the depth map of each original viewpoint and obtaining lost pixels in each depth map, the method further includes: Based on each edge pixel point of the depth map, a pixel area to be masked in the depth map is determined.

12. A method for reconstructing a depth map, comprising: Display depth maps for multiple original viewpoints; Display edge pixel points extracted after edge detection on each of the depth maps; An image result after image mapping is performed on a pixel area to be shielded in the depth map of a target viewpoint during virtual viewpoint mapping of the depth map is displayed, wherein the target viewpoint is a viewpoint among the multiple original viewpoints whose spacing with the virtual viewpoint is within a predetermined range, and the pixel area to be shielded includes at least one pixel point in a foreground direction with the edge pixel point as a reference.

13. A method for processing a depth map, comprising: Determining a target viewpoint from a plurality of original viewpoints, wherein the target viewpoint is a viewpoint among the plurality of original viewpoints whose distance from the virtual viewpoint is within a predetermined range; Acquire a depth map of the target viewpoint, perform edge detection on the depth map of the target viewpoint, and acquire edge pixel points of the depth map of the target viewpoint; Determining a pixel area to be shielded based on edge pixel points of the depth map of the target viewpoint, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction with the edge pixel point as a reference; In the process of performing the virtual viewpoint mapping on the depth maps of the multiple original viewpoints, image mapping is performed on the pixel area to be shielded in the depth map of the target viewpoint.

14. A method for reconstructing a depth map, comprising: During the live broadcast, obtain depth maps of multiple original viewpoints on the live broadcast screen; Performing edge detection on the depth map of each original viewpoint to obtain edge pixels of each depth map; Determining a pixel region to be shielded in the depth map based on each edge pixel point of the depth map, wherein the pixel region to be shielded includes at least one pixel point in a foreground direction with the edge pixel point as a reference; In the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of a target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints whose distance from the virtual viewpoint is within a predetermined range.

15. A method for reconstructing a depth map, comprising: Acquire depth maps of multiple original viewpoints, select an image of a predetermined area in the depth map of each original viewpoint, and obtain a pixel area to be masked in the depth map, wherein the pixel area to be masked includes at least one pixel point in a direction toward the foreground with an edge pixel point as a reference; In the process of performing virtual viewpoint mapping on the depth map, image mapping is performed on the pixel area to be shielded in the depth map of a target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints whose distance from the virtual viewpoint is within a predetermined range.

16. The method according to claim 15, wherein By performing edge detection on the depth map of the original viewpoint, it is determined that edge pixels in the depth map constitute the predetermined area.

17. A depth map reconstruction system, comprising: The client is used to display depth maps from multiple original viewpoints; a cloud server, communicating with the client, configured to obtain depth maps of the plurality of original viewpoints, and after performing edge detection on the depth map of each of the original viewpoints to generate edge pixels, determine a pixel region to be masked in the depth map, wherein the pixel region to be masked includes at least one pixel in a foreground direction relative to the edge pixel; In the process of performing virtual viewpoint mapping on the depth map, the cloud server performs image mapping on the pixel area to be shielded in the depth map of the target viewpoint, and returns the reconstructed image generated based on the mapping result to the client, wherein the target viewpoint is a viewpoint among the multiple original viewpoints whose distance from the virtual viewpoint is within a predetermined range.

18. A depth map reconstruction device, comprising: an acquiring unit, configured to acquire depth maps of a plurality of original viewpoints, perform edge detection on the depth map of each original viewpoint, and acquire edge pixels of each depth map; a determining unit, configured to determine a pixel region to be masked in the depth map based on each edge pixel point of the depth map, wherein the pixel region to be masked includes at least one pixel point in a direction toward the foreground with the edge pixel point as a reference; A prohibition unit is used to perform image mapping on the pixel area to be shielded in the depth map of a target viewpoint during virtual viewpoint mapping of the depth map, wherein the target viewpoint is a viewpoint among the multiple original viewpoints whose spacing with the virtual viewpoint is within a predetermined range.

19. A computer-readable storage medium comprising a stored program, wherein: When the program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 16.

20. A processor for running a program, wherein: When the program is executed, the method according to any one of claims 1 to 16 is executed.

21. A depth map reconstruction system, comprising: processor; A memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: obtaining depth maps of multiple original viewpoints, performing edge detection on the depth map of each of the original viewpoints, and obtaining edge pixel points of each of the depth maps; determining a pixel area to be shielded in the depth map based on the edge pixel points of each of the depth maps, wherein the pixel area to be shielded includes at least one pixel point in a foreground direction with the edge pixel point as a reference; in the process of performing virtual viewpoint mapping on the depth map, performing image mapping on the pixel area to be shielded in the depth map of a target viewpoint, wherein the target viewpoint is a viewpoint among the multiple original viewpoints whose distance from the virtual viewpoint is within a predetermined range.

Images