Depth map processing

By forward projection and depth value adjustment in multi-view video content, the problem of insufficient coherence between different depth maps is solved, the encoding efficiency and view synthesis quality of video content are improved, and the accuracy and coherence of scene point clouds are ensured. sex.

CN113545034BActive Publication Date: 2024-11-26INTERDIGITAL VC HOLDINGS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080019665.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-03-08
Filing Date
2020-02-13
Publication Date
2024-11-26
Estimated Expiration
2040-02-13

AI Technical Summary

Technical Problem

The lack of coherence between different depth maps in existing multi-view video content leads to a degradation in view synthesis quality, and the existing depth estimation methods cannot effectively ensure the coherence of depth information at the same point in the space.

Method used

By accessing different views of multi-view video content, forward projecting camera pixels to target view, measuring texture information differences and depth value differences, adjusting the depth value of the target view to improve the consistency of the depth map, and using the world when deprojecting the world The number of pixels in the coordinate system verifies coherence.

Benefits of technology

It improves the depth map consistency of multi-view video content, improves coding efficiency and view synthesis quality, ensures the accuracy and coherence of scene point clouds, and improves the user's viewing experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113545034B_ABST
    Figure CN113545034B_ABST
Patent Text Reader

Abstract

For multi-view video content represented in MVD (multi-view + depth) format, depth maps can be processed to improve the coherence between them. In one embodiment, in order to process a target view based on an input view, the pixels of the input view are first projected into a world coordinate system and then projected into a target view to form a projected view. The texture of the projected view and the texture of the target view are compared. If the difference at a pixel is small, the depth of the target view at the pixel is adjusted, for example, replaced with the corresponding depth of the projected view. When encoding and decoding multi-view video content in a system, depth map processing can be applied in pre-processing and post-processing modules to improve video compression efficiency and rendering quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present embodiments generally relate to a method and apparatus for depth map processing, and more particularly, to a method and apparatus for processing a depth map associated with multi-view video content represented in an MVD (Multi-View + Depth) format. Background Art

[0002] MVD is a format for transmitting multi-view content (multiple views from multiple cameras), each view having an associated depth map. When representing multi-view video in the MVD format, for each view frame, a texture image is associated with a depth map, for example, at the same pixel resolution. The depth map can be generated by different methods, such as by calculation or measurement. MVD is the input format for extending HEVC (High Efficiency Video Coding) to compress multiple views and associated depth information. This format may also be the input format for future extensions of H.266 / VVC (Versatile Video Coding) to address future 6DoF (six degrees of freedom) content, such as what MPEG calls "windowed 6DoF" today. Summary of the invention

[0003] According to an embodiment, a video processing method is provided, comprising: accessing a first view and a second view of a multi-view video content; projecting a first pixel at a first position in the first view onto a second position in the second view; determining a difference between (1) texture information of the first pixel of the first view and (2) texture information of a second pixel at the second position in the second view; and in response to the difference, adjusting a depth value of the second pixel of the second view based on a depth value of the first pixel of the first view.

[0004] According to another embodiment, a device for video processing is provided, comprising one or more processors, wherein the one or more processors are configured to: access a first view and a second view of multi-view video content; project a first pixel at a first position in the first view onto a second position in the second view; determine a difference between (1) texture information of the first pixel of the first view and (2) texture information of a second pixel at the second position in the second view; and in response to the difference, adjust a depth value of the second pixel of the second view based on a depth value of the first pixel of the first view. The device may also include one or more memories coupled to the one or more processors.

[0005] According to another embodiment, a device for video processing is provided, comprising: a component for accessing a first view and a second view of multi-view video content; a component for projecting a first pixel at a first position in the first view to a second position in the second view; determining a difference between (1) texture information of the first pixel of the first view and (2) texture information of the second pixel at the second position in the second view; and a component for adjusting a depth value of the second pixel of the second view based on the depth value of the first pixel of the first view in response to the difference. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A system for encoding and decoding multi-view video content is illustrated.

[0007] Figure 2 A 16-camera array is shown.

[0008] Figure 3 Illustrated are texture images from a view and their associated depths.

[0009] Figure 4 A process of processing a depth map according to an embodiment is illustrated.

[0010] Figure 5 The diagram shows the world coordinate system, the camera coordinate system and the corresponding transformation matrix.

[0011] Figure 6 Illustration of the forward projection of an input view onto a target view.

[0012] Figure 7 Illustrated is a block diagram of a system in which aspects of the present embodiments may be implemented. DETAILED DESCRIPTION

[0013] To provide motion parallax, the multi-view content should be dense enough to provide enough overlap between the views, but have different perspectives to allow for this parallax. This means that a lot of the information captured by different cameras corresponds to the same part of the scene. This is an element that compression algorithms use to reduce the amount of data to be transmitted. In the 3D-HEVC and MV-HEVC extensions of HEVC, inter-view prediction has been introduced. At the time, multi-view systems were mostly considered horizontal-only systems, and the prediction mechanisms only utilized the horizontal direction. Now the camera arrays are no longer just horizontal, but more in 2D (planar) or even 3D (non-planar) arrangements.

[0014] Depth estimation is a useful element in providing multi-view content. Since the use of the content may require the synthesis of intermediate viewpoints, depth information is usually transmitted. In one example, the way in which the depth information is calculated depends on the matching criteria between the current view and the adjacent view. In order to limit the amount of computation, only the most recent view (usually the view that maximizes the correlation) is used for such calculations. Typically, the depth information associated with a view is calculated independently of other views, so depth maps from different views may lack coherence. The present embodiment relates to a method of modifying depth maps to provide more coherence between them. When the pixels of a view are de-projected to a world coordinate system using the depth map information from the view, the coherence of the two depth maps can be measured by the number of pixels of the corresponding views that are located at the same position in space (the world coordinate system).

[0015] The goal of MPEG-I programs is the delivery of 6DoF content, which allows the end user to perceive parallax. The client-rendered content should adapt in real time to the viewer's head movements. To create this parallax, one should provide not only the usual 2D content, but also content corresponding to content that is not viewed from the initial viewpoint but can be viewed from other viewpoints. This content can typically be captured by an array of cameras, each viewing the scene from a slightly different angle. The distance between the cameras gives roughly the amount of parallax the system will be able to provide.

[0016] The amount of data required to transmit multi-view content is huge. It includes depth maps associated with each view to enable synthesis of other viewpoints on the rendering side. The MVD format has been used in the past to deliver such content. For example, it has been used as the input format for the 3D-HEVC extension of HEVC.

[0017] If view synthesis is used to generate viewpoints that the user may request, depth information is usually transmitted together with the texture. In addition, depth maps from different views should be coherent to achieve correct view synthesis. In a multi-view acquisition system, the same point in the world coordinate system (WCS) may be captured by several cameras and belong to several views. Depth map estimation is applied to each view and it associates depth information with each pixel. For two pixels of two views corresponding to the same point in the WCS, the associated depth should allow the same point to be recreated in the WCS when de-projection is applied. When a given number of pixels verify this property, we have coherence between the depth maps of the two views. Otherwise, their projections in the WCS will not be in the same position, and then projecting them to the synthesized viewpoint will create a double pixel effect. Finally, this difference may significantly reduce the quality of view synthesis.

[0018] A reference software developed within MPEG called DERS (Depth Estimation Reference Software) is available for use by the entire MPEG community. The software has been extensively updated since the first version and now DERS 7.0 is the latest reference version. Nevertheless, even the latest developments still do not address the need for coherence between different depth maps. Each view estimates its own depth map using a set of input views. Since there is no correlation between two estimates corresponding to two different views, the estimated depth maps are generally incoherent. When depth maps from different views are used to generate a point cloud, there is no special treatment to ensure that two depth estimates corresponding to the same point in space will be coherent. The two depth maps may happen to give two separated points in space. Further processing using these depths may fail because the same point in space is associated with more than one depth.

[0019] The present method attempts to provide a more coherent set of depth maps for a set of views in order to have a more coherent point cloud associated with the multi-view plus depth content. Although isolated pixels may remain in the point cloud, the result is a more coherent point cloud where there is a unique 3D point in space for a given point in the scene. Further processing such as view synthesis based on several views and associated depths can therefore be improved.

[0020] Figure 1 A system for encoding and decoding multi-view video content is illustrated. Multi-view video content represented in MVD format is used as input. The depth map can be processed in a pre-processing module to improve coherence. The multi-view content can then be encoded and decoded using (multiple) video encoders / decoders, such as (multiple) HEVC / VVC encoders / decoders. Metadata, such as the pose and intrinsic parameters of the camera used to capture the multi-view video content, can also be transmitted. Due to the enhanced coherence in the depth map, the coding efficiency can be improved. On the decoder side, the depth map can be processed to improve the coherence in the post-processing module. The multi-view video content can then be rendered at a dynamic viewpoint and displayed to the viewer. By enhancing the coherence between views, view synthesis can be improved, and thus the user's viewing experience can be improved.

[0021] Figure 2 An example of a 16-camera array is shown (based on 4 cameras by 4 cameras), and Figure 3 An example of a view and its associated depth map is illustrated.For such a camera array, the overlap between the captured views is important.

[0022] Figure 4 An example method 400 for processing a depth map according to an embodiment is illustrated. The method 400 may be used in pre-processing and post-processing modules, such as Figure 1As shown. For a multi-view video with multiple views, the processing is applied to at least one view and can be applied to a set of input views or all input views. In the following, for the convenience of annotation, we refer to the view whose depth map is to be processed as the "target view" and the view generated by projecting the current input view onto the target view as the "projected view".

[0023] In method 400, at step 410, a target view to be processed is selected. In one example, each input view will be processed, i.e., each input view will be a target view at some point in time. At step 420, another input view other than the target view is selected. The selection is done among all possible input views or among a set of input views defined relative to the target view. For example, we can only select from the nearest view. At step 430, pixels of the input view (m) are forward projected (440) onto the target view. At step 450, the light coherence between the pixels of the projected view and the pixels of the target view is evaluated, for example based on texture differences. At step 455, if the light coherence is above (455) a given threshold (i.e., the difference is below a given threshold), the processing of depth replacement can be started at step 460.

[0024] When there are no more pixels to process (465, 470), the new depth map is available as output (475). To remove potentially isolated erroneous depth information, filtering, such as a median filter, may be applied. The processing may be recursive (490), i.e., at the end of processing a target view, the modified depth is considered as the new depth for processing another target view. Alternatively, the processing is not recursive, i.e., the new depth of the target view is not considered for processing another target view, but the old depth is considered. After all target views are processed (480), the method 400 ends at step 499.

[0025] Below, methods that may be used for forward projection (440), measuring light coherence (450), and depth value replacement (460) are described in more detail.

[0026] Forward Projection

[0027] Let us consider a camera calibrated as a normal pinhole camera. Its intrinsic matrix, where f represents the distance from the exit pupil to the sensor, expressed in pixels, usually called the "focal length"; denotes the pixel coordinates of the so-called “principal point”, i.e. the orthogonal projection of the pinhole onto the sensor; and α and γ denote the aspect ratio of the pixel and the skewness coefficient of the sensor, respectively.

[0028] if is the coordinate of a given point in the camera's coordinate system (CS), then the coordinate of its image projection is (in pixels) is given by:

[0029]

[0030] The symbol “≡” represents the equivalence relationship between homogeneous vectors:

[0031]

[0032] Now suppose represents the camera pose matrix, where and Respectively represent the orientation and position of the camera in the reference coordinate system (CS). The extrinsic matrix of the camera is defined as:

[0033]

[0034] if and Represent the coordinates of the same point in the camera CS and the reference CS respectively, then and like Figure 5 As shown, Figure 5 The world and camera coordinate systems and the corresponding transition matrices are shown.

[0035] Now let us consider the given camera as corresponding to the input view and let m be its index. is the position of the current pixel in the texture image of the input view, and z is its assumed depth. For a pixel P(u,v) of input view m (from camera m), a forward projection of input view m to target view n can be performed, such as Figure 6 shown.

[0036] First, the pixels of the input view m are deprojected into the world coordinate system (XYZ) as Pw. This deprojection uses the pose matrix, the inverse intrinsic matrix of camera m, and the depth Z(u,v) associated with the pixel. Subsequently, the pixel Pw is projected onto the target view n, resulting in P'(u',v'). Using the intrinsic and extrinsic matrices of camera n, the points in space Pw are projected onto the target view n.

[0037] Mathematically, the global calculation from P(u,v) to P'(u',v') can be expressed as:

[0038]

[0039] When the projection does not fall into the region of view n, the projection can be rejected. Rejection may result in holes for some pixels in the projected view, and no texture comparison will be done for these pixels. When the projection falls into the region of camera n, it is very likely that the projection will not fall on the real pixel, but between the 4 pixels. If the pixel is not rejected, the nearest pixel on camera n can be selected as P proj (u p ,v p ). The nearest pixel is the pixel that minimizes the distance between the projection result P'(u',v') and one of the 4 neighboring pixels, where the distance can be calculated as the square root of the sum of the squared differences between the coordinates:

[0040] οP1(int(u'),int(v'))

[0041] οP2(int(u')+1,int(v'))

[0042] οP3(int(u'),int(v')+1)

[0043] οP4(int(u')+1,int(v')+1)

[0044] Light coherence

[0045] To decide whether to consider the depth information of another view from the target view, the light coherence of the projected pixel is evaluated, for example, by a weighted difference between the target and projected textures. When the difference is low, the light coherence is high. In one embodiment, the difference can be based on a comparison of the absolute difference between the texture of the current pixel using both the target view (Rt, Gt, Bt) and the projected view (Rp, Gp, Bp). The comparison is applied to the three components of the texture (R, G, B) and can be applied using different weights for each component or using the same weights.

[0046] Difference=wr*abs(Rt–Rp)+wg*abs(Gt–Gp)+wb*abs(Bt–Bp)

[0047] The weights wr, wg and wb may all be equal to 1 / 3 or may have different values, for example, to better represent their contribution to the brightness.

[0048] In another embodiment, the comparison is based on the squared difference between the textures of the current pixel of both the target view and the projected view. The comparison is applied to the three components (R, G, B) of the texture and can be applied using different weights for each component or using the same weights.

[0049] Difference=wr*(Rt–Rp)2+wg*(Gt–Gp)2+wb*(Bt–Bp)2

[0050] The weights wr, wg and wb may all be equal to 1 / 3 or may have different values, for example, to better represent their contribution to the brightness.

[0051] In another embodiment, the comparison is based on the squared difference or absolute difference between the textures of a set of pixels around the current pixel of both the target view and the projection view. The comparison is applied to the three components of the texture (R, G, B) and can be applied with different weights for each component or with the same weights. The size of the window can be 3*3 or 5*5 pixels or any other value.

[0052] Difference=ΣΣ(wr*(Rt–Rp)2+wg*(Gt–Gp)2+wb*(Bt–Bp)2)

[0053] Depth value replacement

[0054] If the texture comparison shows that the pixels from the projected view are similar to the pixels of the target view (e.g., the texture difference is below a given threshold and the light coherence is high), the depth value of the target view may be replaced with the depth value associated with the projected view. For example, this can be done using one of the following schemes.

[0055] The first solution is to simply replace the depth value of the target view with the depth value of the projected view. Figure 6 For example, in the target view n, P proj (u p ,v p ) is replaced by the depth value at P(u,v) in the input view m. The first solution assumes that the cameras are coplanar with parallel optical axes. Related to equation (1), it means that the relationship between the pose matrix of the input view m and the extrinsic matrix of the target view n is as follows:

[0056]

[0057] where θ represents the angle of rotation about the z axis. As a result, we have z'=z.

[0058] The second solution is to replace the depth value of the target view with the recomputed depth value of the projected view. This recomputation incorporates the pose and intrinsic camera parameters of the target view. The new z' value is calculated using equation (1).

[0059] A third solution is to check if the depth value of the target view meets the criteria for the projected view test for the projected view. Specifically, the depth of the target view is used to implement an inverse process by projecting the target view onto the input view and testing if the light coherence is above a threshold (which can be the same or different). If the inverse process succeeds, no replacement is made, otherwise replacement is made as described above.

[0060] After the target view has been processed (i.e. a new depth map is available), the result can be filtered since isolated depth data may have been introduced during processing. In order to eliminate such isolated depth data, a median filter can be applied. The size of the median filter and the number of times the median filter is applied should be defined. The size can be 3*3 pixels or 5*5 or larger. The number of times the median filter occurs can be once or more than once. Typical values ​​are a 5*5 window and 2 occurrences. Different variants of the median filter are possible, depending on the size of the window and the number of occurrences. Other filters, such as an average filter, can also be used.

[0061] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0062] In the above, HEVC or VVC is used when describing a video encoder / decoder. It should be noted that the present aspect is not limited to VVC or HEVC, and can be applied, for example, when using other standards and recommendations and extensions of any such standards and recommendations. The color space of the texture video is not limited to the RGB space, for example, it can be represented by YUV. Unless otherwise specified or technically excluded, the aspects described in this application can be used individually or in combination.

[0063] Various values ​​are used in this application, such as the size of the median filter. The specific values ​​are used for illustrative purposes, and the described aspects are not limited to these specific values.

[0064] Figure 7The block diagram of the example of the system in which various aspects and embodiments are implemented is illustrated. System 700 can be implemented as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, notebook computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 700 can be implemented in a single integrated circuit, multiple ICs and / or discrete components individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 700 are distributed on multiple ICs and / or discrete components. In various embodiments, system 700 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or through a dedicated input and / or output port. In various embodiments, system 700 is configured to implement one or more aspects described in this application.

[0065] The system 700 includes at least one processor 710, which is configured to execute instructions loaded therein to implement, for example, various aspects described in the present application. The processor 710 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 700 includes at least one memory 720 (e.g., a volatile storage device and / or a non-volatile storage device). The system 700 includes a storage device 740, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a disk drive, and / or an optical drive. As a non-limiting example, the storage device 740 may include an internal storage device, an additional storage device, and / or a network accessible storage device.

[0066] The system 700 includes an encoder / decoder module 730, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 730 may include its own processor and memory. The encoder / decoder module 730 represents a (multiple) module that can be included in the device to perform encoding and / or decoding functions. As known, the device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 730 can be implemented as a separate element of the system 700, or can be incorporated into the processor 710 as a combination of hardware and software known to those skilled in the art.

[0067] Program code to be loaded on the processor 710 or the encoder / decoder 730 to perform various aspects described in this application may be stored in the storage device 740 and subsequently loaded onto the memory 720 for execution by the processor 710. According to various embodiments, one or more of the processor 710, the memory 720, the storage device 740, and the encoder / decoder module 730 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0068] In several embodiments, memory inside the processor 710 and / or encoder / decoder module 730 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 710 or the encoder / decoder module 730) is used for one or more of these functions. The external memory may be a memory 720 and / or a storage device 740, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC.

[0069] Input to the elements of system 700 may be provided through various input devices, as shown at block 705. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted over the air, such as by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0070] In various embodiments, the input device of block 705 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements adapted to (i) select a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) again bandlimit the signal to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulate the downconverted and bandlimited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a near baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band by filtering, down-conversion and again to perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions. Adding element can include inserting element between existing element, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.

[0071] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 700 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented in, for example, a separate input processing IC or in the processor 710 when necessary. Similarly, aspects of USB or HDMI interface processing may be implemented in a separate interface IC or in the processor 710 as needed. The streams of demodulation, error correction, and demultiplexing are provided to various processing elements, including, for example, the processor 710, and the encoder / decoder 730, which operates in combination with memory and storage elements to process the data stream as needed to present on the output device.

[0072] The various components of the system 700 may be provided within an integrated housing. Within the integrated housing, the various components may be interconnected and transmit data therebetween using suitable connection devices 715, such as internal buses known in the art, including I2C buses, wiring, and printed circuit boards.

[0073] The system 700 includes a communication interface 750 capable of communicating with other devices via a communication channel 790. The communication interface 750 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 790. The communication interface 750 may include, but is not limited to, a modem or a network card, and the communication channel 790 may be implemented, for example, within a wired and / or wireless medium.

[0074] In various embodiments, data is streamed to the system 700 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals of these embodiments are received through a communication channel 790 and a communication interface 750 suitable for Wi-Fi communication. The communication channel 790 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that transfers data through an HDMI connection of an input block 705 to provide streaming data to the system 700. Still other embodiments use an RF connection of an input block 705 to provide streaming data to the system 700.

[0075] The system 700 can provide output signals to various output devices, including a display 765, a speaker 775, and other peripheral devices 785. In various examples of embodiments, the other peripheral devices 785 include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 700. In various embodiments, control signals are transmitted between the system 700 and the display 765, the speaker 775, or other peripheral devices 785 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 700 via dedicated connections through respective interfaces 760, 770, and 780. Alternatively, the output devices can be connected to the system 700 using a communication channel 790 via a communication interface 750. The display 765 and the speaker 775 can be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 700. In various embodiments, the display interface 760 includes a display driver, such as a timing controller (T Con) chip.

[0076] For example, if the RF portion of input 705 is part of a separate set-top box, the display 765 and speaker 775 may alternatively be separate from one or more other components. In various embodiments where the display 765 and speaker 775 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0077] According to an embodiment, a video processing method is provided, comprising: accessing a first view and a second view of a multi-view video content; projecting a first pixel at a first position in the first view onto a second position in the second view; determining a difference between (1) texture information of the first pixel of the first view and (2) texture information of a second pixel at the second position in the second view; and in response to the difference, adjusting a depth value of the second pixel of the second view based on a depth value of the first pixel of the first view.

[0078] According to another embodiment, a device for video processing is provided, comprising one or more processors, wherein the one or more processors are configured to: access a first view and a second view of multi-view video content; project a first pixel at a first position in the first view to a second position in the second view; determine a difference between (1) texture information of the first pixel of the first view and (2) texture information of the second pixel at the second position in the second view; and in response to the difference, adjust a depth value of the second pixel of the second view based on the depth value of the first pixel of the first view.

[0079] According to another embodiment, a device for video processing is provided, comprising: a component for accessing a first view and a second view of multi-view video content; a component for projecting a first pixel at a first position in the first view to a second position in the second view; determining a difference between (1) texture information of the first pixel of the first view and (2) texture information of the second pixel at the second position in the second view; and a component for adjusting a depth value of the second pixel of the second view based on the depth value of the first pixel of the first view in response to the difference.

[0080] In one embodiment, the projecting includes: projecting the first pixel to a fourth position in a world coordinate system based on a pose and inverse intrinsic parameters associated with a camera used to capture the first view; projecting the fourth position in the world coordinate system to a fifth position in the second view based on intrinsic and extrinsic parameters associated with a camera used to capture the second view; and obtaining the second position from the fifth position. In one embodiment, the obtaining includes selecting a pixel position closest to the fifth position as the second position.

[0081] In one embodiment, the multi-view video content is represented in a multi-view plus depth (MVD) format.

[0082] In one embodiment, the first view is an adjacent view to the second view.

[0083] In one embodiment, the difference value is determined based on respective differences of three components of (1) the texture information of the first pixel of the first view and (2) the texture information of the second pixel of the second view. In another embodiment, the difference value is determined based on respective differences of three components of (1) the texture information of a group of pixels surrounding the first pixel of the first view and (2) the texture information of a corresponding group of pixels surrounding the second pixel of the second view.

[0084] In one embodiment, the adjusting is performed if the difference is less than a value.

[0085] In one embodiment, the adjusting further comprises applying a gesture and intrinsic parameters associated with the second view.

[0086] In one embodiment, the one or more processors are further configured to perform: projecting the second pixel of the second view to a third position in the first view; and determining a second difference between (1) the texture information of the second pixel of the second view and (2) the texture information of a third pixel at the third position in the first view, wherein the adjustment is performed only when the second difference is less than a second value.

[0087] In one embodiment, the projection is performed on all pixels in the first view. For example, the projection result may be filtered by a median filter. The projection result may form a projection view.

[0088] In one embodiment, the first view is represented by a first texture image and a first depth map, and the second view is represented by a second texture image and a second depth map.

[0089] An embodiment provides a computer program including instructions, which, when executed by one or more processors, causes the one or more processors to perform a video processing method according to any of the above embodiments. One or more of the embodiments also provides a computer readable storage medium having stored thereon instructions for processing depth data according to the above method.

[0090] The embodiments and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams or signals. Even if only discussed in the context of a single implementation form (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). The device can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in a device such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0091] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in conjunction with the embodiment is included in at least one embodiment. Therefore, the appearance of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation", as well as any other variations, in various places in the present application do not necessarily refer to the same embodiment.

[0092] Furthermore, the present application may refer to “determining” various pieces of information. Determining information may include, for example, estimating information, calculating information, predicting information, or retrieving information from a memory.

[0093] Additionally, the present application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0094] Additionally, the application may refer to "receiving" various pieces of information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing the information or retrieving the information (e.g., from a memory). Additionally, "receiving" generally involves, in one way or another, during an operation, e.g., storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0095] It should be understood that the use of the following “ / ,” “and / or,” and “at least one of,” for example, in the case of “A / B,” “A and / or B,” and “at least one of A and B,” is intended to include selecting only the first listed option (A), or selecting only the second listed option (B), or selecting both options (A and B). As a further example, in the case of “A, B, and / or C” and “at least one of A, B, and C,” such language is intended to include selecting only the first listed option (A), or selecting only the second listed option (B), or selecting only the third listed option (C), or selecting only the first and second listed options (A and B), or selecting only the first and third listed options (A and C), or selecting only the second and third listed options (B and C), or selecting all three options (A and B and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to as many items as listed.

[0096] It will be apparent to one of ordinary skill in the art that implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

Claims

1. A video processing method, include: accessing a first view and a second view of the multi-view video content; projecting a first pixel at a first position in the first view to a second position in the second view; determining a difference between (1) texture information of the first pixel of the first view and (2) texture information of a second pixel at the second position in the second view by evaluating light coherence between pixels of the first view and pixels of the second view; and In response to a comparison of the difference value with a threshold, a depth value of the second pixel of the second view is adjusted based on the depth value of the first pixel of the first view.

2. A device for video processing, comprising one or more processors, wherein the one or more processors are configured to: accessing a first view and a second view of the multi-view video content; projecting a first pixel at a first position in the first view to a second position in the second view; determining a difference between (1) texture information of the first pixel of the first view and (2) texture information of a second pixel at the second position in the second view by evaluating light coherence between pixels of the first view and pixels of the second view; and In response to a comparison of the difference value with a threshold, a depth value of the second pixel of the second view is adjusted based on the depth value of the first pixel of the first view.

3. The method of claim 1 or the device of claim 2, wherein the projection include: projecting the first pixel into a fourth position in a world coordinate system based on a pose and inverse intrinsic parameters associated with a camera used to capture the first view; projecting the fourth position in the world coordinate system into a fifth position in the second view based on intrinsic and extrinsic parameters associated with a camera used to capture the second view; and The second position is obtained from the fifth position.

4. The method of claim 3, or the apparatus of claim 3, wherein the obtaining comprises selecting the pixel position closest to the fifth position as the second position.

5. The method of any one of claims 1, 3-4, or the apparatus of any one of claims 2-4, wherein the first view is an adjacent view of the second view.

6. The method of any one of claims 1, 3-4, or the device of any one of claims 2-4, wherein the difference value is determined based on respective difference values ​​of three components of (1) the texture information of the first pixel of the first view and (2) the texture information of the second pixel of the second view.

7. The method according to any one of claims 1, 3-4, or the device according to any one of claims 2-4, in, The difference value is determined based on respective difference values ​​of three components of (1) texture information of a pixel group surrounding the first pixel of the first view and (2) texture information of a corresponding pixel group surrounding the second pixel of the second view.

8. The method according to any one of claims 1, 3-4, or the device according to any one of claims 2-4, in, If the difference is less than a threshold, the adjustment is performed.

9. The method of any one of claims 1, 3-4, or the apparatus of any one of claims 2-4, wherein the adjusting further comprises applying a gesture and intrinsic parameters associated with the second view.

10. The method according to any one of claims 1, 3-4, or the device according to any one of claims 2-4, wherein the method further comprises the following steps, or the one or more processors in the device are further configured to perform the following steps: projecting the second pixel of the second view onto a third position in the first view; Determine a second difference between (1) the texture information of the second pixel of the second view and (2) the texture information of a third pixel at the third position in the first view, wherein the adjusting is performed only when the second difference is less than a second value.

11. The method of any one of claims 1, 3-4, or the apparatus of any one of claims 2-4, wherein the projection is performed for all pixels in the first view.

12. The method of claim 11, or the apparatus of claim 11, wherein the projection results are filtered.

13. The method of any one of claims 1, 3-4, 12, or the apparatus of any one of claims 2-4, 12, wherein the first view is represented by a first texture image and a first depth map, and wherein the second view is represented by a second texture image and a second depth map.

14. The method of any one of claims 1, 3-4, 12, or the apparatus of any one of claims 2-4, 12, wherein the multi-view video content is represented in a multi-view plus depth (MVD) format.

15. A non-transitory machine-readable medium having machine-executable instructions stored thereon, wherein the machine-executable instructions, when executed, implement a method, wherein the method include: accessing a first view and a second view of the multi-view video content; projecting a first pixel at a first position in the first view to a second position in the second view; determining a difference between (1) texture information of the first pixel of the first view and (2) texture information of a second pixel at the second position in the second view by evaluating light coherence between pixels of the first view and pixels of the second view; and In response to a comparison of the difference value with a threshold, a depth value of the second pixel of the second view is adjusted based on the depth value of the first pixel of the first view.