Image processing method for setting transparency and color values ​​of pixels in a virtual image - Patents.com

Abstract

A method is provided for selecting transparency settings and color values ​​for pixels in a virtual image. The virtual image can be formed by combining reference images captured at different angles to generate a virtual image that views an object at a new, uncaptured angle. The method includes determining, for each pixel in the virtual image, what information it has from the reference view image. The pixel's information is used to define a pixel category, which is used to select, based on logical conditions, what information is displayed by that pixel and to set the pixel's color.

Description

[Technical field]

[0001] The present invention relates to the field of image processing, and more particularly to the field of generating a virtual image from a combination of reference view images. The present invention particularly relates to a method for setting transparency and color values ​​of pixels in a virtual image. [Background technology]

[0002] A virtual image of an object can be created by combining reference view images of the object captured from multiple different viewpoints. A virtual image can be generated from the combination of reference view images to view the object from any angle, including angles at which the reference view images were not captured.

[0003] The reference-view images are often captured against a chromakey background so that the resulting virtual image of the target can be used on a variety of graphical backgrounds, for example, in virtual reality or augmented reality (VR / AR). A virtual image formed from a combination of the reference-view images may include overlapping regions of any combination of background and / or object regions from each of the reference-view images.

[0004] Existing image processing techniques deal with layered images by calculating the color of a surface pixel according to a linear combination of the color value and transparency value of the layers in the image. US 2009 / 102857 is an example of this technique, which calculates the color of a surface pixel according to the transparency value of the surface pixel, the transparency of the pixel below, and the color of the pixel below.

[0005] Linear combination techniques such as those in U.S. Patent Application Publication No. 2009 / 102857 introduce regions of pseudo-transparency when used on a virtual image. These regions of pseudo-transparency occur near the edges of objects from each reference-view image, which means that a virtual image formed from an overlapping combination of the reference-view images can contain many regions of pseudo-transparency.

[0006] Pseudo-transparency areas are commonly referred to as ghost areas. These ghost areas cause the graphics background used with the object to be visible through the object, reducing the quality of the virtual image and, therefore, the quality of the VR / AR in which the virtual image may be used. VR / AR users may lose immersion in the scenario in which they are participating and may be unable to perform tasks required of them due to ghost areas that prevent them from seeing or reading the object. Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, what is needed is an image processing method that can set the transparency and color values ​​of pixels in a virtual image without causing ghost areas. [Means for solving the problem]

[0008] The invention is defined by the claims.

[0009] According to an example embodiment of the present invention, there is provided a method of setting color and transparency values ​​for each of a plurality of virtual image pixels in a virtual image, the method comprising:

[0010] receiving a plurality of reference-view images of an opaque object, each of the plurality of reference-view images being captured from a different viewpoint;

[0011] and combining the plurality of reference-view images to generate a virtual image of the object, the virtual image being at a different viewpoint than any of the plurality of reference-view images, and each reference-view image pixel corresponding to a virtual image pixel through a mapping, the generating of the virtual image comprising:

[0012] determining, for each of a plurality of virtual image pixels, transparency information including at least foreground and background information components derived from corresponding pixels from each of a plurality of reference view images;

[0013] classifying each of the plurality of virtual image pixels based on a combination of foreground and background information components;

[0014] For areas away from the edges of the object, the method includes selecting a binary transparency value for each of the plurality of virtual image pixels based on their classification to display foreground or background information, and setting a color value for each of the plurality of virtual image pixels based on the foreground or background information selected to be displayed.

[0015] Each of the multiple reference view images may include depth information, and a mapping between reference view image pixels and virtual image pixels may be based on the depth information.

[0016] The method classifies each pixel in a virtual image formed from multiple reference images taken from different angles or viewpoints. The virtual image is created by combining the reference images, which means that each pixel of the virtual image is constructed using information components from multiple reference images. Each pixel of the reference view image is mapped to at least one virtual image pixel. The transparency information components (e.g., foreground information from reference view image 1 and background information from reference view image 2) mapped to each of the multiple virtual image pixels can be determined, and a classification for each pixel is generated based on the combination of foreground / background information components. The virtual pixel classification is used to determine which component should be displayed, and then the color of the virtual image pixel is set to a specific respective value. By selecting one of the components of transparency information to be displayed, the invention ensures that the virtual image of the object does not contain background artifacts that overlap the object, or cause ghosting when projected onto the background if the object was captured on a chromakey (such as a green screen).

[0017] The transparency value indicates whether a pixel is in the foreground or background.

[0018] Each pixel of the constructed virtual image contains, for example, an RGBA value, with the alpha channel (A) containing the required background / foreground information. The reference images may be coded in the same format as the RGBA pixels. They are warped to generate the virtual image. The reference images also typically have an associated depth or disparity map (D) that contains 3D information, such that a camera shift results in a warped image guided by the local 3D information. Thus, the reference image may contain five channels, namely RGBA, AD.

[0019] Each of the reference view pixels includes transparency information, for example in the form of foreground information, background information, or undetermined information, and determining the transparency information further includes determining an undetermined component for each of the plurality of virtual image pixels.

[0020] Thus, a pixel in a reference image can be coded as foreground, background, or undetermined. The combination of these three states from multiple reference images determines the classification for each pixel, which in turn determines the transparency information.

[0021] The edge regions of an object may contain a combination of background and foreground components and may cause ghosting effects. This undetermined component allows ghosting regions to be identified.

[0022] Classification of virtual image pixels (referred to herein as pixels) may further include classification of ghost regions for pixels that include a combination of undetermined region components and foreground information.

[0023] This classification allows ghost regions to be located and targeted for correction.

[0024] The method may include, for each virtual image pixel having transparency information that includes a component of foreground information only, selecting a transparency value that corresponds to the foreground.

[0025] Selecting only foreground information for display can help eliminate the ghosting effect, since it helps to eliminate the presence of background color on the surface of the object introduced by ghost regions. Additionally, it can ensure that the background information component of the image does not overlay the foreground information component in pixels that contain both foreground and background components.

[0026] A new outer boundary of the object can be created in the virtual image by selecting a transparency value that corresponds to the foreground for all virtual image pixels that have transparency information that includes only a combination of background and undetermined components or a combination of multiple undetermined components.

[0027] This new outer boundary can connect the overlapping boundaries from the reference view images into a single continuous outer edge.

[0028] This choice of pixel classification selects pixels that are at or near the outer boundary of the overlapping reference view images of the object, reducing the reduction in object size that may occur when defining the new outer boundary.

[0029] The setting of color values ​​in the method may set the respective color value of each of the pixels having transparency information including only foreground information components as an average color value of the respective foreground information components from the reference view image pixels corresponding to each pixel. A weighted combination may be used, with a higher weight being given if the target virtual viewpoint is closer to a particular reference view.

[0030] This allows an object in the virtual image to have smooth transitions of color across its surface, since the reference view image can show the object in different colors due to lighting, reflections and / or objects having different colors on different surfaces.

[0031] The method can include selecting a binary transparency value for all of the virtual image, so that pixels throughout the virtual image can be classified as either foreground or background.

[0032] One limit of transparency is for the background and the other limit is for the foreground.Transparency values ​​between these limits can be set for the edge regions of objects.

[0033] In this way, the sharpness of the color transition at the edge of the object to the background can be reduced.

[0034] The method includes, for example, for each of a plurality of virtual image pixels having transparency information that includes an undetermined component and does not include a foreground component, setting a transparency value for each of the pixels using a color difference between the pixel and at least one adjacent pixel.

[0035] Virtual image pixels that contain undetermined content but no foreground content are those that define the edges of objects. They are set to an intermediate transparency value rather than a binary foreground / background setting. The transparency is set based on the color of adjacent virtual image pixels that contain foreground information.

[0036] This allows the edges of the object to blend with the new background, allowing for a smooth transition that does not look sharp or unrealistic. Transparency can be calculated using the color difference of adjacent pixels, and color difference from a known (e.g., green) background color can be calculated using a Euclidean distance weighted combination for adjacent pixels, or other averaging methods known in the art, and any combination thereof.

[0037] The transparency information component of each of the plurality of virtual pixels may be derived from the color of the reference-view image pixel corresponding to each of the plurality of virtual pixels. Each reference-view image pixel that is combined to form a virtual image pixel contributes some color to the virtual image pixel. This color may be used to determine whether the pixel displays part of the image background or part of the target object.

[0038] The background of the reference view image may be a chromakey background.

[0039] This makes it simple to distinguish between background and foreground components by color selection since the background is a known and consistent color.

[0040] The method may be a computer program comprising computer program code means adapted to perform the method when the computer program is run on a computer.

[0041] An image processing apparatus is also provided, the image processing apparatus comprising:

[0042] an input for receiving a plurality of reference view images of an object, each of the plurality of reference view images including foreground and background information captured from a different viewpoint;

[0043] A processor for processing a plurality of reference view images to generate a virtual image, and an output for outputting said virtual image, said processor configured to implement the method defined above.

[0044] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0045] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Figure 1A] FIG. 1 illustrates an example setup for capturing a reference view image of a semi-transparent window through which the background can be seen. [Figure 1B] 1 illustrates an exemplary setup for capturing a reference view image of an object in front of a chromakey background. [Diagram 2] FIG. 2 illustrates an example of a top view of two reference cameras capturing reference view images of an object against a chromakey background. [Diagram 3] FIG. 2 shows an example of a virtual image formed from two reference view images with classifications for use in the method of the present invention. [Figure 4] 2 is a flow chart showing the steps of an example of the present invention. [Diagram 5] 4 illustrates the virtual image of FIG. 3 after color and transparency values ​​have been set in accordance with an embodiment of the present invention. [Figure 6] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] The present invention will now be described with reference to the drawings.

[0047] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0048] The present invention provides a method for setting transparency and color values ​​of pixels in a virtual image. A virtual image can be formed by combining reference images taken at different angles to generate a virtual image that views an object at a new, uncaptured angle. The method includes determining, for each pixel of the virtual image, what information from the reference view image it has. The pixel's information is used to define a pixel category, which category is used to select, based on logical conditions, what information is displayed by that pixel and to set the color and transparency values ​​of that pixel. The present invention particularly relates to blending a virtual image of an object with a new background that is always assumed to be further away than the object.

[0049] Before describing the present invention in detail, we first discuss the issues related to image transparency and depth, as well as conventional approaches to creating new images (called "virtual images") from different viewpoints.

[0050] 1A shows an example of how transparency occurs in multi-camera capture. The scene being captured has an object with a foreground 100 and a background 102, with the background 102 visible through a semi-transparent window 104. The entire scene is captured by a set of cameras 106, each of which produces a reference image. The images are processed to produce a new image from a virtual camera 108 at a different viewpoint than each camera in the set of cameras 106. This new image is called a virtual image, meaning that it is an image that would be captured from a virtual camera position, i.e. a position where no real camera image is available.

[0051] For a semi-transparent window 104, each camera 106 sees a different mix of foreground 100 and background 102. A standard approach to synthesizing a new virtual image from the viewpoint of a virtual camera 108 is to weight warped versions of two or more reference views based on pose proximity, de-occlusion (stretching), and possibly depth. This naive approach can give sufficient image quality if the pose of the views to be synthesized is close to that of the reference views.

[0052] Figure 1B shows an example of the relevance of transparency when using a chromakey, where the background is set to a screen of a known color, such as green, allowing easy extraction of the foreground from the background in image processing.

[0053] This is a special case of transparency, in that the object to be captured is opaque, and outside its outer boundary is considered to be completely transparent (as in showing the background screen). Of course, the object does not have to be a closed solid shape, it may have openings through which the screen can be seen.

[0054] The problem here is more severe because the green screen background 102 must be replaced with a new background to generate the virtual image. For example, the foreground is superimposed on another background to create the overall desired image. If transparency is ignored, blended pixels that still contain green components remain at object boundaries. This is very visible when using a naive view blending method that ignores transparency, or when using linear combination methods.

[0055] The standard approach to chromakeying is to compute a transparency map: using the standard alpha matting formula, the image color of a pixel can be written as a linear combination of the foreground and background colors, i = αf + (1-α)b.

[0056] In this formula, i is the pixel color to be determined, f is the foreground color, b is the background color, and α is the transparency value. This formula is applied to the entire image, pixel by pixel, to create the transparency map.

[0057] Given the color characteristics of a particular background used for chromakeying (e.g., green), the foreground color f and transparency value α for a pixel can be estimated using one of the many algorithms that exist in the literature. The color data can be used to determine the transparency of each reference image pixel by comparing it to the known background color. If pixel color = chromakey color, the pixel is transparent. If a pixel contains only a fractional amount of chromakey color, it is semi-transparent. If it does not contain any chromakey color, it is not transparent.

[0058] Different algorithms exist for this purpose. They typically use the known color properties of the background material in combination with processing in a local neighborhood around the pixel. Even without a green screen, transparency can still be estimated around the detected depth step in the depth map. In such situations, mixed pixels are usually present around defocused foreground edges.

[0059] The value of α varies between 0 and 1. In this specification, a notation has been chosen in which 1 indicates zero transparency, where the foreground color shows through, and 0 indicates full transparency, where the background color shows through.

[0060] Thus, transparent areas of the image (α=0) are areas where the background is visible, and non-transparent areas of the image (α=1) are areas where the foreground is visible.

[0061] Typically, a small neighborhood of pixels is used to estimate these variables. When applied to k=1...N reference view images and taking into account the presence of perspective resulting from a depth sensor or depth estimation process, the following data is obtained: f 1, d1, α1, b1 … f N , d N , α N , b N

[0062] This data is for each pixel of reference images 1...N that corresponds to a pixel of the virtual image. The virtual image as a whole uses many sets of this data to determine values ​​for all of its pixels.

[0063] For all original pixels of the green screen, only the chromakey color is seen, so α = 0. For pixels on the foreground object, none of the chromakey colors are seen, so α = 1. For pixels on the border or in transparent areas, 0≦α≦1, since part of the pixel color is the chromakey color.

[0064] A transparency map, i.e. a set of values ​​for the entire image, can be used to blend a single reference view with a new background. However, known rendering approaches that use multiple reference views cannot handle transparency.

[0065] Now, multiple reference views are each warped to the viewpoint to be synthesized, and then the multiple predictions are blended to predict a single new synthesized view (virtual image). The following formula calculates the color of a single virtual image pixel based on the reference view image pixels from each reference view image that are mapped to that particular virtual image pixel:

number

[0066] Here, the tilde "~" above i indicates that the reference-view images were warped before being combined to form the weighted prediction. In this formula, the value w is a weighting coefficient given to each of the reference-view images or to each pixel of the reference-view images. The value in TIFF0007643456000002.tif1023 refers to the color value of a pixel in a particular reference view image, where the reference view image is given as a subscripted number. TIFF0007643456000003.tif1023 are the calculated color values ​​of the virtual image pixels to which all the reference view image pixels correspond.

[0067] The reference view image is warped before combination to determine which reference view image pixels correspond to which virtual image pixels, since the mapping between the two changes between the reference image and the desired viewpoint of the reference image. This formula must be applied for each virtual image pixel to determine its color based on its corresponding reference image pixel. The above formula simply ignores the presence of α.

[0068] A straightforward approach is to warp and weight α in a manner similar to how colors are averaged. Again, the process must be performed for each virtual pixel, calculating a value for a single pixel based on the characteristics of the reference image pixel that maps to that particular virtual pixel. Again, the value w is a weighting factor given to each of the reference-view images or to each pixel of the reference-view image. Subscripted α values ​​refer to the transparency value of a pixel in a particular reference-view image, the reference-view image being identified by the number of subscripts. Unsubscripted α is the calculated transparency value for the virtual image pixel to which each of the reference image pixels corresponds:

number

[0069] The color of the resulting virtual image pixel TIFF0007643456000005.tif1023 and transparency TIFF0007643456000006.tif1012 can be used to composite the captured multiview foreground object onto a new (graphics) background.

[0070] However, this approach to handling transparency introduces artifacts around the boundaries of foreground objects.

[0071] FIG. 2 shows two reference cameras 106, C1 and C2, for capturing a foreground object 202 in front of a green screen 102.

[0072] A naive weighted combination of the transparency values ​​of the reference views results in ghost regions 200 where the background "shines" through the foreground objects. These semi-transparent ghost regions 200 occur near the boundaries of the foreground objects 202 generated by this method.

[0073] Figure 2 shows different regions along the green screen 102. The two reference cameras 106 will see different parts of the scene when they point towards those regions. The first region is denoted B1, B2. This means that the first camera C1 will see background information B1 when shooting in that direction, and the second camera C2 will see background information B2 when shooting in that direction. The last region is denoted F1, F2. This means that the first camera C1 will see foreground information F1 when shooting in that direction, and the second camera C2 will see foreground information F2 when shooting in that direction. In other regions there are discrepancies between the reference images. U denotes an undefined region close to the border.

[0074] The 3D shape of the object can cause the ghosting problem, but depth estimation and / or filtering errors can also cause the problem.

[0075] FIG. 3 shows two elliptical images 300, 302 of a foreground object warped from two differently positioned reference cameras used to image the scene.

[0076] As in Figure 2, the combination of two letters indicates, for each warped reference view, whether the pixel is a foreground pixel (F), a background pixel (B), or an undetermined pixel (U) that features coloring from both the foreground and background. The subscript indicates which reference view image, and therefore which camera, the pixel originates from.

[0077] The reference-view image of object 202 captured by camera C1 is an ellipse 300. This is combined with a reference-view image of object 202 captured by camera C2, an ellipse 302. Regions in the virtual image where the edge of one reference-view image ellipse overlaps with the interior of the other reference-view image ellipse produce ghost regions 200.

[0078] The approach of the present invention will now be described.

[0079] FIG. 4 illustrates a method 400 for setting respective color and transparency values ​​for a plurality of virtual image pixels in a virtual image.

[0080] A plurality of reference view images of an object are received in step 402. Each of the plurality of reference view images includes foreground and background information captured from a different viewpoint.

[0081] Next, in step 403, the reference view images are mapped such that there is a mapping between the pixels of each reference image to a single virtual image. This mapping depends on the viewpoint of each reference image and may involve, for example, warping and combining the images. There is then a correspondence between the pixels of the reference images and the pixels of the virtual image. In particular, the content of each pixel of the virtual image is determined from the content of its "corresponding" (i.e. linked by a mapping) pixel from the reference images. Typically this is a many-to-one mapping, but this is not required. For example, due to occlusions, two or more input pixels may be mapped to one and the same output.

[0082] A set of steps 404 is then performed to combine the multiple reference-view images to generate a virtual image of the object, the virtual image being at a different perspective than any of the multiple reference-view images.

[0083] At step 406, transparency information is determined that includes at least foreground and background information components for each of a plurality of virtual image pixels, the components being derived from corresponding pixels of a plurality of reference-view images.

[0084] For example, one region of the image in Figure 3 is denoted as B1, F2, where B1 is the background information component obtained from reference image camera C1, and F2 is the foreground information component obtained from reference image camera C2.

[0085] For one pixel in that region, sets B1 and F2 constitute the "foreground and background information components."

[0086] In step 408, each of the plurality of virtual image pixels is classified based on a combination of the foreground and background information components.

[0087] The classification is used to determine which components should be displayed. Specifically, for each pixel in the virtual image, a component of foreground / background information is determined that is derived from corresponding pixels in multiple reference view images (i.e., based on the mapping used to derive the virtual image from the reference images). The pixels are classified according to the foreground / background information component. A transparency value can then be derived. The transparency value can have only a binary component that corresponds to "foreground" or "background" in the simplest implementation. As is clear from the above description, the transparency data may be derived from color data. The transparency value determines whether foreground or background scene data is recorded. The color of the green screen is basically measured, and the transparency value depends on how close the observed color is to the green screen color. More sophisticated methods exist.

[0088] The present invention exploits the non-linear dependence of the foreground / background / undefined information (F, B and U) components on transparency. This is, for example, the classification of each pixel of the criterion with one of three different components:

number

[0089] where F means definitely foreground, B means definitely background, and U denotes an "undetermined" pixel. Δ is the threshold transparency value against which the pixel transparency α is compared. The threshold transparency may be set depending on the object being imaged, and may be 0.5 or less.

[0090] Alternatively, the pixels may be classified according to irrelevant ranges, for example as follows:

number

[0091] In the example of chromakeying, these undetermined pixels are pixels with mixed color values, where the background color is still visible in the output color. In some examples, more classifications may be used, especially for objects with various levels of transparency. In an image with multiple foreground objects, or objects with various degrees of transparency, there may be multiple foreground classifications corresponding to each object or regions of each object.

[0092] The threshold for α can be adjusted to ensure that all possible values ​​are encompassed by F and B only, so the invention can also be practiced with only the F and B components.

[0093] In step 410, either foreground or background information is selected to be displayed by each of a plurality of virtual image pixels based on their classification. This includes selecting a transparency value from one of two binary values. Thus, the virtual image pixels are selected to binary display only foreground or background information to avoid ghosting problems, as will be further described below.

[0094] In step 412, a color value for each of a plurality of virtual image pixels is set based on the information selected to be displayed.

[0095] Referring again to FIG. 3, the ghost regions are from the following set of component combinations: C1, C2 = {(F1, U2), (U1, F2)}

[0096] This set of values ​​represents a classification of the nature of that portion of the virtual image. This classification for a particular pixel is used to select which components are displayed by the pixel by encoding an appropriate transparency value, which is used to set its respective color.

[0097] An example of an algorithm that selects the components to display and sets the colors based on the transparency information of the reference view image is shown below: if (C1==F)

[0098] if (C2==F)

[0099] Create a weighting for the reference color and set the output transparency to 1

[0100] (i.e. transparency setting = foreground)

[0101] else select F with transparency 1 (i.e. transparency setting = foreground) else

[0102] if (C2==F)

[0103] Select F with transparency 1 (i.e. transparency setting = foreground)

[0104] else

[0105] Select any color with transparency 0 (i.e. transparency setting = background)

[0106] The algorithm in this example will always return a transparency equal to 0 if: C1, C2=U1, U2

[0107] In this case, there is no need for an exact transparency estimate, as long as the classification is done correctly: the above algorithm only returns transparency values ​​of either 0 or 1 (and thus pixels classified as background or foreground), and therefore no intermediate values ​​are produced.

[0108] Figure 5 shows an example of a virtual image created by applying the above algorithm to the virtual image of Figure 3. The combined projected foreground of all reference views results in a new outer boundary that is a combination of the elliptical images 300, 302. This new outer boundary can be identified by a combination of the following classes: C1, C2= {(B1, U2), (U1, B2), (U1, U2)}

[0109] These are virtual pixels with transparency information that contains undetermined components and no foreground components.

[0110] For example, the exemplary algorithm above chooses any desired color and makes the new outer boundary pixels completely transparent to blend with the new background, however the effect of this is that the size of the foreground object is slightly reduced since pixels with only undetermined information components occurring at points around the outer edge of the object are made transparent.

[0111] Another problem is that objects appear to have sharp color transitions to the background. Foreground colors f1...f for each virtual image pixel 1...N are N and the corresponding transparency values ​​α1...α N If , ...

[0112] Thus, one embodiment of the present invention may use intermediate transparency values ​​for this boundary region. Thus, for the edge region of the object, a transparency value may be set between the limit values. Each transparency value of a pixel within this edge region may be set using the color of at least one adjacent pixel.

[0113] In this way, a new method of blending the outer boundary of the object can be implemented, which for each pixel of the virtual image uses pure foreground pixels from the adjacent, surrounding or nearby spatial neighborhood of that pixel to blend the foreground color. We form an estimate of TIFF0007643456000009.tif1134. These foreground pixels are, by definition, from the set of foreground-containing pixels in any one of the warped reference views: C1, C2= {(F1, B2), (B1, F2), (F1, F2)}

[0114] If both foregrounds are available for the pixel used in the averaging, the method can choose one color from one of the reference view images, or use the average color value of the two. The resulting transparency can be calculated from the color difference between the current (undetermined) pixel and the neighboring pixels, and / or as a weighted combination by Euclidean distance to the neighboring pixels, and / or using other averaging or weighting methods known in the art.

[0115] As we approach the edge of the object, an increasing proportion of the pixel color is the chromakey color, so the output pixel should be more transparent toward the object's boundary. This causes the object's new outer boundary to blend smoothly with the background.

[0116] The invention generalizes to using more than the two reference views used in the example, and again a combination of pixel classes derived from multiple reference views can be used to determine output color (a selection or blend of multiple colors) and output transparency settings.

[0117] The above approach is a non-linear method for processing reference image pixels with a binary selection of transparency based on pixel classification, which is described above in the context of opaque objects (optionally with fully transparent areas).

[0118] If the object has semi-transparent areas, the above approach can be modified. For example, the least transparent reference image may dominate and its transparency value is used as the output transparency value for the virtual image pixel. This makes it possible to process, for example, images of cloudy, misty or dirty windows, or images of hair. Instead of selecting the minimum transparency, a non-linear selection can be made.

[0119] A GPU implementation of the reference image combining process is shown below. TIFF0007643456000010.tif77101

[0120] This code receives 2 vector components (x, y position of the pixel) from the reference-view image and outputs a 4-vector (3 color components and a transparency component). For two reference-view image pixels t1 and t2, their opacity "a" is compared to an opacity criterion "opaque = 0.9f". If both reference image pixels are found to meet or exceed this opacity threshold, then the color of a virtual pixel t3 is set as a blend of t1 and t2, and the opacity of the virtual pixel is set to 1.0.

[0121] If both reference image pixels are not opaque according to the opacity criterion, their opacity is compared. A virtual image pixel t3 is set to match the color and opacity of the most opaque reference image pixel. In this case, the virtual image pixel can have an opacity that is neither 0 nor 1, but is an intermediate value.

[0122] 6 shows an image processing apparatus 600. An input 602 receives a number of reference view images of an object captured by the set of cameras 106.

[0123] A processor 604 processes the multiple reference view images to generate a virtual image, which is provided at an output 606, for example to a display 608. The processor implements the methods described above.

[0124] In alternative examples, the algorithm may include one or more intermediate transparency values ​​to which the pixel may be set. Additionally, the algorithm may set the transparency of the pixel where C1, C2=U1, U2 to a value other than 1, for example when creating virtual images of objects with various degrees of transparency from each viewpoint, or for objects that need to be artificially transparent.

[0125] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. Where a computer program is described above, the computer program can be stored or distributed on a suitable medium, for example an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, for example via the Internet or other wired or wireless telecommunication systems. The term "adapted to" when used in the claims or the description has the same meaning as the term "configured to". Any reference signs in the claims should not be interpreted as limiting the scope.

[0126] In general, an image processing device, an image processing method, and a computer program for implementing the method are illustrated in the following embodiments.

[0127] Embodiments:

[0128] 1. A method (400) for setting respective color and transparency values ​​for a plurality of virtual image pixels in a virtual image, the method comprising: (402) receiving a plurality of reference-view images of an opaque object, each of the plurality of reference-view images having depth information captured from a different viewpoint; (404) combining the plurality of reference-view images to generate a virtual image of the object, the virtual image being at a different viewpoint than any of the plurality of reference-view images, and each reference-view image pixel corresponding to a virtual image pixel by mapping, the generating of the virtual image comprising: (406) determining transparency information for each of the plurality of virtual image pixels, the transparency information having at least components of foreground and background information, the components being derived from corresponding pixels of the plurality of reference-view images; (408) classifying each of the plurality of virtual image pixels based on a combination of foreground and background information components; (410) For regions away from the edges of the opaque object, select a binary transparency value for each of the virtual image pixels based on their classification. (412) setting a color value for each of the plurality of virtual image pixels based on the information selected to be displayed.

[0129] 2. The method of embodiment 1, wherein each of the reference view pixels has depth information in the form of foreground information, background information or undetermined information, and the step of determining transparency information further comprises determining an undetermined component for each of the plurality of virtual image pixels.

[0130] 3. The method of embodiment 2, further comprising, for each of the plurality of virtual image pixels having transparency information as a combination of undetermined components and foreground information components, classifying the virtual image pixel as a ghost region causing a ghost effect.

[0131] 4. selecting, for each of a plurality of virtual image pixels having transparency information that includes only a foreground information component, a first binary transparency value corresponding to opacity; A method as described in embodiment 2 or 3, comprising a step of selecting a second binary transparency value corresponding to transparency for each of a plurality of virtual image pixels having transparency information including only background information components.

[0132] 5. A method according to any one of embodiments 2 to 4, comprising a step of creating a new outer boundary of the object in the virtual image by selecting a first binary transparency value corresponding to opacity for virtual image pixels having transparency information having only a combination of background information components and undetermined components or only a combination of multiple undetermined components.

[0133] 6. A method according to any one of embodiments 1 to 5, comprising a step of setting the color of the pixel to the average color of the foreground components of the reference view image for each of the plurality of virtual image pixels having transparency information that includes only foreground information components.

[0134] 7. A method according to any one of the preceding claims, comprising the step of selecting a binary transparency value for every pixel of the virtual image.

[0135] 8. A method according to any one of the preceding claims, comprising setting a transparency value between the two values ​​for areas at the edge of the object.

[0136] 9. The method of embodiment 8, comprising, for each of a plurality of virtual image pixels having transparency information that includes an undetermined component and no foreground component, setting a transparency value for each of the pixels using the color of at least one adjacent pixel.

[0137] 10. A method according to any one of embodiments 1 to 9, comprising, for each of a plurality of virtual image pixels, determining transparency information derived from the reference view image using the color of the corresponding reference view image pixel.

[0138] 11. The method of any of the preceding embodiments, wherein the background is a chromakey.

[0139] 12. A computer program comprising computer program code means adapted to perform the method according to any one of embodiments 1 to 11 when the computer program is executed on a computer.

[0140] 13. An image processing device (600), comprising: an input (602) for receiving a plurality of reference view images of an object, each of the plurality of reference view images including foreground and background information captured from a different viewpoint; a processor (604) for processing the plurality of reference view images to generate a virtual image; an output unit (606) for outputting the virtual image; The processor is adapted to perform the method according to any one of the preceding embodiments.

[0141] More specifically, the present invention is defined by the following claims.

Claims

1. 1. A method for setting respective color and transparency values ​​for a plurality of virtual image pixels in a virtual image, the method comprising: receiving a plurality of reference-view images of an opaque object, each of the plurality of reference-view images having depth information captured from a different viewpoint; combining the plurality of reference-view images to generate a virtual image of the object, the virtual image being at a different viewpoint than any of the plurality of reference-view images, and a reference-view image pixel of each of the reference-view images corresponding to a virtual image pixel of the virtual image by mapping; and the generating the virtual image step comprises: determining transparency information for each of a plurality of said reference-view image pixels, each of said reference-view image pixels including transparency information in the form of foreground information, background information or undetermined information; classifying each of the plurality of virtual image pixels based on the transparency information of a corresponding reference-view image pixel, where a ghost region classification is applied to the virtual image pixel and the corresponding reference-view image pixel has transparency information in the form of both undetermined information and foreground information; deriving transparency information for each of the virtual image pixels from corresponding pixels of the plurality of reference-view images, comprising selecting a binary transparency value for each of the virtual image pixels, for regions away from edges of the opaque object based on their classification to display foreground or background information; setting a color value for each of the plurality of virtual image pixels based on foreground or background information selected to be displayed.

2. The method of claim 1 , comprising selecting a binary transparency value for each of the virtual image pixels within the ghost region.

3. selecting a first binary transparency value corresponding to opacity for each of the plurality of virtual image pixels having corresponding transparency information that includes only foreground information; 3. The method of claim 1, further comprising: selecting a second binary transparency value corresponding to transparency for each of the plurality of virtual image pixels having corresponding transparency information that includes only background information.

4. 4. The method of claim 1, further comprising: generating a new outer boundary of the object in the virtual image by selecting a first binary transparency value that corresponds to the transparency of virtual image pixels having corresponding transparency information with both background information and undetermined information only, or with undetermined information only.

5. 5. The method of claim 1, wherein for each pixel of the plurality of virtual image pixels having corresponding transparency information that includes undetermined information and does not include foreground information, the transparency value of the respective pixel is set using a color of at least one adjacent pixel.

6. 6. The method of claim 1, wherein for each of the plurality of virtual image pixels having corresponding transparency information that includes only foreground information, the color of that virtual image pixel is set as the average color of the reference view image pixels.

7. The method of claim 1 , comprising selecting a binary transparency value for every pixel of the virtual image.

8. The method according to claim 1 , further comprising setting a transparency value between binary values ​​for areas at the edge of the object.

9. A method according to claim 1 , comprising determining the transparency information for each of the reference-view image pixels using a colour of the reference-view image pixel.

10. The method of claim 1 , wherein the background is a chromakey.

11. In the step of determining transparency information of the reference-view image pixels, The foreground information is determined by comparing transparency values ​​of the reference view image pixels to a first threshold; The background information is determined by comparing the transparency values ​​of the reference view image pixels to a second threshold; The undetermined information is determined when a transparency value of a reference-view image pixel of each of the reference-view images does not satisfy either the first threshold or the second threshold.

11. The method according to any one of claims 1 to 10.

12. A computer program adapted to carry out the method according to any one of claims 1 to 11 when it is run on a computer.

13. An image processing device, an input for receiving a plurality of reference-view images of an object, each of the plurality of reference-view images having transparency information and captured from a different viewpoint; a processor for processing the plurality of reference view images to generate a virtual image; an output unit for outputting the virtual image; 12. An image processing apparatus comprising:

Images