Method and device for reconstructing a depth image
The method uses affine transformations and a binary window dictionary to reconstruct depth images, addressing noise sensitivity and inefficiencies in existing methods, providing faster and more resource-efficient depth image reconstruction.
Patent Information
- Application Number
- FR2022013892
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-12-20
- Publication Date
- 2024-11-01
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing depth image reconstruction methods are sensitive to noise, resource-intensive, and limited to specific filter configurations, making them slow and inefficient.
A method for reconstructing a depth image using an affine transformation and a binary window dictionary, where each window is selected from a stored dictionary based on measurements, and depth values are estimated through affine operations, incorporating color and depth channel information to enhance noise robustness and efficiency.
The method effectively reconstructs depth images with reduced noise sensitivity and improved processing speed, adaptable to various filter configurations, and reduces resource consumption.
Smart Images

Figure 00000025_0000 
Figure 00000025_0001 
Figure 00000026_0000
Abstract
Description
Title of the invention: Method and device for reconstructing a depth image Technical field
[0001] The present description relates generally to the field of imagers, and more particularly to the field of depth imagers and methods of dematrixing depth images. Prior art
[0002] Imagers are often formed of pixels of several types, configured to capture different channels. However, each pixel is generally sensitive to only one channel, and for each pixel of the raw image captured by the imager, information from one or more other channels is therefore missing. For example, color-depth imagers (RGBZ) use colored filters on a portion of the pixels, separating the trichotomy information in the light. These imagers further include pixels intended to measure a depth channel. Thus, the acquired raw image is a mosaic of pixels, one portion of which contains color-related information and another portion of which contains depth-related information.
[0003] In order to reconstruct a complete depth image, the unmeasured depth values of certain pixels can then be estimated.
[0004] Methods for reconstructing a depth image from sparse measurements exist but are generally sensitive to noise occurring during depth measurements. In addition, these methods are only suitable for certain filter configurations. These methods are generally slow to execute and / or consume processing resources. Summary of the invention
[0005] One embodiment provides a method for reconstructing a depth image, by a processor of an electronic device, from measurements, carried out by an imager and according to a matrixing pattern, of at least the depth channel, the method comprising, for at least one window of the image: - the selection, by the processor and on the basis of the measurements associated with the window, of a binary window, from a dictionary of binary windows stored in a non-volatile memory; - the estimation, by the processor, of affine transformation coefficients associated with the depth, the affine transformation coefficients comprising a direction coefficient associated with the depth and an ordinate coefficient at the origin associated with the depth, on the basis of the measurements associated with the window; and - estimating, by the processor, a depth window, by applying an affine operation, on the basis of the estimated coefficients, to the binary window, the method further comprising generating, by the processor, the depth image on the basis of the at least one estimated depth window.
[0006] According to one embodiment, the raw image is covered by a plurality of windows such that each window is distant from its neighboring windows by a cutting step multiple of the size of the matrix pattern.
[0007] According to one embodiment, for each window, the binary window is further estimated on the basis of an estimation of the coefficient of the origin associated with the depth and on the basis of an a priori distribution of the direction coefficient associated with the depth.
[0008] According to one embodiment, for each window, the estimate of the ordinate coefficient at the origin is equal to the average of the depth measurements on said window.
[0009] According to one embodiment, the matrix pattern is such that the measurements associated with each window further comprise measurements of other channels, the other channels being the red, green and blue color channels and / or an infrared channel and / or a depth-correlated channel.
[0010] According to one embodiment, for each window, the binary window is further determined on the basis of an estimate of ordinate coefficients at the origin associated with the other channels and on the basis of an a priori distribution of direction coefficients associated with the other channels.
[0011] According to one embodiment, for each window and for each other channel, a window associated with the other channel is estimated by affine transformation of the estimated binary window, on the basis of the affine transformation coefficients associated with the other channel.
[0012] According to one embodiment, the estimation of the director coefficient associated with the depth is further carried out on the basis of the determined binary window.
[0013] According to one embodiment, each binary window of the binary window dictionary comprises a set of pixels assigned a minimum value and another set of pixels assigned a maximum value.
[0014] According to one embodiment, each binary window of the dictionary is associated with an occurrence value in the binary window dictionary.
[0015] According to one embodiment, an a priori law on the measurements carried out by the imager is a normal law whose covariance matrix is a diagonal matrix per block, each block having a value specific to the measurement noise of the associated channel.
[0016] One embodiment provides a method of constructing a binary window dictionary, stored in non-volatile memory, based on a plurality depth images, the method comprising: - selecting a first window by sampling a depth image from among the plurality of depth images; - binarizing the selected window by assigning a minimum value to the pixels of the window having a depth value less than a threshold value and by assigning a maximum value to the pixels of the window having a depth value greater than the threshold value; - if the binary window already exists in the dictionary, incrementing an occurrence value associated with the binary window, in the dictionary; and - if the binary window does not exist in the dictionary, storing the binary window, in non-volatile memory, in association with an initial occurrence value.
[0017] According to one embodiment, the sampling of the depth image comprises: - a random drawing of a window of a depth image; - the calculation of the standard deviation of the depth values on the drawn window; and - if the standard deviation of the depth values is zero, drawing a new window.
[0018] According to one embodiment, the threshold value is the average value of the depth values on the selected window.
[0019] One embodiment provides an electronic device comprising a memory and at least one processor, the memory storing instructions configured to implement the above method when executed by the at least one processor.
[0020] One embodiment provides a non-transient memory device configured to store instructions enabling the implementation of the above method when executed by at least one processor. Brief description of the drawings
[0021] These characteristics and advantages, as well as others, will be explained in detail in the following description of particular embodiments given without limitation in relation to the attached figures among which:
[0022] [Fig.l] is a block diagram showing an electronic device according to one embodiment of the present description;
[0023] [Fig.2] illustrates an example of an image scene captured by a color-depth imager as well as a resulting raw image;
[0024] [Fig.3] schematically represents a raw image as well as several windows covering it;
[0025] [Fig.4] illustrates the correlation between the color channels and the depth channel on a portion of raw image;
[0026] [Fig.5] illustrates the binarization of a depth window;
[0027] [Fig.6A] is a flowchart representing steps in constructing a binary window dictionary;
[0028] [Fig.6B] illustrates the selection of windows in a depth image;
[0029] [Fig.7A] represents an example of a binary window dictionary;
[0030] [Fig.7B] is a curve representing the occurrences of binary windows in a binary window dictionary
[0031] [Fig.8A] represents an example of a matrixing operation of a window of the raw image;
[0032] [Fig.8B] represents an example of a demosaicing matrix of the window illustrated in [Fig.8A];
[0033] [Fig.9] illustrates an example of different steps of a method for reconstructing a depth image according to an embodiment of the present description; and
[0034] [Fig. 10] is a flowchart representing different steps of a method for reconstructing a depth image, according to an embodiment of the present description. Description of the embodiments
[0035] The same elements have been designated by the same references in the different figures. In particular, the structural and / or functional elements common to the different embodiments may have the same references and may have identical structural, dimensional and material properties.
[0036] For the sake of clarity, only the steps and elements useful for understanding the described embodiments have been shown and are detailed. In particular, the operation of the imagers is not described in detail. The methods and devices for measuring depth, or color channels, are known to those skilled in the art.
[0037] Unless otherwise specified, when referring to two elements connected to each other, this means directly connected without intermediate elements other than conductors, and when referring to two elements connected (in English "coupled") to each other, this means that these two elements can be connected or be connected by means of one or more other elements.
[0038] In the following description, when reference is made to absolute position qualifiers, such as the terms "front", "back", "top", "bottom", "left", "right", etc., or relative position qualifiers, such as the terms "above", "below", "upper", "lower", etc., or to orientation qualifiers, such as the terms "horizontal", "vertical", etc., reference is made unless otherwise specified to the orientation of the figures.
[0039] Unless otherwise specified, the expressions "about", "approximately", "substantially", and "of the order of" mean to within 10%, preferably to within 5%.
[0040] [Fig.l] is a block diagram showing an electronic device 100 according to an embodiment of the present description.
[0041] By way of example, the electronic device 100 comprises an imager 102 (IMAGE SENSOR), and in this case, the electronic device 100 is a digital camera, a smartphone comprising a photographic functionality, etc. In another example, the imager 102 is not part of the electronic device 100, and in this case the device 100 is for example an image processing device, such as a computer.
[0042] The imager 102 is for example a color-depth imager (RGBZ) comprising color pixels configured to measure information related to the color of an image scene, for example by measuring the wavelengths associated with the colors red, green or blue. A color pixel then allows the measurement of a single channel from among one of the three color channels red (r), green (g) and blue (b). The color-depth imager further comprises depth pixels (d) configured to measure the distance at which an object, or a target, is located in the image scene.
[0043] In another example, the imager 102 is a depth imager, comprising only depth pixels. In yet another example, the imager 102 comprises, in addition to depth pixels, infrared (IR) pixels configured to measure wavelengths belonging to the infrared range.
[0044] The imager 102 is for example connected to an integrated circuit 104, via a bus 105.
[0045] The circuit 104 comprises, for example, a non-volatile memory 106 (NV MEM) and a volatile memory 108 (RAM), for example a random access memory (in English “Random Memory Access”). The circuit 104 further comprises a processor 110 (CPU), for example connected to the memories 106 and 108 via a bus 112.
[0046] By way of example, the imager 102 is configured to store the measurements made in the volatile memory 108 or in the non-volatile memory 106. The processor 110 is for example configured to execute instructions, for example, stored in the non-volatile memory 106 and / or in the volatile memory 108 in order to reconstruct a depth image from the stored measurements. Once reconstructed, the depth image is for example stored in the non-volatile memory 106 and / or in the volatile memory 108.
[0047] [Fig. 2] illustrates an example of an image scene 200 captured, for example, by the imager 102. The image scene 200 is sub-represented by two images 201 and 201'. Image 201 illustrates the colors (in grayscale in [Fig. 2]) of the image scene 200 and image 201' illustrates the depth of the image scene 200. Images 201 and 201' come from a public database and are the subject of the publication "High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth." published in 2014 in German Conference on Pattern Recognition, 31-42, Springer by Scharstein, Daniel, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nesic, Xi Wang, and Porter Westling.
[0048] In this example, the imager 102 is a color-depth imager, comprising certain pixels measuring a color component and other pixels measuring the depth. The arrangement of the different types of pixels is determined by a matrix pattern 202 replicated on the surface of the imager 102. A raw image 204, or mosaic, is obtained, directly at the output of the imager 102 and is for example stored in the non-volatile memory 106 or in the memory 108. Each pixel of the raw image 204 comprises a measurement of a single channel among for example the red, green, blue and depth channels.
[0049] In the following, we will note the position of the pixels of the raw and reconstructed images, as well as the position of the pixels of the imager 102 and of the matrix patterns by a value ( / , j ) where i and j are respectively the rank of the row and of the column of the pixel, the position (1,1) being the position of the pixel at the top left of the object considered.
[0050] Thus, in the example of Figure 2, the pixels of the imager in position (ù j), where i and j are odd, are configured to measure the green component (g) of the image scene. The pixels in position ( j ) where i is odd and j is even, are configured to measure the red component (r) of the image scene 200. The pixels in position (i, j), where i is even and j is odd, are configured to measure the blue component (b) of the image scene 200. The pixels in position ( f, y), where i and j are even, are configured to measure the depth (d). This sampling configuration of the image scene 200 is, for example, entirely determined by the matrix pattern 202. In the example of Figure 2, the matrix pattern 202 is of size 4x4. The filters on the pixels at positions (1,1), (1,3), (3,1) and (3,3) of pattern 202 allow the green component of the color to be separated.The filters on the pixels at positions (1,2), (1,4), (3,2) and (3,4) allow the red component of the color to be separated. The filters on the pixels at positions (2,1), (2,3), (4,1) and (4,3) allow the blue component of the color to be separated. The pixels at positions (2,2), (2,4), (4,2) and (4,4) of the pattern 202 are depth pixels.
[0051] Figure 3 schematically represents the raw image 204 as well as several windows covering it. For example, the reconstruction of the raw image 204 into a depth image is carried out following window-by-window processing of the raw image 204. Each window is an extraction of a part of the raw image 204 and contains the measurements taken by the imager 102 for this area. Each window is for example of size xw, w being an integer greater than or equal to 2. In other examples, the windows are of different shapes, for example rectangular or example in the shape of an L, provided that all the windows cover the entire raw image 204. In other words, whatever the shape of the windows, the arrangement of the windows is for example chosen so that any pixel of the raw image 204 is covered by at least one window.
[0052] In the example illustrated by Figure 3, windows of size 9x9 are extracted from the raw image 204, according to a cutting step equal to 4. The cutting step represents the distance, in number of pixels, between two successive, or neighboring, windows. A window 300 comprises, for example, the measurements of the pixels of the raw image 204 having the position (i, j), 1 j - 9. By way of example, the cutting step is identical over the entire raw image 204. Another window 302, successive to the window 300, comprises, for example, the measurements of the pixels of the raw image 204 having the position ( j ), 5 < i, / < 13. The cutting step is for example identical vertically and horizontally. Thus, a window 304 includes, for example, the measurements of the pixels of the raw image 204 having the position (i, j), 5 < i < 13 and 1 < j < 9.Each window is for example processed, for example by the processor 110, in order to obtain a depth window comprising estimates of the depth values for each pixel. A depth image is then progressively reconstructed from the depth windows.
[0053] For example, when the cutting step is less than w, two successive windows comprise common pixels. For example, the pixels in position (i, j), 1 z - 9 and 5 < j < 9, are common to the windows 300 and 302. For example, a final depth value for the pixels common to several windows is obtained by averaging the estimates for said pixel on each window.
[0054] In other examples, the clipping step varies in the processing of the raw image 204, provided that all of the windows cover the entire raw image 204. In other words, whatever the values of the clipping step, any pixel of the raw image 204 is covered by at least one window.
[0055] Figure 4 illustrates the correlation between the color channels and the depth channel on a window 400 of the raw image 204. In particular, the window 400 comprises the measurements of the raw image 204 from the imager 102. For example, the imager 102 is an RGBZ imager and the matrix pattern used is for example the pattern 202. Thus, the pixels of the window 400 in positions (i, J), where i and j are even, comprise measurements related to depth. The other pixels comprise color measurements.
[0056] Windows 402 (R), 404 (G), 406 (B) and 408 (D) correspond respectively to the ground truths of the red, green, blue and depth components.
[0057] Window 404 has a slightly pronounced outline, i.e. a visual variation of the channel values, separating the left and right sides of window 404. Window 406 has a very pronounced outline separating the left and right sides of window 406.
[0058] Similarly, window 408 shows an outline in depth. A difference in depth exists between the left and right sides of the window, and this outline maps onto the outlines appearing for the green and blue components.
[0059] According to one embodiment, the information related to the colors in the mosaic 400 is used in order to reconstruct a depth window.
[0060] According to one embodiment, when the size of the windows is relatively small, a depth window, such as window 408, is approximated by a window comprising only two depth values. In other words, the contour in a small depth window, for example of length less than 15 pixels, is sufficiently marked to cut the image into two parts, one part comprising pixels having the same depth values and another part comprising pixels also having other same depth values.In another example, the outline in window 408 does not demarcate two areas in which the pixels have exactly the same depth value but depth values relatively close to each other, the variability being for example due to image measurement noise, the difference between the depth values of the two areas being greater than the variability of the depth values in the same area.
[0061] [Fig.5] illustrates the binarization of the depth window 408. For example, the depth values of the pixels of the window 408 are included in the interval [0,1] plus or minus a margin of error due to measurement noise. For example, the pixels in the left part of the window 408 have a depth value lower than the value of the pixels in the right part of the window 408. The window 408 is then approximated by an affine transformation of a binary window 500, the binary window 500 comprising the same contour as the window 408. The pixels on the left side of the binary window 500 (illustrated in white) have for example the depth value 1 and the pixels to the right of the binary window 500 (illustrated in black) have for example the depth value -LA For example, the depth value of the pixels on the left side of the window 408 is for example equal to -0.01.For example, when depth values are strictly less than 0 or strictly greater than 1, they are brought back into the interval [0,1]. For example, all depth values are rescaled so that they all belong to the interval [0,1]. In another example, values less than 0 are brought back to the value 0 and values greater than 1 are brought back to the value 1. and the depth value of the pixels on the right side of the window 408 is equal to 0.59. Each . The depth value of a pixel in the 408 window is then equal to -0.3 times the depth value of the pixel in the same position in the 500 binary window plus 0.29. The value 0.29 corresponds, for example, to the average of the depth values across the entire 408 window.
[0062] According to one embodiment, each of the two zones of the window 408 does not comprise pixels having exactly the same depth value. A relatively low variability, linked to the measurement noise, of the depth values on each zone then exists. In this case, the affine transformation of the binary window 500 is an approximation, or an estimation, of the window 408. However, when the size of the window 408 is small, for example 9x9 pixels in the examples illustrated by figures 4 and 5, the variability within the same zone is sufficiently low so that the error in the affine approximation is negligible. However, by choosing a cutting step of the raw image so as to estimate several times the depth values of the same pixel and to take the average value over all the windows covering it, the reconstructed image will be smoothed and the error, already negligible, will be reduced.
[0063] The coefficients -0.3 and 0.29 are given as examples, and in relation to the window 408. In the following, the leading coefficient will be designated by and the ordinate at the origin by bd. Another affine transformation of the same binary window 500 approximates for example the window 406, the affine transformation being for example carried out with coefficients ab and bh. Another affine transformation, for example on the basis of coefficients as and bg, of the binary window 500 approximates for example the window 404. The window 402 being uniform, it is also approximated for example as being an affine transformation of the binary window 500, on the basis of coefficients ar = 0 and br, br being the average of the values of the pixels of the window 402, that is to say, the average red value on the window 402.
[0064] In certain embodiments, the binarization of the window 408 uses as a threshold, i.e. as a value below which the pixels are assigned the value -1 and above which the pixels are assigned the value 1, the average value of the depth values on the window 408. The threshold value makes it possible to draw the outline in the binary window 500.
[0065] Although, in the example illustrated by [Fig.5], the depth values of the pixels of the window 408 have been fixed as belonging to the interval [0,1], it is entirely possible to consider another measurement scale. Similarly, the values -1 and 1 assigned to the pixels of the binary window 500 are presented as an example and are not limiting. As an example, the values 0 and 255 are assigned to the pixels of the binary window.
[0066] According to one embodiment, a dictionary comprising a set of windows binaries is for example stored in the non-volatile memory 106. In another example, the dictionary is stored in a memory external to the device 100, such as for example in a server.
[0067] According to one embodiment, for each window of the raw image, a binary window is for example selected in the dictionary in order to estimate, following an affine transformation, a depth window.
[0068] [Fig.6A] is a flowchart representing steps in constructing a dictionary of binary windows. For example, the dictionary 600 is constructed upstream of the manufacture of the device 100. The construction of the binary dictionary is for example carried out by a device similar to the device 100 and for example provided with a depth imager.
[0069] In a step 600 (DEPTH IMAGES), a plurality of depth images, for example captured beforehand by a depth imager, are for example stored in a memory. For example, a length w, describing the length of the binary windows is also stored in the memory. The length w is the same as the length of the windows used for processing the raw image 204 by the processor 110.
[0070] In a step 601 (i=l), a variable i is initialized, for example to 1.
[0071] In a step 602 (RANDOM PATCH), subsequent to step 600, a window of size wxw pixels is drawn, for example randomly from one of the depth images. In another example, the depth images are scanned window by window, deterministically. A first implementation of step 602 then consists of taking the first window of a first depth image, each new implementation of step 602 taking the window successive to the last window considered.
[0072] Embodiments of step 602 are for example illustrated in [Fig.6B]. From a depth image 611, windows 612, 614, 616 and 618 are for example randomly drawn during four different embodiments of step 602.
[0073] In a step 603 (sd=0?), the standard deviation of the depth values on the selected window is then calculated. If the standard deviation is zero (Y branch), this means that there is no difference in depth in the window. The method then resumes in an embodiment of step 602, in which a new depth window is considered. For example, the standard deviation on the depth values of window 618 is zero.
[0074] In the case where the standard deviation of the depth values is not zero (branch N), this means that the window does indeed include a contour and the method continues in a step 604 (BINARIZATION). For example, windows 612, 614 and 616 include a contour. In step 604, the depth window is binarized. For example, Pixels with a value less than a threshold value are assigned a minimum value, for example, the value -1, and pixels with a depth value greater than the threshold value are assigned a maximum value, for example, the value 1. For example, the threshold value is the average value of the depth values across the entire window.
[0075] In a step 605 (ALREADY EXITS?), the binary window generated in step 604 is compared to a set of binary windows, of the same size, included in a non-volatile memory. If the binary window corresponds to another window of the set (Y branch), an occurrence value, associated with the window of the set, is incremented in a step 606 (INCREMENT). If the binary window generated in step 604 does not correspond to any of the binary windows of the set (N branch), the method continues in a step 607 (CREATE) in which the window is stored in the memory, in association with an initial occurrence value, for example the value 1. The set of binary windows then constitutes the dictionary of binary windows mentioned above, and each binary window added in step 607 then constitutes a new entry in the dictionary.
[0076] Following the completion of step 606 or 607, the method continues in a step 608 (END?) in which it is determined whether the dictionary is complete or not. For example, the dictionary is complete if the sum of all the occurrence values is equal to a value determining the size of the dictionary. This size is for example fixed upstream. For example, the desired size of the dictionary is reached when the value of the variable i described in relation to step 601 is equal to a maximum value fixed upstream. In another example, the dictionary is complete when it contains a given number of binary windows.
[0077] If the dictionary is complete (branch Y), the method ends in a step 609 (END). The binary window dictionary is then constructed and is for example ready to be stored in the non-volatile memory 106 of the device 100.
[0078] If, during step 608, it is determined that the dictionary is not complete (branch N), the method continues, for example, in a step 610 (i+1) in which the variable i is incremented and the method resumes in a new implementation of step 602.
[0079] The occurrence value associated with each binary window then represents the number of times that the binary window was generated during the implementations of step 604.
[0080] For example, following the creation of the dictionary, the occurrence values associated with each binary window are divided by the sum of all occurrence values. Thus, a probability associated with each binary window is obtained.
[0081] [Fig.7A] represents a dictionary 700 of binary windows, for example stored in the memory 106, and constructed according to the embodiment described in relation to [Fig.6A]. In the example of [Fig.7A], the dictionary 700 comprises a set of 100 binary windows. Each binary window is for example associated with an occurrence value. For example, a binary window 702 representing a horizontal contour is associated with an occurrence value equal to 1204, meaning that this window has for example been obtained 1244 times during the implementation of the method described in relation to [Fig.6A]. Another window 704 representing a vertical contour is associated with an occurrence value equal to 1001, meaning that this window has for example been obtained 1001 times during the implementation of the method described in relation to [Fig.6A]. Another binary window 706, representing a relatively diagonal contour, is associated with the occurrence value 85, meaning that this window has for example been obtained 85 times during the implementation of the method described in relation to [Fig.6A].The higher the occurrence value, the more the associated binary window represents a contour that appears relatively often in a depth image. For example, horizontal and vertical contours are the contours that appear most often in depth images. Different contours, for example, with diagonal parts, or in a V-shape, are less frequent in depth images.
[0082] [Fig.7B] is a curve 708 representing the occurrences of binary windows in a dictionary comprising 12000 binary windows.
[0083] For example, binary windows are ordered (POSITION) from the largest to the smallest occurrence value (OCCURRENCE). Thus, the binary window with the largest occurrence in the dictionary is a window with a clear horizontal outline. The following windows are, for example, rotations of windows with a vertical or horizontal outline. Windows with more specific and less frequent shapes arrive at the end of the dictionary.
[0084] Figures 7A and 7B illustrate examples of binary window dictionaries of size wx w. If the windows covering the raw image 204 are chosen to have different shapes, for example rectangular, or for example L-shaped, the binary window dictionary will comprise windows of the shape chosen for processing the raw image 204. The method described in relation to Figure 6 is easily adaptable to windows other than square. Indeed, only step 602 needs to be adapted and it is sufficient to select windows of the desired shape and size in the depth images.
[0085] Figure 8A shows an example of a matrixing operation of a 3x3 window of a raw image.
[0086] For example, a matrix pattern 800, of size 3x3, is used to capture a raw image. The matrix pattern 800 is for example a pattern of a color-depth imager. The matrix pattern 800 is such that the green components of the pixels at positions (1,1), (1,3) (3,1) and (3,3) are measured. The components red components of the pixels at position (1,2) and (3,2) and the blue components of the pixels at position (2,1) and (2,3) are also measured. The matrix pattern 800 further allows the measurement of the depth of the pixel at position (2,2).
[0087] As an example, the measurements carried out are grouped in a matrix 802 Al A ^22 Have
[0088] As described in connection with Figure 2, only sparse measurements of the channels colors and depth channel are realized. Thus the ground reality of the 4 channels for all the pixels in the window are represented by an 804 matrix. The 804 matrix includes 4 sub-matrices ■ni ri2 ^13' 9 • apa $11 $12 $13 9 AA A3 and r21 r22 r23 ^21 A ^23 ^2! ^22 A3 .r31 r32 r33. Al A A. .Al A 2 A3. 12 ^13 grouping the field values of the components respectively ^22 ^23 AA Al ^32 ^33. red, green, blue and depth values for the window. Field values assume that the measured values are without measurement noise.
[0089] As an example, the dematrixing of the matrix 802 aims to estimate the sub- depth matrix 'd «na 12 u13 Ai AA .Al A3? J33 . In particular, the estimated values are by denoised example.
[0090] [Fig.8B] represents an example of a demosaicing matrix of the window illustrated in [Fig.8A].
[0091] The measurements of the matrix 802 are arranged in a vector m, thus comprising the measurements carried out by the imager 102 according to the matrix pattern 800 on a window. The vector m is therefore of size H'2 = 9. The vector m concatenates the measurements for each channel, thus rn — [ md, md ] where mr, mb and md are vectors comprising the measurements of the red, green, blue and depth channels respectively. In the example of the 800 matrix pattern and a 3 x 3 window, the md vector comprises a single value.
[0092] As an example, the ground values of the matrix 804 are arranged in a vector x of size 4x9= 36. The vector x concatenates, for example, the values of the 4 sub-matrices of the matrix 804, rearranged into a vector. In other words, x = [ xr, Xq, X^, Xd ], rll r21 ^131, etc. A matriculation matrix M is such that the relation Mx = m r23 .r31 r32 r33. is verified. The matrix M is a block-diagonal matrix comprising 4 blocks Mr, Mg, Mb and Md. The blocks Mr, Mh and Md allow respectively to select, among the "terrain" values, the values which were measured for the red, green and blue channels and for the depth channel. The blocks Mr, Ms, Mb and Md are respectively matrices of size 2 x 9, 4 x 9, 2 x 9 and 1 x 9. More generally, each block has a size equal to the number of pixels measuring the channel concerned in the window times w2. The blocks Mg, Mb and Md are then such that Mrxr - mr, MsXg = mg, = mb and Mdxd - md
[0093] The matrix M, and in particular the sub-matrices Mr, Mg, Mh and Md, are easily adaptable to the matrixing pattern used by the imager 102. In particular, the matrix M is adaptable to matrixing patterns comprising filters other than color and / or depth filters, such as for example infrared filters. In the example, where the imager 102 is an imager configured to also take measurements in the infrared range, the matrix M comprises for example a sub-matrix Mir in addition to, or instead of, one of the matrices Mr, Mg, Mb and / or Md.
[0094] For example, when the cutting step of the raw image is a fixed value, not varying in the processing of the raw image, the matrix M and in particular the sub-matrices M,-, Mg, Mb and Md are identical for each window of the raw image. In another example, the cutting step varies in the processing of the raw image, and the matrix M, in particular the sub-matrices Mr, Mg, Mb and Md, vary from one window to another.
[0095] Generally, the imager 102 introduces noise into the measurements. Thus, considering C the set of measured channels, for example C = {r, g, h, d], r8,b designating respectively the red, green and blue channels and d designating the depth, each measurement mi, i^C is such that = Mixi + where xi is the part of the vector x comprising the values associated with channel i. For example, mi is of size kb kt being the number of measurements of channel i on a window. The variable £i represents the noise linked to the measurement. In particular, each matrix M; is of size k, x w2. For example, £i is modeled by a centered normal random variable with a covariance matrix equal to tflk, (7? being the variance of the noise induced by the imager for channel i. The distinction of modeling noise by channel makes it possible to take into account the generally higher importance of noise in measurements of the depth channel compared to, for example, color channels.
[0096] [Fig.9] illustrates different stages of a process for reconstructing a depth image.
[0097] [Fig. 10] is a flowchart representing different steps of the method of reconstructing a depth image. According to one embodiment, the method is implemented by the device 100.
[0098] In particular, figures 9 and 10 illustrate the reconstruction of a depth image from a raw image captured by the imager 102. By way of example, the raw image is stored in the memory 106 or in the memory 108.
[0099] Although in the example illustrated by [Fig.9], the raw image is captured from the matrix pattern 202, the image reconstruction method described in relation to FIGS. 9 and 10 adapts to any other matrix pattern comprising one or more channels from, for example, color, depth and / or infrared.
[0100] In the following, each window in the raw image is designated by its position (p, q), where l <p<Petl Q, P étant le nombre de rangées, ou lignes de fenêtres et Q étant le nombre de colonnes de fenêtres sur l’image brute. Par convention, la fenêtre en position (1,1) est la fenêtre en haut à gauche de l’image brute. Tout autre système de classification, ou de numérotation des fenêtres, peut bien sûr être utilisé.
[0101] In a step 1000 ( ( p. q ) = (1,1)), an initial window position (p, q) is for example determined. For example, the initial position is the position (1,1) - In another example, the initial position is the position (P, Q).
[0102] In a step 1001 (PATCH POSITION (p, q)), the window of the raw image in position (p, q) is for example cut by the processor 110. As an example, the window in position (1, 1) is the window 400 described in relation to [Fig. 4] comprising measurements of the color and depth channels according to the matrixing pattern 202. The windows 402, 404, 406 and 408 represent the ground realities, not known, of the color and depth channels.
[0103] In a step 1002 (DETERMINATION OF BINARY PATCH), a binary window in the dictionary of binary windows is selected. The purpose of the selection is to determine the binary window of the dictionary that most closely resembles the ground reality 408. The selection is for example carried out by the processor 110 and from the window 400. For example, the binary window selected by the processor 110 from the window 400 is the binary window 500.
[0104] According to one embodiment, step 1002 consists of determining an estimator z in the dictionary of binary windows approximating xd to within an affine transformation. In other words, step 1002 consists of determining the binary window z most faithfully verifying xd ™ a^z + b^ where ad and bd are the coefficients of the affine transformation associated with the depth.
[0105] According to one embodiment, for each channel i GC, the estimator ê is also such that Xj ~ aZ 4- bj. The estimator z is therefore the same for each channel, only the coefficients affine coefficients ai and b; vary from one channel to another. The z estimator also allows the reconstruction of the other channels.
[0106] For example, for a window with no depth variation, the estimator z represents, for example, contours in one of the color channels. The coefficient ad is then equal to zero and the coefficient bd is equal to the average of the depth values over the window.
[0107] By way of example, the processor 110 is further configured to estimate, in a step 1003 (ESTIMATION OF COEFFICIENTS), estimators àd and £ of the affine transformation coefficients associated with the depth.
[0108] As an example, an estimator of the coefficient bd is, first of all, such that [Math 1] £ _ imjjj kd being the number of measurements of the depth pixel in the window in position ( p. q ).
[0109] In one example, the cutting step of the raw image is chosen so that each window has the same filter configuration, the kd value then does not vary from one window to another. In the example of window 400, the value of kd is equal to 16. In another example, the cutting step varies in the processing of the raw image and the filter configuration differs from one window to another. Consequently, the kd value varies from one window to another and it is appropriate, for example, to recalculate it each time a window is processed.
[0110] Similarly, for any channel ie C, an estimator of the coefficient b, is such that [Math 2] î _ ir • i ki being the number of measurements of the depth pixel in the window at position (p, q}.
[0111] The vector measures m = [md ], then verifies the following matrix equation: [Math 3] m — MZa+ Z, where M is the block-diagonal matrix, in other words 'Mr 0 0 ■■■ M. 0 ' . The matrix Z is such that t-; O * 0 ■■■ 0' eta M = . o ■ B 0 0 Md, Z = .0 ■ 0 0 is the vector comprising the direction coefficients: a = [ân ad]- The matrix Z and the vector a are therefore unknown and the purpose of steps 1002 and 1003 is to calculate an estimator for them. For example, only the element ad of the vector a is estimated. The variable £ is for example a random vector of law jq £ j, where is a vector comprising the estimators of the coefficients at the origin: £ — The covariance matrix 52 is a block diagonal matrix, each block being equal to a covariance matrix E, of size x kh ie C, in other words, 0 ■■■ 0'. 0 Eg: ! 0 ,0 ■■■ 0 Ldt
[0112]
[0113]
[0114]
[0115] When performing step 1002, the vector a is modeled by a random vector following a normal distribution, centered and with covariance matrix A, where with, for all ie C, the a priori standard deviation for the variable «2 () A = o 4 ai. The values of the a priori standard deviations are for example adjusted to the values which can be taken by the pixels of the binary window. For example, the a priori standard deviations are equal to zero when the binary windows are described by pixels equal to -1 and 1. (Correct?). In another example, the binary windows are described by pixels being equal to 0 or 255 and the a priori standard deviations of the vector a are equal to 128. These examples of values for the a priori standard deviations are given as an example and are of course not limiting, the person skilled in the art will know how to choose relevant a priori standard deviation values for the description of the chosen model. The vector a is thus marginalized and an estimator z belonging to a dictionary D of binary windows is for example determined, by the processor 110, following the calculation of [Math 4] z - argmax^p ( z ) f(( m - MZa J z, a ) pa ( a ) da, where'for all ze D'P (z ) is the probability-renormalized occurrence of the window in the dictionary D. The distributions P^. and Pa represent respectively the distributions associated with the random vectors e and a. Thus, p (ni- MZa jz, a) corresponds to the probability that the measure m is equal to MZa + e, knowing the values of and a. For example, the estimated binary window for window 400 is binary window 500. Once the estimator 2 is determined, the method remains in step 1003 in order to determine an estimator αd of aa. The estimator αd is obtained from the estimator z determined in step 1002. In particular, the processor 110 is for example further configured to determine the estimator àd by calculating [Math 5] ad = argmaxü(ip^ ( md - adMd^id> ê ) pOd ( ad ), where P,d and 7¼ represent respectively the distributions associated with the random variables e<.t and ad. In other words, P?d is the distribution associated with a normal distribution with mean and covariance matrix of / U / The distribution Pad is associated with a centered normal distribution with variance
[0116] Optionally, estimators àr, and ah of the coefficients ar, a% and ab are calculated in the same way. In other words, for all i 6 C, [Math 6] âi = argmaXaj)^(- cqM^fp, z)pa{a,), WHERE P^ and Pa, respectively represent the distributions associated with the random variables £i and aZ In other words, Pe, is the distribution associated with a normal law with mean and covariance matrix <7? The distribution P^ is associated with a centered normal law and variance a?.
[0117] In a step 1004 (DEPTH PATCH ESTIMATION), the depth values of the pixels associated with the window in position (p, q), for example window 400, are estimated.
[0118] An estimate xd of the depth values for the window at position ( p, q ] is a vector of size H'2, the first w elements of the vector being respectively, for example, the estimates associated with the w pixels of the first row of the window.
[0119] According to one embodiment, the estimator is calculated, for example by the processor 110, by affine transformation of the estimated binary window and according to the relation [Math 7]
[0120] A depth window is for example reconstructed from the xd-estimator
[0121] As an example, the binary transformation of the binary window 500 results in a depth window 900.
[0122] Following step 1004, the method continues in a step 1005 (DEPTH PATCH IN IMAGE), in which the depth window is for example pasted into a depth image, for example stored in the non-volatile memory 106. By way of example, the depth window is pasted in position (p, q) of the depth image.
[0123] As an example, the depth window 900 is pasted into a depth image 902. The depth image 902 is for example initially an image empty.
[0124] The method then continues in a step 1006 ({p, q} = (P,Q} ?) in which it is determined whether at least one other depth window remains to be estimated. For example, the values P and 4 of the current position (p. q) are compared to the values P and Q. In the case where at least one other depth window is to be estimated (branch N), the method continues in a step 1007 ((p,q) INCREMENTATION) in which another position is considered. For example, the value P and / or the value *7 is incremented.
[0125] As an example, the raw image is scanned from top to bottom and from left to right and the first window is the window in position (1,1). In another example, the raw image is scanned from bottom to top and from right to left, and the first window is then for example the window in position (P, Q). These examples are of course not limiting and the raw image can be scanned in different ways.
[0126] If, when performing step 1007, it is determined that all depth windows have been estimated (Y branch), the method ends in a step 1008 (END).
[0127] For example, during step 1008, a final depth image is for example reconstructed. For example, in the case where a pixel is covered by several depth windows, a final estimate of the depth value associated with it is obtained by averaging the depth values estimated for this pixel on each window.
[0128] For example, the coefficients ar, as and ab are not calculated. However, the measurements of the color channels are used in the calculation of the estimator j. Thus, the color-related information is used to estimate the depth values.
[0129] As an example, following the estimations of the coefficients / e C, the calculation of the estimator z, during step 1002, is simplified and rewritten [Math 8] A / x IV / / a \T \2 z = argmax logp (2)+2 \ ( mi+ U: ) / •
[0130] Similarly, the expression of the estimator of the coefficient ad is simplified and rewritten [Math 9] A _ üd ~
[0131] The values previously chosen for the standard deviations of the measurement noises ai, i.e. C , depend on the dynamics of the measurements of each channel. For example, if the range of values of the measurements is the interval [0,1], the standard deviations ai are generally of the order of 1Q'3 if the noise is low, or of the order of IQr1 if the noise is greater. For example, the values and Gb linked to the color channels are the order of 10'3 and the value °d linked to the depth channel is of the order of ] 0*!-
[0132] The method described in relation to figures 9 and 10 of reconstruction of a depth image exploits the fact that a visible contour in the depth image is generally also visible in one of the color channels. The method also exploits a representation of the contours by a dictionary of binary windows, which reinforces the robustness of the method in the face of noise present in the measurement of the depth channel.
[0133] Although the method described in relation to Figure 10 is described for the color-depth channels, it is of course possible to consider another set of channels. For example, only the depth channel is part of the set C. In this case, the method described acts as a method for denoising or resampling the depth image. In other cases, a channel other than, or in addition to, the color channels can be used. For example, the infrared channel can be part of the set C. Generally, any channel having a correlation with the depth channel can be used. The person skilled in the art will know how to adapt the matrix M to the set of channels measured by the imager. Similarly, the person skilled in the art will know how to describe the a priori law, and in particular quantify the variance of the measurement noise, on the measured channels.
[0134] Furthermore, the representation in the form of binary windows destroys the texture information present in the color channels, and this makes it possible to preserve only the salient contours which are more likely to be linked to a change in depth.
[0135] An advantage of the described embodiments is that the estimation of depth values is mainly based on searching for a binary window in a previously constructed binary window dictionary. Once the binary window and the depth-related steering coefficient have been estimated, the operations simply boil down to multiplications of coefficients by the values of the pixels in the binary window, for example -1 or 1.
[0136] Another advantage of the described embodiments is that they are adaptable to any type of matrix pattern. The matrix matrix M adapts, in fact, very easily to the matrix pattern used by the imager.
[0137] Another advantage of the described embodiments is that they can be configured according to the desired computational complexity. Indeed, the size of the windows H', the cutting step s, as well as the size of the dictionary are decisive in the reconstruction quality.
[0138] Various embodiments and variations have been described. Those skilled in the art will understand that certain features of these various embodiments and variations could be combined, and other variations will occur to those skilled in the art. In particular, channels other than the color channels may be measured. By way of example, the imager is further configured to perform measurements in the infrared domain and the infrared channel is one of the measured channels.
[0139] Finally, the practical implementation of the embodiments and variants described is within the reach of the person skilled in the art from the functional indications given above. In particular, several imagers can be used to acquire the image, the person skilled in the art will know how to reposition the measured pixels on the same optical axis.
Claims
Claims
1. Method for reconstructing a depth image, by a processor (110) of an electronic device (100), from measurements (m), carried out by an imager (102) and according to a matrixing pattern (202, 800), of at least the depth channel (d), the method comprising, for at least one window (400) of the image: - the selection, by the processor and on the basis of the measurements associated with the window, of a binary window (500), from a dictionary (700) of binary windows stored in a non-volatile memory (106); - the estimation, by the processor, of affine transformation coefficients associated with the depth, the affine transformation coefficients comprising a director coefficient associated with the depth and an ordinate coefficient at the origin associated with the depth, on the basis of the measurements associated with the window;and - estimating, by the processor, a depth window (900), by applying an affine operation, based on the estimated coefficients, to the binary window, the method further comprising generating, by the processor, the depth image based on the at least one estimated depth window.;
2. The method of claim 1, wherein the image is covered by a plurality of windows (300, 302, 304, 306, 308, 310, 312, 400) such that each window is spaced from its neighboring windows by a clipping pitch(s) multiple of the size of the matrix pattern (202, 800).
3. The method of claim 1 or 2, wherein, for each window (400), the binary window (500) is further estimated based on an estimate of the depth-associated intercept coefficient and based on a priori distribution of the depth-associated leading coefficient.
4. Method according to claim 3, in which, for each window (400), the estimate of the coefficient of ordinate at the origin is equal to the average of the depth measurements on said window.
5. A method according to any one of claims 1 to 4, wherein the matrixing pattern (202, 800) is such that the measurements associated with each window (400) further comprise measurements of other channels, the other channels being the red, green and blue color channels and / or of a infrared channel and / or a depth-correlated channel.
6. The method of claim 5, wherein, for each window (400), the binary window (500) is further determined based on an estimate of intercept coefficients associated with the other channels and based on a priori distribution of direction coefficients associated with the other channels.
7. The method of claim 6, wherein, for each window (400) and for each other channel, a window associated with the other channel is estimated by affine transformation of the estimated binary window, based on the affine transformation coefficients associated with the other channel.
8. Method according to any one of claims 1 to 7, in which the estimation of the direction coefficient associated with the depth (d) is further carried out on the basis of the determined binary window (500).
9. A method according to any one of claims 1 to 8, wherein each binary window (702, 704, 706) of the dictionary (700) of binary windows comprises a set of pixels assigned a minimum value and another set of pixels assigned a maximum value.
10. A method according to any one of claims 1 to 9, wherein each binary window (702, 704, 706) of the dictionary (700) is associated with an occurrence value in the binary window dictionary.
11. Method according to any one of claims 1 to 10, in which a priori law on the measurements carried out by the imager (102) is a normal law whose covariance matrix is a block diagonal matrix, each block having a value specific to the measurement noise of the associated channel.
12. A method comprising: constructing a dictionary (700) of binary windows, stored in a non-volatile memory (106), based on a plurality of depth images, the construction of the dictionary comprising: - selecting a first window by sampling a depth image from the plurality of depth images; - binarizing the selected window by assigning a minimum value to pixels in the window having a depth value less than a threshold value and assigning a maximum value to pixels in the window having a depth value greater than the threshold value; - if the binary window already exists in the dictionary, incrementing an occurrence value associated with the binary window, in the dictionary; and - if the binary window does not exist in the dictionary, storing the binary window, in the non-volatile memory, in association with an initial occurrence value; and reconstructing a depth image, by applying the method according to any one of claims 1 to 11 and based on the dictionary of binary windows.
13. A construction method according to claim 12, wherein the sampling of the depth image comprises: - randomly drawing a window of a depth image; - calculating the standard deviation of the depth values on the drawn window; and - if the standard deviation of the depth values is zero, drawing a new window.
14. A method according to claim 12 or 13, wherein the threshold value is the average value of the depth values over the selected window.
15. An electronic device comprising a memory (106, 108) and at least one processor (110), the memory storing instructions configured to implement the method of any one of claims 1 to 14 when executed by the at least one processor (110).
16. Non-transitory memory device configured to store instructions enabling the implementation of the method of any one of claims 1 to 14 when executed by at least one processor (110).
